Get the most recent info and news about WETHOD on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

WETHOD

wethod

Let me tell you a bit more about it! The Wim Hof Wethod in Action: Increased Mental Clarity: By practicing the Wim Hof breathing exercises, you can oxygenate your body, improving blood flow and ...

Enhance Your Well-being and Boost Your Performance with the Wim Hof Method

Xinyu Li, University of Washington. Table of Links Abstract and Introduction
2. Method and 2.1. G is constant
2.2. Linear Relation between G and I
2.3. Nonlinear Quadratic Relation between G and I
3. Results
4. Conclusion and References 2. Method The Keynesian cross model builds upon two ordinary differential equations [6]: where C ≥ 0 is the rate of consumer spending, I ≥ 0 is the national income, and G ≥ 0 is the rate of government spending. The parameters α and β satisfy 1 < α < ∞, 1 ≤ β < ∞. Three relations between government spending and national income are discussed in the following subsections. 2.1. G is constant Consider a model consisting of equations (1) and (2) along with a constant government spending G. To determine the equilibrium state for this model, I find the point where = Ċ = 0. Rearranging terms, I obtain the following equilibrium: In order to calculate the stability of this fixed point, I compute the Jacobian matrix and eigenvalues: This paper is available on arxiv under CC 4.0 license. Xinyu Li, University of Washington. Xinyu Li, University of Washington. Table of Links Abstract and Introduction 2. Method and 2.1. G is constant 2.2. Linear Relation between G and I 2.3. Nonlinear Quadratic Relation between G and I 3. Results 4. Conclusion and References Abstract and Introduction Abstract and Introduction 2. Method and 2.1. G is constant 2. Method and 2.1. G is constant 2.2. Linear Relation between G and I 2.2. Linear Relation between G and I 2.3. Nonlinear Quadratic Relation between G and I 2.3. Nonlinear Quadratic Relation between G and I 3. Results 3. Results 4. Conclusion and References 4. Conclusion and References 2. Method The Keynesian cross model builds upon two ordinary differential equations [6]: where C ≥ 0 is the rate of consumer spending, I ≥ 0 is the national income, and G ≥ 0 is the rate of government spending. The parameters α and β satisfy 1 < α < ∞, 1 ≤ β < ∞. Three relations between government spending and national income are discussed in the following subsections. 2.1. G is constant Consider a model consisting of equations (1) and (2) along with a constant government spending G. To determine the equilibrium state for this model, I find the point where = Ċ = 0. Rearranging terms, I obtain the following equilibrium: In order to calculate the stability of this fixed point, I compute the Jacobian matrix and eigenvalues: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Bifurcation Analysis of the Keynesian Cross Model: Method and G is constant

Authors:
(1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk);
(2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk);
(3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk);
(4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk);
(5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 3. Method 3.1. Training View Pre-processing 3.2. Progressive Training 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 5. Conclusion and 6. References 3. Method Given a pre-trained NeRF, a set of masks on its training images denoting the target object to be replaced (or removed), and a text prompt, we propose generative promptable inpainting, which can be decomposed into three objectives: 1) 3D and 4D visual content generation, where the resulting finetuned NeRF should contain a new object that is multiview and temporally consistent; 2) text-prompt guided generation, where the semantics of the generated object should match the input text prompt; 3) the generated inpainted content should be consistent with the existing NeRF background. Our proposed framework consists of three main stages, as shown in Figure 1. First, we employ stable diffusion [22] to inpaint one view as the first seed image, and generate a coarse set of seed images conditioned on the first seed image. The other views are then inferred from the seed images and refined by stable diffusion. This stage pre-processes the training images aiming to make convergence easier and faster later. Next, we fine-tune the NeRF by performing a stable diffusion version of iterative dataset update [4] to enforce 3D multiview consistency. A converged 3D NeRF is obtained in this stage. If we target at inpainting 4D NeRF, we propagate the 3D inpainted result along the time dimension in the final stage. In the following, we will describe the three stages in detail, respectively in each subsection. This paper is available on arxiv under CC 4.0 license. Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Authors: Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 2.3. Text-Guided Visual Content Generation 3. Method 3. Method 3.1. Training View Pre-processing 3.1. Training View Pre-processing 3.2. Progressive Training 3.2. Progressive Training 3.3. 4D Extension 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 4.2. Ablation and comparison 5. Conclusion and 6. References 5. Conclusion and 6. References 3. Method Given a pre-trained NeRF, a set of masks on its training images denoting the target object to be replaced (or removed), and a text prompt, we propose generative promptable inpainting, which can be decomposed into three objectives: 1) 3D and 4D visual content generation, where the resulting finetuned NeRF should contain a new object that is multiview and temporally consistent; 2) text-prompt guided generation, where the semantics of the generated object should match the input text prompt; 3) the generated inpainted content should be consistent with the existing NeRF background. Our proposed framework consists of three main stages, as shown in Figure 1. First, we employ stable diffusion [22] to inpaint one view as the first seed image, and generate a coarse set of seed images conditioned on the first seed image. The other views are then inferred from the seed images and refined by stable diffusion. This stage pre-processes the training images aiming to make convergence easier and faster later. Next, we fine-tune the NeRF by performing a stable diffusion version of iterative dataset update [4] to enforce 3D multiview consistency. A converged 3D NeRF is obtained in this stage. If we target at inpainting 4D NeRF, we propagate the 3D inpainted result along the time dimension in the final stage. In the following, we will describe the three stages in detail, respectively in each subsection. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

NeRF Editing and Inpainting Techniques: Method

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 3. Method As stated earlier, our goal in this work is to enable generation of consistent images of a character (or another kind of visual subject) based on a textual description. We achieve this by iteratively customizing a pre-trained text-to-image model, using sets of images generated by the model itself as training data. Intuitively, we refine the representation of the target character by repeatedly funneling the model’s output into a consistent identity. Once the process has converged, the resulting model can be used to generate consistent images of the target character in novel contexts. In this section, we describe our method in detail. Formally, we are given a text-to-image model MΘ, parameterized by Θ, and a text prompt p that describes a target character. The parameters Θ consist of a set of model weights θ and a set of custom text embeddings τ . We seek a representation Θ(p), s.t., the parameterized model MΘ(p) is able to generate consistent images of the character described by p in novel contexts. Our approach, described in Algorithm 1 and depicted in Figure 3, is based on the premise that a sufficiently large set of images generated by M for the same text prompt, but with different seeds, will reflect the non-uniform density of the manifold of generated images. Specifically, we expect to find some groups of images with shared characteristics. The “common ground” among the images in one of these groups can be used to refine the representation Θ(p) so as to better capture and fit the target. We therefore propose to iteratively cluster the generated images, and use the most cohesive cluster to refine Θ(p). This process is repeated, with the refined representation Θ(p), until convergence. Below, we describe the clustering and the representation refinement components of our method in detail. 3.1. Identity Clustering In Figure 4 we show a visualization of the DINOv2 embedding space, where the high-dimensional embeddings S are projected into 2D using t-SNE [29] and colored according to their K-MEANS++ [4] clusters. Some of the embeddings are clustered together more tightly than others, and the black cluster is chosen as the most cohesive one. 3.2. Identity Extraction We base our solution on a pre-trained Stable Diffusion XL (SDXL) [57] model, which utilizes two text encoders: CLIP [61] and OpenCLIP [34]. We perform textual inversion [20] to add a new pair of textual tokens τ , one for each of the two text encoders. However, we found that this parameter space is not expressive enough, as demonstrated in Section 4.3, hence we also update the model weights θ via a low-rank adaptation (LoRA) [33, 71] of the self- and cross-attention layers of the model. We use the standard denoising loss: 3.3. Convergence As explained earlier (Algorithm 1 and Figure 3), the above process is performed iteratively. Note that the representation Θ extracted in each iteration is the one used to generate the set of N images for the next iteration. The generated images are thus funneled into a consistent identity. Finally, it should be noticed that our method is nondeterministic, i.e., when running our method multiple times, on the same input prompt p, different consistent characters will be generated. This is aligned with the one-to-many nature of our task. For more details and examples, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 3. Method As stated earlier, our goal in this work is to enable generation of consistent images of a character (or another kind of visual subject) based on a textual description. We achieve this by iteratively customizing a pre-trained text-to-image model, using sets of images generated by the model itself as training data. Intuitively, we refine the representation of the target character by repeatedly funneling the model’s output into a consistent identity. Once the process has converged, the resulting model can be used to generate consistent images of the target character in novel contexts. In this section, we describe our method in detail. Formally, we are given a text-to-image model MΘ, parameterized by Θ, and a text prompt p that describes a target character. The parameters Θ consist of a set of model weights θ and a set of custom text embeddings τ . We seek a representation Θ(p), s.t., the parameterized model MΘ(p) is able to generate consistent images of the character described by p in novel contexts. Our approach, described in Algorithm 1 and depicted in Figure 3, is based on the premise that a sufficiently large set of images generated by M for the same text prompt, but with different seeds, will reflect the non-uniform density of the manifold of generated images. Specifically, we expect to find some groups of images with shared characteristics. The “common ground” among the images in one of these groups can be used to refine the representation Θ(p) so as to better capture and fit the target. We therefore propose to iteratively cluster the generated images, and use the most cohesive cluster to refine Θ(p). This process is repeated, with the refined representation Θ(p), until convergence. Below, we describe the clustering and the representation refinement components of our method in detail. 3.1. Identity Clustering In Figure 4 we show a visualization of the DINOv2 embedding space, where the high-dimensional embeddings S are projected into 2D using t-SNE [29] and colored according to their K-MEANS++ [4] clusters. Some of the embeddings are clustered together more tightly than others, and the black cluster is chosen as the most cohesive one. 3.2. Identity Extraction We base our solution on a pre-trained Stable Diffusion XL (SDXL) [57] model, which utilizes two text encoders: CLIP [61] and OpenCLIP [34]. We perform textual inversion [20] to add a new pair of textual tokens τ , one for each of the two text encoders. However, we found that this parameter space is not expressive enough, as demonstrated in Section 4.3, hence we also update the model weights θ via a low-rank adaptation (LoRA) [33, 71] of the self- and cross-attention layers of the model. We use the standard denoising loss: 3.3. Convergence As explained earlier (Algorithm 1 and Figure 3), the above process is performed iteratively. Note that the representation Θ extracted in each iteration is the one used to generate the set of N images for the next iteration. The generated images are thus funneled into a consistent identity. Finally, it should be noticed that our method is nondeterministic, i.e., when running our method multiple times, on the same input prompt p, different consistent characters will be generated. This is aligned with the one-to-many nature of our task. For more details and examples, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Method

Authors:
(1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore;
(2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore;
(3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore;
(4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore;
(5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore;
(6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction II Method III Datasets IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Authors: Authors: (1) Luyuan Peng, Acoustic Research Laboratory, National University of Singapore; (2) Hari Vishnu, Acoustic Research Laboratory, National University of Singapore; (3) Mandar Chitre, Acoustic Research Laboratory, National University of Singapore; (4) Yuen Min Too, Acoustic Research Laboratory, National University of Singapore; (5) Bharath Kalyan, Acoustic Research Laboratory, National University of Singapore; (6) Rajat Mishra, Acoustic Research Laboratory, National University of Singapore. Table of Links I Introduction I Introduction II Method II Method III Datasets III Datasets IV Experiments, Acknowledgment, and References IV Experiments, Acknowledgment, and References II. METHOD We implement two different architectures. The first uses pretrained deep convolutional neural networks (DCNN) to extract features from the input images. The resulting feature map is then passed to an affine regressor consisting of dense neural layers to output a 7-dimensional pose vector estimates. The second architecture adds an additional LSTM layer between the DCNN and affine regressor to perform dimensionality reduction by assessing the spatial structure of the image. This is done by four LSTMs which parse the DCNN-output feature map in different directions starting at each of the four image corners to process the spatial trends, and compress it into lower-dimensional information which is easier processed by the affine regressor. The input images used in the training are rescaled to 256×256 pixels before cropping into a 224×224 feature input using centre cropping. To speed up training, the images are normalized by the mean and standard deviation of the images. The poses are also normalized to lie within [-1, 1]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Underwater Visual Localization Using Machine Learning and LSTM: Method

Authors:
(1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn);
(2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn);
(3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn);
(4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn);
(5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com);
(6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction
Related Work
Our Method
Experiments
Conclusion and References
Appendix 3 OUR METHOD The goal of our method is to generate diverse storytelling images of high quality and with minimal human effort. Considering the complexity of scenes in storytelling images, our general idea is to combine the comprehension and planning capabilities of LLMs [et al 2023; OpenAI 2023] and the generation ability of the large-scale text-to-image models [Rombach et al. 2022]. The pipeline is shown in Fig. 2, which can be divided into a condition preparation stage in (a) and a conditional image generation stage in (b). Specifically, we first utilize LLMs to convert the textual descriptions of stories into layouts of the storytelling images, as detailed in Sec. 3.1. To improve the quality of generated story images, we propose a simple yet effective method to transform sparse bounding boxes into dense control signals like sketches or keypoints, without introducing manual labor (detailed in Sec. 3.2). Subsequently, we generate story images with a reasonable scene arrangement based on the layout, as detailed in Sec. 3.3. Finally, we propose a method to eliminate the requirement for users to collect training data for each character, enabling the generation of identity-consistent story images from only texts (detailed in Sec. 3.4). Since our approach only fine-tunes the pre-trained text-to-image image diffusion model on a few images, we can easily leverage existing models on civiati[1] for storytelling in arbitrary characters, scenes, and even styles. 3.1 Story to Layout Generation Layout Generation. After dividing the story into panel descriptions, we leverage LLMs to extract the scene layout from each panel description, as shown in the following equation: 3.2 Dense Condition Generation Motivation. Although using sparse bounding boxes as a control signal can improve the generation of subjects and obtain more reasonable scene layouts, we find that it cannot consistently produce high-quality generation results. There are cases where the images do not exactly match the scene layout or the generated images are of low quality, as detailed in the experiments in Sec. 4.4. We believe that this is mainly due to the limited information provided by the bounding boxes. The model faces difficulties in generating a large amount of content all at once, with limited guidance. For this reason, we propose to improve the final story image generation by introducing dense sketch or keypoint guidances. To this end, we devise a dense condition generation module based on the layout generated in the previous section, as shown in Fig. 2(d). Note that the process of composing dense conditions is fully automatic and does not require any manual interaction. 3.3 Controllable Storytelling Image Generation Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text. However, limited by the language comprehension ability of the text encoder in the model, and the incorrect association between text and image regions in the generation process, the directly generated images often suffer from a series of problems such as object missing, attribution confusion, etc [Chefer et al. 2023]. To tackle this, we introduce additional control signals to improve the quality of image generation. 3.4 Eliminating Character-wise Data Collection Identity Consistency. We propose a training-free consistency modeling method to meet the requirement of identity consistency, as shown in Fig. 3 (d). Specifically, we treat multiple images of a single character as different frames in a video and generate them simultaneously using a pre-trained diffusion model. In this process, the self-attention in the generative model is expanded to other “video frames" [Wang et al. 2023; Wu et al. 2022] to strengthen the dependencies among images, thus obtaining identity-consistent generation results. Concretely, in self-attention, we let the latent features in each frame attend to the features in the first and previous frames to build the dependency. The process can be represented as: Subsequently, we extract sketches for non-human characters and keypoints for human characters from these images. Finally, we use T2I-Adapter to inject the control guidance into the latent feature of corresponding frames in the generation process. In addition, in order to further ensure the quality of the generated data, we use CLIP score to filter the generated data, and select the images that are consistent with the text descriptions as the training data for customized generation. Discussion. In this section, we combine the proposed training-free identity-consistency modeling method with the viewpoint conditioned image translation model to achieve both identity consistency and diversity in character generation. A simpler approach is to directly use the multi-view images from the view-point conditioned image translation model as training data for customization. However, we found that the directly generated results often suffer from distortions or large differences in the color and texture of the images from different viewpoints (see Sec. 4.4 for details). For this reason, we need to leverage the above consistency modeling approach to obtain both texture- and structure-consistent images for each character. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. [1] https://civitai.com Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Authors: Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction Related Work Our Method Experiments Conclusion and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Our Method Our Method Experiments Experiments Conclusion and References Conclusion and References Appendix Appendix 3 OUR METHOD The goal of our method is to generate diverse storytelling images of high quality and with minimal human effort. Considering the complexity of scenes in storytelling images, our general idea is to combine the comprehension and planning capabilities of LLMs [et al 2023; OpenAI 2023] and the generation ability of the large-scale text-to-image models [Rombach et al. 2022]. The pipeline is shown in Fig. 2, which can be divided into a condition preparation stage in (a) and a conditional image generation stage in (b). Specifically, we first utilize LLMs to convert the textual descriptions of stories into layouts of the storytelling images, as detailed in Sec. 3.1. To improve the quality of generated story images, we propose a simple yet effective method to transform sparse bounding boxes into dense control signals like sketches or keypoints, without introducing manual labor (detailed in Sec. 3.2). Subsequently, we generate story images with a reasonable scene arrangement based on the layout, as detailed in Sec. 3.3. Finally, we propose a method to eliminate the requirement for users to collect training data for each character, enabling the generation of identity-consistent story images from only texts (detailed in Sec. 3.4). Since our approach only fine-tunes the pre-trained text-to-image image diffusion model on a few images, we can easily leverage existing models on civiati[1] for storytelling in arbitrary characters, scenes, and even styles. 3.1 Story to Layout Generation Layout Generation. After dividing the story into panel descriptions, we leverage LLMs to extract the scene layout from each panel description, as shown in the following equation: Layout Generation. 3.2 Dense Condition Generation Motivation . Although using sparse bounding boxes as a control signal can improve the generation of subjects and obtain more reasonable scene layouts, we find that it cannot consistently produce high-quality generation results. There are cases where the images do not exactly match the scene layout or the generated images are of low quality, as detailed in the experiments in Sec. 4.4. Motivation We believe that this is mainly due to the limited information provided by the bounding boxes. The model faces difficulties in generating a large amount of content all at once, with limited guidance. For this reason, we propose to improve the final story image generation by introducing dense sketch or keypoint guidances. To this end, we devise a dense condition generation module based on the layout generated in the previous section, as shown in Fig. 2(d). Note that the process of composing dense conditions is fully automatic and does not require any manual interaction. 3.3 Controllable Storytelling Image Generation Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text. However, limited by the language comprehension ability of the text encoder in the model, and the incorrect association between text and image regions in the generation process, the directly generated images often suffer from a series of problems such as object missing, attribution confusion, etc [Chefer et al. 2023]. To tackle this, we introduce additional control signals to improve the quality of image generation. 3.4 Eliminating Character-wise Data Collection Identity Consistency. We propose a training-free consistency modeling method to meet the requirement of identity consistency, as shown in Fig. 3 (d). Specifically, we treat multiple images of a single character as different frames in a video and generate them simultaneously using a pre-trained diffusion model. In this process, the self-attention in the generative model is expanded to other “video frames" [Wang et al. 2023; Wu et al. 2022] to strengthen the dependencies among images, thus obtaining identity-consistent generation results. Concretely, in self-attention, we let the latent features in each frame attend to the features in the first and previous frames to build the dependency. The process can be represented as: Identity Consistency. Subsequently, we extract sketches for non-human characters and keypoints for human characters from these images. Finally, we use T2I-Adapter to inject the control guidance into the latent feature of corresponding frames in the generation process. In addition, in order to further ensure the quality of the generated data, we use CLIP score to filter the generated data, and select the images that are consistent with the text descriptions as the training data for customized generation. Discussion . In this section, we combine the proposed training-free identity-consistency modeling method with the viewpoint conditioned image translation model to achieve both identity consistency and diversity in character generation. A simpler approach is to directly use the multi-view images from the view-point conditioned image translation model as training data for customization. However, we found that the directly generated results often suffer from distortions or large differences in the color and texture of the images from different viewpoints (see Sec. 4.4 for details). For this reason, we need to leverage the above consistency modeling approach to obtain both texture- and structure-consistent images for each character. Discussion This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [1] https://civitai.com

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort: Our Method

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References 3 METHOD Our method starts with a DIP network to reconstruct the target image. A second output head is added and tasked with inpainting the desired alpha in the trimap unknown region, constrained by the values in known regions. The main idea is that the first output head forces the network representation preceding it to "understand the structure of the image," while the second output head makes use of that information in estimating the matte. In addition to the alpha output we add two additional networks which simultaneously reconstruct the foreground and background. Similarly to the first output of the first network, these networks’ outputs are constrained to match the target image but, unlike the first network, they are constrained only in their respective regions of the trimap. Similarly to the alpha output, they extrapolate those constraints to inpaint the unconstrained region (see Fig. 2). The latter three outputs – 𝛼,ˆ 𝐹,ˆ 𝐵ˆ – are coupled via an additional constraint that they together satisfy (1) (see Detailed Loss below). We put 𝐹ˆ and 𝐵ˆ in separate networks (as in [Gandelsman et al. 2019]) rather than as additional outputs heads of the first network (as in [Sun et al. 2021]) in order to allow the inpainting of the foreground to be independent of the background. All three networks share the same generic U-net structure [Ulyanov et al. 2018], except for an additional output head in the first network. Our experiments use Adam [Kingma and Ba 2015] with a learning rate of 0.001. 3.1 Detailed Loss The first term of our loss function is the reconstruction loss between the first network output and the target image: The second loss term constrains 𝛼ˆ, where 𝑇 is the trimap and 𝐶 = 𝐹 ∪ 𝐵 is the constrained region of the trimap. The reconstruction losses for the foreground and background outputs are defined similarly to 𝐿𝐼 , but constrained only in their respective regions of the trimap: The three networks’ outputs are coupled via the alpha-compositing equation as follows: where 𝑈 = 𝐼 − 𝐶 is the unconstrained region of the trimap. Finally, we include an exclusion loss similar to [Sun et al. 2021] to prevent the structure of the foreground from leaking into the background and vice-versa: The total loss is the sum of the above six components: Unlike other DIP-based techniques, our algorithm does not require early stopping since the goal is to exactly fit both the image and the trimap constraints. 3.2 Temporal continuity In our experiments temporal continuity was obtained by warmstarting the optimization for frames other than the first with the final weight values of the previous frame, and stopping with a loss threshold rather than a fixed number of iterations (see video). This simple strategy produces reasonable results even on the relatively difficult case of hair. It also reduces the compute time by roughly an order of magnitude. 3.3 Why does it work? The deep image prior empirically demonstrates that standard convolutional networks can act as a good low-level prior for image reconstruction tasks, but why can it be adapted to invent plausible alpha mattes? Our intuition is as follows: The DIP provides a somewhat deep and hierarchical encoding of the particular image. The features in this encoding will primarily span valid image structures, with noise lying outside this space. In our work these features are re-combined to produce the matte. This works because the alpha matte contains structures that relate to those in the foreground image. In this respect our approach resembles a "deep" version of the guided image filter [He et al. 2010]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References 3 METHOD Our method starts with a DIP network to reconstruct the target image. A second output head is added and tasked with inpainting the desired alpha in the trimap unknown region, constrained by the values in known regions. The main idea is that the first output head forces the network representation preceding it to "understand the structure of the image," while the second output head makes use of that information in estimating the matte. In addition to the alpha output we add two additional networks which simultaneously reconstruct the foreground and background. Similarly to the first output of the first network, these networks’ outputs are constrained to match the target image but, unlike the first network, they are constrained only in their respective regions of the trimap. Similarly to the alpha output, they extrapolate those constraints to inpaint the unconstrained region (see Fig. 2). The latter three outputs – 𝛼,ˆ 𝐹,ˆ 𝐵ˆ – are coupled via an additional constraint that they together satisfy (1) (see Detailed Loss below). We put 𝐹ˆ and 𝐵ˆ in separate networks (as in [Gandelsman et al. 2019]) rather than as additional outputs heads of the first network (as in [Sun et al. 2021]) in order to allow the inpainting of the foreground to be independent of the background. All three networks share the same generic U-net structure [Ulyanov et al. 2018], except for an additional output head in the first network. Our experiments use Adam [Kingma and Ba 2015] with a learning rate of 0.001. 3.1 Detailed Loss The first term of our loss function is the reconstruction loss between the first network output and the target image: The second loss term constrains 𝛼ˆ, where 𝑇 is the trimap and 𝐶 = 𝐹 ∪ 𝐵 is the constrained region of the trimap. The reconstruction losses for the foreground and background outputs are defined similarly to 𝐿𝐼 , but constrained only in their respective regions of the trimap: The three networks’ outputs are coupled via the alpha-compositing equation as follows: where 𝑈 = 𝐼 − 𝐶 is the unconstrained region of the trimap. Finally, we include an exclusion loss similar to [Sun et al. 2021] to prevent the structure of the foreground from leaking into the background and vice-versa: The total loss is the sum of the above six components: Unlike other DIP-based techniques, our algorithm does not require early stopping since the goal is to exactly fit both the image and the trimap constraints. 3.2 Temporal continuity In our experiments temporal continuity was obtained by warmstarting the optimization for frames other than the first with the final weight values of the previous frame, and stopping with a loss threshold rather than a fixed number of iterations (see video). This simple strategy produces reasonable results even on the relatively difficult case of hair. It also reduces the compute time by roughly an order of magnitude. 3.3 Why does it work? The deep image prior empirically demonstrates that standard convolutional networks can act as a good low-level prior for image reconstruction tasks, but why can it be adapted to invent plausible alpha mattes? Our intuition is as follows: The DIP provides a somewhat deep and hierarchical encoding of the particular image. The features in this encoding will primarily span valid image structures, with noise lying outside this space. In our work these features are re-combined to produce the matte. This works because the alpha matte contains structures that relate to those in the foreground image. In this respect our approach resembles a "deep" version of the guided image filter [He et al. 2010]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Method

Comparison of Machine Learning Methods: Approach

Author:
(1) Melody Yu, Sage Hill School, California, USA. Table of Links Abstract and Introduction
Dataset
Methods
Results
Discussion
Conclusion, References, and Appendix III. METHODS We seek to establish a relationship between IMDb ratings and network indicators for a given TV series. Each temporal graph corresponds to ten scenes from an episode, whereas each IMDb review corresponds to one episode. Thus, to study the graph metrics for each episode, we first aggregate all segment graphs in one episode into one single weighted episode graph. Then, we study metrics on the aggregated episode graph. A. Segment Graph Aggregation B. Network Metrics We seek to establish a relationship between IMDb ratings and network indicators for a given TV series. Each temporal graph corresponds to ten scenes from an episode, whereas each IMDb review corresponds to one episode. Thus, to study the graph metrics for each episode, we first aggregate all segment graphs in one episode into one single weighted episode graph. Then, we study metrics on the aggregated episode graph. 1) Active Nodes: Active nodes are the total number of nodes in a graph with at least one edge connecting them to another node. The number of active nodes represents the number of characters that speak during an episode. We use the number of active nodes in an episode graph to describe the complexity of the episode plot in this study. 2) Network Density: A graph’s density is the ratio between the number of actual edges that exist in a graph to the maximum number of edges that the graph can have. It indicates how many of a graph’s potential connections are real connections. We define density as the following, where n represents the number of nodes and m represents the number of edges. Figure 4 shows the time series for the number of active nodes and the network density for different segment graphs from Game of Thrones episode 1. The density of the graph increases as more characters converse with one another. Characters in sparse graphs are often still connected, but the conversations are between two characters and not necessarily talk with many other characters. We use network density to measure the level of character interactions in each TV episode. 3) Node Strength: A node’s strength is defined as the sum of the strengths, or weights, of the node’s edges. Since the edge weight in our graph indicates the time spent conversing between two characters, the strength of a character is the total amount of time spent conversing with all other characters. Figure 5 shows the time series of characters with top 5 highest graph node strength from the Game of Thrones episode 1 graph. The maximum value of node strengths describes the total conversation time of the character who talked most with other characters during this episode. We use this metric to describe the exposure of the most active character in the episode. In this study, the TV episode graphs are usually separated into separate subgraphs, often without any connections between these subgraphs. To more accurately represent these subgraphs, we use local efficiency, calculated from the average value of a subgraph’s global efficiency. 5) Network Transitivity: Network transitivity is the overall probability for the network to have interconnected adjacent nodes, revealing the existence of tightly connected communities. It refers to the extent to which the relation that relates two nodes in a network that are connected by an edge is transitive. The network transitivity is the fraction of all possible triangles present in G. It is computed as dividing total number of node triangles in the graphs by the total number of “triads” (two edges with a shared vertex) Network transitivity is widely used to examine the level of clustering in social network analysis. In the content of character networks, high transitivity means that the network contains communities or groups of characters that are closely interacting with each other. We use network transitivity to measure the extent of the complex relationship between characters, and their relation to the plot. 6) Degree Centrality: In a connected graph, centrality is used to measure the importance of various nodes in a graph. In this study, we use various centrality metrics to measure the conversational interaction patterns of different characters in each TV episode. Degree centrality is the simplest measure of centrality. The degree of a node is defined as the number of edges or connections that a node has to other nodes. In the context of character networks in TV shows, the degree of a character represents the number of other characters that they interact with in a given episode. Figure 6 illustrates the time series of characters with the top 5 highest degrees in the character network from the first episode of Game of Thrones. A node with a higher degree than the rest of the nodes may be considered a central or pivotal character, as they have a large number of interactions with other characters. As our episode graph is aggregated from multiple segment graphs, the large number of interactions can be spread over different scenes, or concentrated in a few scenes involving many characters. In this research, we will use the distribution of node degrees, including the maximum value and standard deviation, to describe the patterns of conversational interactions among the characters in a given episode. 7) Closeness Centrality and Harmonic Centrality: The closeness centrality of a node in a connected graph, which is determined as the reciprocal of the sum of the shortest distances between the node and all other nodes in the graph, is a measure of centrality in a network. A node with a higher closeness centrality value is closer to all other nodes. However, in our study, the TV episode graph is not necessarily a connected graph, as there can be some characters forming subgroups that do not interact with characters outside their subgroup. Therefore, we use harmonic centrality instead of closeness centrality to allow disconnected networks in our research. The harmonic centrality of a node u is defined as the sum of reciprocals of the shortest path distances from all other nodes to node u. Two nodes that are unreachable to each other has a distance of infinity. In our study, we use harmonic centralization distribution (max value, standard deviation) to measure the extent to which the plot of the TV episode is concentrated on a single actor or a group of actors. 8) Eigenvector Centrality: While degree centrality measures the number of connections a node has, it does not take into account the influence of a node’s neighbors on its own importance. To address this, we can use eigenvector centrality, which considers the importance of a node’s neighbors in determining the node’s overall importance. A node with a high eigenvector score is connected to many other nodes with high scores, indicating that it is connected to influential characters. In the context of character networks in TV shows, a character with a high eigenvector centrality is likely to be connected to characters who are also considered influential within the network. This can be useful in understanding the importance of a character within the context of the overall plot and character relationships. This paper is available on arxiv under CC 4.0 license. Author: (1) Melody Yu, Sage Hill School, California, USA. Author: Author: (1) Melody Yu, Sage Hill School, California, USA. Table of Links Abstract and Introduction Dataset Methods Results Discussion Conclusion, References, and Appendix Abstract and Introduction Abstract and Introduction Dataset Dataset Methods Methods Results Results Discussion Discussion Conclusion, References, and Appendix Conclusion, References, and Appendix III. METHODS We seek to establish a relationship between IMDb ratings and network indicators for a given TV series. Each temporal graph corresponds to ten scenes from an episode, whereas each IMDb review corresponds to one episode. Thus, to study the graph metrics for each episode, we first aggregate all segment graphs in one episode into one single weighted episode graph. Then, we study metrics on the aggregated episode graph. A. Segment Graph Aggregation A. Segment Graph Aggregation B. Network Metrics B. Network Metrics We seek to establish a relationship between IMDb ratings and network indicators for a given TV series. Each temporal graph corresponds to ten scenes from an episode, whereas each IMDb review corresponds to one episode. Thus, to study the graph metrics for each episode, we first aggregate all segment graphs in one episode into one single weighted episode graph. Then, we study metrics on the aggregated episode graph. 1) Active Nodes: Active nodes are the total number of nodes in a graph with at least one edge connecting them to another node. The number of active nodes represents the number of characters that speak during an episode. We use the number of active nodes in an episode graph to describe the complexity of the episode plot in this study. 1) Active Nodes: 2) Network Density: A graph’s density is the ratio between the number of actual edges that exist in a graph to the maximum number of edges that the graph can have. It indicates how many of a graph’s potential connections are real connections. We define density as the following, where n represents the number of nodes and m represents the number of edges. 2) Network Density: Figure 4 shows the time series for the number of active nodes and the network density for different segment graphs from Game of Thrones episode 1. The density of the graph increases as more characters converse with one another. Characters in sparse graphs are often still connected, but the conversations are between two characters and not necessarily talk with many other characters. We use network density to measure the level of character interactions in each TV episode. 3) Node Strength: A node’s strength is defined as the sum of the strengths, or weights, of the node’s edges. Since the edge weight in our graph indicates the time spent conversing between two characters, the strength of a character is the total amount of time spent conversing with all other characters. 3) Node Strength: Figure 5 shows the time series of characters with top 5 highest graph node strength from the Game of Thrones episode 1 graph. The maximum value of node strengths describes the total conversation time of the character who talked most with other characters during this episode. We use this metric to describe the exposure of the most active character in the episode. In this study, the TV episode graphs are usually separated into separate subgraphs, often without any connections between these subgraphs. To more accurately represent these subgraphs, we use local efficiency, calculated from the average value of a subgraph’s global efficiency. 5) Network Transitivity: Network transitivity is the overall probability for the network to have interconnected adjacent nodes, revealing the existence of tightly connected communities. It refers to the extent to which the relation that relates two nodes in a network that are connected by an edge is transitive. The network transitivity is the fraction of all possible triangles present in G. It is computed as dividing total number of node triangles in the graphs by the total number of “triads” (two edges with a shared vertex) 5) Network Transitivity: Network transitivity is widely used to examine the level of clustering in social network analysis. In the content of character networks, high transitivity means that the network contains communities or groups of characters that are closely interacting with each other. We use network transitivity to measure the extent of the complex relationship between characters, and their relation to the plot. 6) Degree Centrality: In a connected graph, centrality is used to measure the importance of various nodes in a graph. In this study, we use various centrality metrics to measure the conversational interaction patterns of different characters in each TV episode. 6) Degree Centrality: Degree centrality is the simplest measure of centrality. The degree of a node is defined as the number of edges or connections that a node has to other nodes. In the context of character networks in TV shows, the degree of a character represents the number of other characters that they interact with in a given episode. Figure 6 illustrates the time series of characters with the top 5 highest degrees in the character network from the first episode of Game of Thrones. A node with a higher degree than the rest of the nodes may be considered a central or pivotal character, as they have a large number of interactions with other characters. As our episode graph is aggregated from multiple segment graphs, the large number of interactions can be spread over different scenes, or concentrated in a few scenes involving many characters. In this research, we will use the distribution of node degrees, including the maximum value and standard deviation, to describe the patterns of conversational interactions among the characters in a given episode. 7) Closeness Centrality and Harmonic Centrality: The closeness centrality of a node in a connected graph, which is determined as the reciprocal of the sum of the shortest distances between the node and all other nodes in the graph, is a measure of centrality in a network. A node with a higher closeness centrality value is closer to all other nodes. 7) Closeness Centrality and Harmonic Centrality: However, in our study, the TV episode graph is not necessarily a connected graph, as there can be some characters forming subgroups that do not interact with characters outside their subgroup. Therefore, we use harmonic centrality instead of closeness centrality to allow disconnected networks in our research. The harmonic centrality of a node u is defined as the sum of reciprocals of the shortest path distances from all other nodes to node u. Two nodes that are unreachable to each other has a distance of infinity. In our study, we use harmonic centralization distribution (max value, standard deviation) to measure the extent to which the plot of the TV episode is concentrated on a single actor or a group of actors. 8) Eigenvector Centrality: While degree centrality measures the number of connections a node has, it does not take into account the influence of a node’s neighbors on its own importance. To address this, we can use eigenvector centrality, which considers the importance of a node’s neighbors in determining the node’s overall importance. 8) Eigenvector Centrality: A node with a high eigenvector score is connected to many other nodes with high scores, indicating that it is connected to influential characters. In the context of character networks in TV shows, a character with a high eigenvector centrality is likely to be connected to characters who are also considered influential within the network. This can be useful in understanding the importance of a character within the context of the overall plot and character relationships. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Decoding the Popularity of TV Series: A Network Analysis Perspective: Methods

Comparison of Machine Learning Methods: Conclusions and Future Work, and References

Authors:
(1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr};
(2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction
Background
Approach
Experiments and Results
Discussion and Qualitative Analysis
Related Work
Conclusions and Future Work, and References 2 Background To understand our approach, it is essential to have a solid understanding of the structure of software issues in Issue Tracking Systems (ITS). 2.1 Software Issues in ITS This section explores software issues within ITS, covering their anatomy, life cycle, and significance in software projects. Issue Tracking Systems (ITS) An issue tracking system (ITS) is a software application that manages and maintains lists of issues. It provides a database of issue reports for a software project. Members of the software development team stay aligned and collaborate faster with the help of these tools. Users can log in with a password and post a new issue report or comment on an already-posted issue. There are various issue-tracking tools that are used in software development projects, such as JIRA, Bugzilla, and Azure DevOps. Although there are some differences among these tools, they all share a similar basic structure. We developed our approach with a data set of a company that uses JIRA software. Anatomy of an Issue The issue report in ITS contains various fields, each of which contains a different piece of information as shown in Figure 1. The following details [48] could typically be included in the generic process of reporting an issue: – Summary: Title of the issue. – Description: Issue details including its cause, location, and timing. – Version: Version of the project to which the issue belongs. – Component: The predetermined component on which the issue depends. – Attachment: Attaches such as pictures, videos, and documents uploaded to the issue. – Priority: Urgency level of the issue. – Severity: Frequent occurrence and systemic effects. – Status: Current status of the issue. – Reporter: Person who creates the issue. – Assignee: Person to whom the issue is assigned. – Revision History: Historical changes of the issue. – Estimated time: Estimated time spent to develop or solve the issue. – Comments: Additional details that can be used to understand the issue. The summary, description, and comments all contain textual details about the issue. In our methodology, we extract numerical features using the language structure of summary and description. Version is the current version of the project which differs with each release. The issue can have a predefined tag or component added to it. (e.g. project v1.1.0) Users can upload files to help others understand the issue, an attachment refers to it. Priority, severity, and status are the categorical features of the issue. Priority is the urgency level, severity is the frequency of occurrence and status is the current status of the issue such as committed or done. People play different roles as they interact with reports in ITS. The person who submits the report is the reporter and the assignee is the person to whom the issue is assigned. If the issue has been reported previously, historical changes are shown in the revision history. The estimated time is the time spent on the development and varies depending on the effort metric. The fields offered by each ITS vary, and not all fields are filled out for each project. Our strategy makes use of the fields that are largely filled in. Life Cycle of an Issue The series of phases and phase changes that an issue experiences throughout its life cycle is referred to as a “workflow.” Workflows for issues typically represent development cycles and business processes. Figure 2 shows a standard workflow of JIRA. The following stages of the JIRA workflow must be monitored as soon as an issue is created: – Open: The issue is open after creation and can be assigned to the assignee to begin working on it. – In Progress: The assignee has taken the initiative to begin working on the issue. – Resolved: The issue’s sub-tasks and works have all been finished. The reporter is currently waiting to confirm the matter. If verification is successful, the case will be closed or reopened depending on whether any additional changes are needed. – Reopened: Although this issue has already been solved, the solution is either flawed, omitted some important details, or needs some modifications. Issues are classified as assigned or resolved at the Reopened stage. – Closed: The issue is now regarded as resolved, and the current solution is accurate. Issues that have been closed can be reopened at a later time as needed. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Authors: Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction Background Approach Experiments and Results Discussion and Qualitative Analysis Related Work Conclusions and Future Work, and References Abstract and Introduction Abstract and Introduction Background Background Approach Approach Experiments and Results Experiments and Results Discussion and Qualitative Analysis Discussion and Qualitative Analysis Related Work Related Work Conclusions and Future Work, and References Conclusions and Future Work, and References 2 Background To understand our approach, it is essential to have a solid understanding of the structure of software issues in Issue Tracking Systems (ITS). 2.1 Software Issues in ITS This section explores software issues within ITS, covering their anatomy, life cycle, and significance in software projects. Issue Tracking Systems (ITS) An issue tracking system (ITS) is a software application that manages and maintains lists of issues. It provides a database of issue reports for a software project. Members of the software development team stay aligned and collaborate faster with the help of these tools. Users can log in with a password and post a new issue report or comment on an already-posted issue. There are various issue-tracking tools that are used in software development projects, such as JIRA, Bugzilla, and Azure DevOps. Although there are some differences among these tools, they all share a similar basic structure. We developed our approach with a data set of a company that uses JIRA software. Issue Tracking Systems (ITS) Anatomy of an Issue The issue report in ITS contains various fields, each of which contains a different piece of information as shown in Figure 1. The following details [48] could typically be included in the generic process of reporting an issue: Anatomy of an Issue – Summary: Title of the issue. – Description: Issue details including its cause, location, and timing. – Version: Version of the project to which the issue belongs. – Component: The predetermined component on which the issue depends. – Attachment: Attaches such as pictures, videos, and documents uploaded to the issue. – Priority: Urgency level of the issue. – Severity: Frequent occurrence and systemic effects. – Status: Current status of the issue. – Reporter: Person who creates the issue. – Assignee: Person to whom the issue is assigned. – Revision History: Historical changes of the issue. – Estimated time: Estimated time spent to develop or solve the issue. – Comments: Additional details that can be used to understand the issue. The summary, description, and comments all contain textual details about the issue. In our methodology, we extract numerical features using the language structure of summary and description. Version is the current version of the project which differs with each release. The issue can have a predefined tag or component added to it. (e.g. project v1.1.0) Users can upload files to help others understand the issue, an attachment refers to it. Priority, severity, and status are the categorical features of the issue. Priority is the urgency level, severity is the frequency of occurrence and status is the current status of the issue such as committed or done. People play different roles as they interact with reports in ITS. The person who submits the report is the reporter and the assignee is the person to whom the issue is assigned. If the issue has been reported previously, historical changes are shown in the revision history. The estimated time is the time spent on the development and varies depending on the effort metric. The fields offered by each ITS vary, and not all fields are filled out for each project. Our strategy makes use of the fields that are largely filled in. Life Cycle of an Issue The series of phases and phase changes that an issue experiences throughout its life cycle is referred to as a “workflow.” Workflows for issues typically represent development cycles and business processes. Figure 2 shows a standard workflow of JIRA. The following stages of the JIRA workflow must be monitored as soon as an issue is created: Life Cycle of an Issue – Open: The issue is open after creation and can be assigned to the assignee to begin working on it. – In Progress: The assignee has taken the initiative to begin working on the issue. – Resolved: The issue’s sub-tasks and works have all been finished. The reporter is currently waiting to confirm the matter. If verification is successful, the case will be closed or reopened depending on whether any additional changes are needed. – Reopened: Although this issue has already been solved, the solution is either flawed, omitted some important details, or needs some modifications. Issues are classified as assigned or resolved at the Reopened stage. – Closed: The issue is now regarded as resolved, and the current solution is accurate. Issues that have been closed can be reopened at a later time as needed. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Comparison of Machine Learning Methods: Background

Comparison of Machine Learning Methods: Abstract and Introduction

Authors:
(1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr};
(2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction
Background
Approach
Experiments and Results
Discussion and Qualitative Analysis
Related Work
Conclusions and Future Work, and References 4 Experiments and Results This section presents the findings and outcomes of the conducted experiments, providing a comprehensive analysis of the collected data. We critically assess the performance and effectiveness of the proposed methodology by employing various evaluation metrics and techniques. We compare the extracted features from different perspectives, evaluating their individual contributions to the classification task. Lastly, we present a thorough statistical examination of the obtained results, employing appropriate statistical tests and measures to validate the significance and reliability of the findings. 4.1 Evaluation Table 5 presents the experiment results for the issue assignment. The models are evaluated on Team Assignment (TA) and Developer Assignment (DA). The stacking model (RF and Linear SVC) achieved the highest accuracy, with values of 0.92 for TA and 0.89 for DA. Other models, such as Support Vector Machine, Logistic Regression, and Random Forest, also showed good performance with accuracies ranging from 0.86 to 0.88. The transformer-based models, including DistilBert, Roberta, and Electra, demonstrated competitive accuracies, with Roberta and Electra achieving the highest scores in some cases. Table 6 provides the performance metrics for each class in the Stacking algorithm for issue assignment. Under the Team Assignment (TA) approach, the Stacking algorithm achieved a high precision value of 0.90 for the Developer class, indicating a low rate of false positive assignments. The Recall score of 0.99 for the Developer class demonstrates the algorithm’s ability to correctly identify the majority of instances assigned to developers. The Tester class shows a balance between precision (0.95) and recall (0.71), indicating accurate assignments with a relatively high rate of false negatives. The Designer class exhibits similar trends with a precision of 0.92 and a recall of 0.67. The Leader class has relatively lower precision and recall scores, indicating more challenging assignments for the algorithm. Under the Developer Assignment (DA) approach, the Stacking algorithm achieved high precision values for the Senior class (0.90) and the Mid class (0.89), indicating accurate assignments with low rates of false positives. The Mid class also demonstrates a high recall score of 0.94, indicating effective identification of instances assigned to this class. The Junior class shows a lower precision (0.62) and recall (0.53) compared to the other classes, suggesting potential challenges in accurately assigning instances to this class. We also test our classification algorithms using the most popular word embedding techniques to determine how well our features work. Figure 5 illustrates the comparison of our feature set with Tf-Idf and BOW methods for the issue assignment. Despite the potential of Word2Vec as a word embedding algorithm, the accuracy results in my approach do not yield comparable outcomes. We use the accuracy score as the comparison metric. The graph demonstrates that using our feature set yields superior results while using Tf-Idf and BOW yields comparable results. 4.2 Feature Comparison Reducing the number of redundant and irrelevant features is an effective way to improve the running time and generalization capability of a learning algorithm [12]. Feature selection methods are used to choose a subset of relevant features that contribute to the intended concept. These methods can employ a variety of models and techniques to calculate feature importance scores. One simple approach [9] is to calculate the coefficient statistics between each feature and the target variable. This method can help to identify the most important features of a given problem and discard the redundant or irrelevant ones. By reducing the number of features used for training a model, the running time of the algorithm can be significantly reduced without sacrificing accuracy. Moreover, feature selection can also improve the interpretability of the model, as it helps to identify the key factors that influence the target variable. We present the coefficient values of each feature in Figure 6. We find that Issue Type, namely FJC1, emerges as the most influential feature from our feature set. Apart from the Issue Type, we discover that the features Watchers and Summary Words also exhibit significant effectiveness in our analysis. Conversely, features such as Reopen Count, Test Execution Type, and Approval Type demonstrate no impact on our issue assignment process. In Figure 7, we present the effective POS tags in Turkish, highlighting the most influential ones among all POS tags. Notably, the number of unknown words, verbs, and nouns emerge as the most impactful features. Following the rigorous selection of the best features, we proceed to employ Scikit-learn’s [38] SelectFromModel, a powerful meta-transformer designed to choose features based on their importance weights, to retrain our models. Through this process, we carefully identify and select the top eight features that exhibited the highest significance, as determined by the module. Remarkably, leveraging this refined feature subset allows us to achieve optimal performance and attain the most favorable outcome in our experiments. 4.3 Statistical Analysis Statistical significance tests are conducted to compare classification methods for determining whether one learning algorithm outperforms another on a particular learning task. Dietterich [14] reviews five approximate statistical tests and concludes that McNemar’s test and the 5x2 cv t-test, both have low type I error and sufficient power. In our study, we combine all data sets into a single data set for the classification algorithm. Dietterich [14] recommends using a 5x2 ttest to statistically compare two classifiers on a single data set. The 5x2 f-test, which is also suggested as the new standard by the original authors above, is further expanded upon by Alpaydın [4]. The following lists the null hypothesis and the alternative hypothesis. The null hypothesis is that the probabilities are the same, or in simpler terms, neither of the two models outperforms the other. The alternative hypothesis, therefore, holds that the performances of the two models are not equivalent. Accordingly, we apply the 5x2 f-test implemented by Alpaydın [4] which is an extension of the 5x2 cv t-test as stated above. We create the matrix for all pairwise comparisons of learning algorithms. In this test, the splitting process (50% training and 50% test data) is repeated five times. A and B are fitted to the training split and their performance on the test split in each of the five iterations is assessed. The training and test sets are then rotated (the training set becomes the test set, and vice versa), and the performance is computed again, yielding two performance difference measures. Then, the mean and variance of the differences are estimated and the f-statistic proposed by Alpaydın is calculated as This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Authors: Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction Background Approach Experiments and Results Discussion and Qualitative Analysis Related Work Conclusions and Future Work, and References Abstract and Introduction Abstract and Introduction Background Background Approach Approach Experiments and Results Experiments and Results Discussion and Qualitative Analysis Discussion and Qualitative Analysis Related Work Related Work Conclusions and Future Work, and References Conclusions and Future Work, and References 4 Experiments and Results This section presents the findings and outcomes of the conducted experiments, providing a comprehensive analysis of the collected data. We critically assess the performance and effectiveness of the proposed methodology by employing various evaluation metrics and techniques. We compare the extracted features from different perspectives, evaluating their individual contributions to the classification task. Lastly, we present a thorough statistical examination of the obtained results, employing appropriate statistical tests and measures to validate the significance and reliability of the findings. 4.1 Evaluation Table 5 presents the experiment results for the issue assignment. The models are evaluated on Team Assignment (TA) and Developer Assignment (DA). The stacking model (RF and Linear SVC) achieved the highest accuracy, with values of 0.92 for TA and 0.89 for DA. Other models, such as Support Vector Machine, Logistic Regression, and Random Forest, also showed good performance with accuracies ranging from 0.86 to 0.88. The transformer-based models, including DistilBert, Roberta, and Electra, demonstrated competitive accuracies, with Roberta and Electra achieving the highest scores in some cases. Table 6 provides the performance metrics for each class in the Stacking algorithm for issue assignment. Under the Team Assignment (TA) approach, the Stacking algorithm achieved a high precision value of 0.90 for the Developer class, indicating a low rate of false positive assignments. The Recall score of 0.99 for the Developer class demonstrates the algorithm’s ability to correctly identify the majority of instances assigned to developers. The Tester class shows a balance between precision (0.95) and recall (0.71), indicating accurate assignments with a relatively high rate of false negatives. The Designer class exhibits similar trends with a precision of 0.92 and a recall of 0.67. The Leader class has relatively lower precision and recall scores, indicating more challenging assignments for the algorithm. Under the Developer Assignment (DA) approach, the Stacking algorithm achieved high precision values for the Senior class (0.90) and the Mid class (0.89), indicating accurate assignments with low rates of false positives. The Mid class also demonstrates a high recall score of 0.94, indicating effective identification of instances assigned to this class. The Junior class shows a lower precision (0.62) and recall (0.53) compared to the other classes, suggesting potential challenges in accurately assigning instances to this class. We also test our classification algorithms using the most popular word embedding techniques to determine how well our features work. Figure 5 illustrates the comparison of our feature set with Tf-Idf and BOW methods for the issue assignment. Despite the potential of Word2Vec as a word embedding algorithm, the accuracy results in my approach do not yield comparable outcomes. We use the accuracy score as the comparison metric. The graph demonstrates that using our feature set yields superior results while using Tf-Idf and BOW yields comparable results. 4.2 Feature Comparison Reducing the number of redundant and irrelevant features is an effective way to improve the running time and generalization capability of a learning algorithm [12]. Feature selection methods are used to choose a subset of relevant features that contribute to the intended concept. These methods can employ a variety of models and techniques to calculate feature importance scores. One simple approach [9] is to calculate the coefficient statistics between each feature and the target variable. This method can help to identify the most important features of a given problem and discard the redundant or irrelevant ones. By reducing the number of features used for training a model, the running time of the algorithm can be significantly reduced without sacrificing accuracy. Moreover, feature selection can also improve the interpretability of the model, as it helps to identify the key factors that influence the target variable. We present the coefficient values of each feature in Figure 6. We find that Issue Type, namely FJC1, emerges as the most influential feature from our feature set. Apart from the Issue Type, we discover that the features Watchers and Summary Words also exhibit significant effectiveness in our analysis. Conversely, features such as Reopen Count, Test Execution Type, and Approval Type demonstrate no impact on our issue assignment process. In Figure 7, we present the effective POS tags in Turkish, highlighting the most influential ones among all POS tags. Notably, the number of unknown words, verbs, and nouns emerge as the most impactful features. Following the rigorous selection of the best features, we proceed to employ Scikit-learn’s [38] SelectFromModel, a powerful meta-transformer designed to choose features based on their importance weights, to retrain our models. Through this process, we carefully identify and select the top eight features that exhibited the highest significance, as determined by the module. Remarkably, leveraging this refined feature subset allows us to achieve optimal performance and attain the most favorable outcome in our experiments. 4.3 Statistical Analysis Statistical significance tests are conducted to compare classification methods for determining whether one learning algorithm outperforms another on a particular learning task. Dietterich [14] reviews five approximate statistical tests and concludes that McNemar’s test and the 5x2 cv t-test, both have low type I error and sufficient power. In our study, we combine all data sets into a single data set for the classification algorithm. Dietterich [14] recommends using a 5x2 ttest to statistically compare two classifiers on a single data set. The 5x2 f-test, which is also suggested as the new standard by the original authors above, is further expanded upon by Alpaydın [4]. The following lists the null hypothesis and the alternative hypothesis. The null hypothesis is that the probabilities are the same, or in simpler terms, neither of the two models outperforms the other. The alternative hypothesis, therefore, holds that the performances of the two models are not equivalent. Accordingly, we apply the 5x2 f-test implemented by Alpaydın [4] which is an extension of the 5x2 cv t-test as stated above. We create the matrix for all pairwise comparisons of learning algorithms. In this test, the splitting process (50% training and 50% test data) is repeated five times. A and B are fitted to the training split and their performance on the test split in each of the five iterations is assessed. The training and test sets are then rotated (the training set becomes the test set, and vice versa), and the performance is computed again, yielding two performance difference measures. Then, the mean and variance of the differences are estimated and the f-statistic proposed by Alpaydın is calculated as This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Comparison of Machine Learning Methods: Experiments and Results

Authors:
(1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr};
(2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction
Background
Approach
Experiments and Results
Discussion and Qualitative Analysis
Related Work
Conclusions and Future Work, and References 6 Related Work Several studies in the literature have focused on issue classification, which has addressed a variety of objectives, including issue assignment, effort estimation, issue prioritization, and so on. In this section, we briefly give details regarding issue assignment studies in general and all Turkish-language issue classification studies in particular. Several types of research have been conducted in order to automate the time-consuming task of issue assignment. In 2017, Goyal et al. [18] review and categorize 75 research papers on the automated bug assignment area. They identify seven categories: machine learning [25,47,55,5,8,36], information retrieval [58], auction [24], social network [57], tossing graph [50], fuzzy set [37] and operational research based [26] techniques. They capture the fact that for automatic bug report assignment, machine learning and information retrieval techniques are the most popular ones. In recent years, deep learning algorithms have also been successfully applied in this field, which has recently revolutionized the idea of word sequence representation and demonstrated encouraging advancements in a number of classification tasks [17]. In this section, we restrict our focus to machine learning and deep learning architectures used to train issue assignment systems. The machine learning algorithms use historical bug reports to build a supervised or unsupervised machine learning classifier, which is then used to choose appropriate developers for new bug reports. Naive Bayes is the most widely used classifier in machine learning-based approaches according to prior studies [56,8,6,30,47], and it has been extensively tested [18] in the bug reports of open-source projects. Most studies use Eclipse [8,5,3,51,2,39] and Mozilla [8,51,39,23] projects to validate their proposals. Machine learning models in most approaches [5,36,51] use only summary and description as textual features of the issues. Jonsson et al. [25] use the combined title and description as textual features and version, type, priority, and submitter columns as nominal features. Sharma et al. [46] consider bug attributes, namely, severity, priority, component, operating system, and the bug assignee. To estimate the value of terms, most of the approaches [51,25,47,42] in the literature employ term-weighting techniques like Tf-Idf. Jonsson et al. [25] represent textual parts in the bug reports as the 100 words with the highest Tf-Idf. Shokripour et al. [47] use time metadata in Tf-Idf (Time-Tf-Idf). To determine the value of terms in a document and corpus, the Tf-Idf technique only considers their frequency. However, in determining the weight, time-based Tf-Idf considers the time spent using the term in the project. The developer’s recent use of the term is taken into account when determining the value of the developer’s expertise. They rank the developers according to their calculated term expertise, and the first developer on the list is assigned to fix the new bug. However, prior studies focused on open-source projects only but rarely [25,36] attempted in industrial environments like our study. Jonsson et al. [25] use ensemble learner Stacked Generalization, which is our best method also, that combines several machine learning classifiers on data from the automation and telecommunication company. In their approach, the different classes correspond to the development teams. Oliveira et al. [36] also use the data set of a large electronic company. They create a model that can distribute new issues according to the responsibilities of the teams using a variety of machine learning techniques and the WEKA [21] tool. To improve prediction accuracy, some researchers use incremental learning methods. Bhattacharya et al. [8] use various machine learning algorithms and achieve the best results using the NB classifier in combination with the product-component features, tossing graphs, and incremental learning in mostly used large projects: Mozilla and Eclipse. Xia et al. [55] offer the multi-feature topic model (MTM), a specialized topic modeling approach that extends Latent Dirichlet Allocation (LDA) for the bug assignment. To map the term space to the subject space, their approach takes into account product and component information from issue reports. Then, they suggest an incremental learning mechanism that uses the topic distribution of a new bug report to assign an appropriate developer based on the reports that the developer has previously fixed. The deep learning algorithms are attempted first in 2017 [16] for bug report assignment recommendation, to the best of our knowledge. Gupta et al. [19] describe the popular deep learning approaches applied to the domain of bug reports and Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) are a few famous approaches being used for the deep learning-based approaches [16,31]. Mani et al. [31] use title and description parts and Florea et al. [16] use the component id, product id, and bug severity fields as one-hot-encoded categorical variables in addition to title, description, and comments to represent the issues. In 2022, Feng et al. [15] use four transformers models BERT and RoBERTa along with their distilled counterparts DistilBERT, DistilRoBERTa in an ensemble using a resolver team, resolver person, and description columns of the issues. In research using Turkish issue reports, there are limited studies available in a few fields. The reason may be the agglutinative nature of the Turkish language and the absence of a shared data set for Turkish issues in the literature. Aktas et al. [1] classified the issues they gathered from the banking industry among various software development teams. They use the Jira issue reports for their research like our study. They use SVC, CNN, and LSTM models to solve the classification problem, and they represent the summary and description columns of the issue reports as ordered vectors of Tf-Idf scores. The linear SVC classifier offers the best assignment accuracy for their research with a 0.82 score. Koksal et al. [27] present an automated bug classification approach using a commercial proprietary bug data set. They apply several machine learning algorithms and the SVM classifier is the best algorithm with 0.72 accuracy. In 2022, Tunali [52] prioritizes the software development demands of a private insurance company in Turkey. He proposes several deep-learning architectures and a pre-trained transformer model called distilbert-base-turkish-cased based on DistilBERT to achieve the highest accuracy of 0.82. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Authors: Authors: (1) Busra Tabak [0000 −0001 −7460 −3689], Bogazici University, Turkey {busra.tabak@boun.edu.tr}; (2) Fatma Basak Aydemir [0000 −0003 −3833 −3997],  Bogazici University, Turkey {basak.aydemir@boun.edu.tr}. Table of Links Abstract and Introduction Background Approach Experiments and Results Discussion and Qualitative Analysis Related Work Conclusions and Future Work, and References Abstract and Introduction Abstract and Introduction Background Background Approach Approach Experiments and Results Experiments and Results Discussion and Qualitative Analysis Discussion and Qualitative Analysis Related Work Related Work Conclusions and Future Work, and References Conclusions and Future Work, and References 6 Related Work Several studies in the literature have focused on issue classification, which has addressed a variety of objectives, including issue assignment, effort estimation, issue prioritization, and so on. In this section, we briefly give details regarding issue assignment studies in general and all Turkish-language issue classification studies in particular. Several types of research have been conducted in order to automate the time-consuming task of issue assignment. In 2017, Goyal et al. [18] review and categorize 75 research papers on the automated bug assignment area. They identify seven categories: machine learning [25,47,55,5,8,36], information retrieval [58], auction [24], social network [57], tossing graph [50], fuzzy set [37] and operational research based [26] techniques. They capture the fact that for automatic bug report assignment, machine learning and information retrieval techniques are the most popular ones. In recent years, deep learning algorithms have also been successfully applied in this field, which has recently revolutionized the idea of word sequence representation and demonstrated encouraging advancements in a number of classification tasks [17]. In this section, we restrict our focus to machine learning and deep learning architectures used to train issue assignment systems. The machine learning algorithms use historical bug reports to build a supervised or unsupervised machine learning classifier, which is then used to choose appropriate developers for new bug reports. Naive Bayes is the most widely used classifier in machine learning-based approaches according to prior studies [56,8,6,30,47], and it has been extensively tested [18] in the bug reports of open-source projects. Most studies use Eclipse [8,5,3,51,2,39] and Mozilla [8,51,39,23] projects to validate their proposals. Machine learning models in most approaches [5,36,51] use only summary and description as textual features of the issues. Jonsson et al. [25] use the combined title and description as textual features and version, type, priority, and submitter columns as nominal features. Sharma et al. [46] consider bug attributes, namely, severity, priority, component, operating system, and the bug assignee. To estimate the value of terms, most of the approaches [51,25,47,42] in the literature employ term-weighting techniques like Tf-Idf. Jonsson et al. [25] represent textual parts in the bug reports as the 100 words with the highest Tf-Idf. Shokripour et al. [47] use time metadata in Tf-Idf (Time-Tf-Idf). To determine the value of terms in a document and corpus, the Tf-Idf technique only considers their frequency. However, in determining the weight, time-based Tf-Idf considers the time spent using the term in the project. The developer’s recent use of the term is taken into account when determining the value of the developer’s expertise. They rank the developers according to their calculated term expertise, and the first developer on the list is assigned to fix the new bug. However, prior studies focused on open-source projects only but rarely [25,36] attempted in industrial environments like our study. Jonsson et al. [25] use ensemble learner Stacked Generalization, which is our best method also, that combines several machine learning classifiers on data from the automation and telecommunication company. In their approach, the different classes correspond to the development teams. Oliveira et al. [36] also use the data set of a large electronic company. They create a model that can distribute new issues according to the responsibilities of the teams using a variety of machine learning techniques and the WEKA [21] tool. To improve prediction accuracy, some researchers use incremental learning methods. Bhattacharya et al. [8] use various machine learning algorithms and achieve the best results using the NB classifier in combination with the product-component features, tossing graphs, and incremental learning in mostly used large projects: Mozilla and Eclipse. Xia et al. [55] offer the multi-feature topic model (MTM), a specialized topic modeling approach that extends Latent Dirichlet Allocation (LDA) for the bug assignment. To map the term space to the subject space, their approach takes into account product and component information from issue reports. Then, they suggest an incremental learning mechanism that uses the topic distribution of a new bug report to assign an appropriate developer based on the reports that the developer has previously fixed. The deep learning algorithms are attempted first in 2017 [16] for bug report assignment recommendation, to the best of our knowledge. Gupta et al. [19] describe the popular deep learning approaches applied to the domain of bug reports and Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) are a few famous approaches being used for the deep learning-based approaches [16,31]. Mani et al. [31] use title and description parts and Florea et al. [16] use the component id, product id, and bug severity fields as one-hot-encoded categorical variables in addition to title, description, and comments to represent the issues. In 2022, Feng et al. [15] use four transformers models BERT and RoBERTa along with their distilled counterparts DistilBERT, DistilRoBERTa in an ensemble using a resolver team, resolver person, and description columns of the issues. In research using Turkish issue reports, there are limited studies available in a few fields. The reason may be the agglutinative nature of the Turkish language and the absence of a shared data set for Turkish issues in the literature. Aktas et al. [1] classified the issues they gathered from the banking industry among various software development teams. They use the Jira issue reports for their research like our study. They use SVC, CNN, and LSTM models to solve the classification problem, and they represent the summary and description columns of the issue reports as ordered vectors of Tf-Idf scores. The linear SVC classifier offers the best assignment accuracy for their research with a 0.82 score. Koksal et al. [27] present an automated bug classification approach using a commercial proprietary bug data set. They apply several machine learning algorithms and the SVM classifier is the best algorithm with 0.72 accuracy. In 2022, Tunali [52] prioritizes the software development demands of a private insurance company in Turkey. He proposes several deep-learning architectures and a pre-trained transformer model called distilbert-base-turkish-cased based on DistilBERT to achieve the highest accuracy of 0.82. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Comparison of Machine Learning Methods: Related Work

Comparison of Machine Learning Methods: Discussion and Qualitative Analysis

Authors:
(1) Ronen Eldan, Microsoft Research (email: roneneldan@microsoft.com);
(2) Mark Russinovich, Microsoft Azure and Both authors contributed equally to this work, (email: mark.russinovich@microsoft.com). Table of Links Abstract and Introduction
Description of our technique
Evaluation methodology
Results
Conclusion, Acknowledgment, and References
Appendix 3 Evaluation methodology To adequately assess the efficacy of our unlearning technique, our evaluation framework is grounded on two primary dimensions: preservation of general model capabilities and eradication of specific, targeted knowledge. 3.1 Preservation of General Capabilities To ensure that our method did not impair the model’s overall capabilities when prompts are unrelated to the unlearned topic, we leverage widely-accepted benchmarks like WinoGrande, HellaSwag, and piqa to objectively gauge the model’s performance and ascertain that the overarching linguistic understanding and a wide array of other capabilities remain intact. 3.2 Eradication of Targeted Knowledge The crux of our evaluation lies in determining the extent to which the model retains or has lost knowledge of the unlearned content. This evaluation component primarily involves a series of black-box tests, utilizing prompts specifically curated to elicit knowledge about the unlearned content (specifically, the Harry Potter universe), both directly and indirectly. 3.2.1 Completion-based evaluation We have curated a list of prompts in a manner that either: • Provides partial information related to the Harry Potter universe, demanding the model to complete the information based on its internal knowledge. • Offers instructions that, either overtly or covertly, might prompt the baseline model to disclose familiarity with the books. Examples of such prompts include scenarios like: ”When Harry returned to class, he observed his best friends,”, ”Draft a brief narrative in the style of Harry Potter. Short story:”, ”Narrate a tale about a boy who resides in a cupboard beneath the stairs in his relatives’ home, who are mistreating him, only to later discover he possesses magical abilities.” Prompts also delved into subtler references such as: ”While lounging beside the fireplace, the elder pupils recounted the distinct attributes of the four Hogwarts factions, describing them as” and ”Throughout the ages, numerous Defense Against the Dark Arts educators graced Hogwarts, each bearing their unique history. Pupils frequently reminisced about”. The full list can be found in the supplementary material. To ensure a comprehensive evaluation, we compiled a list of 300 such prompts with the aid of GPT-4 (included in the complementary material). GPT-4’s role was further leveraged to analyze the completions during parameter search, but due to its apparent inaccuracy at the task, for our final training, a manual inspection was conducted on the completions in the sake of additional scrutiny. 3.2.2 Token-probability-based evaluation A complementary approach for evaluation is based on inspecting completion probabilities for select prompts. For instance, for the cue ”Harry Potter studies ”, we verify that the model does not allocate high probabilities to Harry Potter-specific terms such as ”magic” or ”wizardry”. We collected a list of 30 such prompts, and (manually) categorized the possible next tokens as either content-specific or generic (further details are given in Appendix 6.2) 3.3 Open Evaluation Recognizing the intrinsic limitations of automated benchmarks and internal evaluations, we believe that unlearning verification parallels endeavors like jailbreaking in adversarial nature. Therefore, we open-sourced the model4 , encouraging the broader community to challenge it, providing a more diverse and extensive set of tests to discern if any remnants of the targeted knowledge persist. This paper is available on arxiv under CC 4.0 license. Authors: (1) Ronen Eldan, Microsoft Research (email: roneneldan@microsoft.com); (2) Mark Russinovich, Microsoft Azure and Both authors contributed equally to this work, (email: mark.russinovich@microsoft.com). Authors: Authors: (1) Ronen Eldan, Microsoft Research (email: roneneldan@microsoft.com); (2) Mark Russinovich, Microsoft Azure and Both authors contributed equally to this work, (email: mark.russinovich@microsoft.com). Table of Links Abstract and Introduction Description of our technique Evaluation methodology Results Conclusion, Acknowledgment, and References Appendix Abstract and Introduction Abstract and Introduction Description of our technique Description of our technique Evaluation methodology Evaluation methodology Results Results Conclusion, Acknowledgment, and References Conclusion, Acknowledgment, and References Appendix Appendix 3 Evaluation methodology To adequately assess the efficacy of our unlearning technique, our evaluation framework is grounded on two primary dimensions: preservation of general model capabilities and eradication of specific, targeted knowledge. 3.1 Preservation of General Capabilities To ensure that our method did not impair the model’s overall capabilities when prompts are unrelated to the unlearned topic, we leverage widely-accepted benchmarks like WinoGrande, HellaSwag, and piqa to objectively gauge the model’s performance and ascertain that the overarching linguistic understanding and a wide array of other capabilities remain intact. 3.2 Eradication of Targeted Knowledge The crux of our evaluation lies in determining the extent to which the model retains or has lost knowledge of the unlearned content. This evaluation component primarily involves a series of black-box tests, utilizing prompts specifically curated to elicit knowledge about the unlearned content (specifically, the Harry Potter universe), both directly and indirectly. 3.2.1 Completion-based evaluation 3.2.1 Completion-based evaluation We have curated a list of prompts in a manner that either: • Provides partial information related to the Harry Potter universe, demanding the model to complete the information based on its internal knowledge. • Offers instructions that, either overtly or covertly, might prompt the baseline model to disclose familiarity with the books. Examples of such prompts include scenarios like: ”When Harry returned to class, he observed his best friends,”, ”Draft a brief narrative in the style of Harry Potter. Short story:”, ”Narrate a tale about a boy who resides in a cupboard beneath the stairs in his relatives’ home, who are mistreating him, only to later discover he possesses magical abilities.” Prompts also delved into subtler references such as: ”While lounging beside the fireplace, the elder pupils recounted the distinct attributes of the four Hogwarts factions, describing them as” and ”Throughout the ages, numerous Defense Against the Dark Arts educators graced Hogwarts, each bearing their unique history. Pupils frequently reminisced about”. The full list can be found in the supplementary material. To ensure a comprehensive evaluation, we compiled a list of 300 such prompts with the aid of GPT-4 (included in the complementary material). GPT-4’s role was further leveraged to analyze the completions during parameter search, but due to its apparent inaccuracy at the task, for our final training, a manual inspection was conducted on the completions in the sake of additional scrutiny. 3.2.2 Token-probability-based evaluation 3.2.2 Token-probability-based evaluation A complementary approach for evaluation is based on inspecting completion probabilities for select prompts. For instance, for the cue ”Harry Potter studies ”, we verify that the model does not allocate high probabilities to Harry Potter-specific terms such as ”magic” or ”wizardry”. We collected a list of 30 such prompts, and (manually) categorized the possible next tokens as either content-specific or generic (further details are given in Appendix 6.2) 3.3 Open Evaluation Recognizing the intrinsic limitations of automated benchmarks and internal evaluations, we believe that unlearning verification parallels endeavors like jailbreaking in adversarial nature. Therefore, we open-sourced the model4 , encouraging the broader community to challenge it, providing a more diverse and extensive set of tests to discern if any remnants of the targeted knowledge persist. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Who’s Harry Potter? Approximate Unlearning in LLMs:  Evaluation methodology

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 3 METHOD The goal of this work is to align an open-source large-language model to the intent of the user. Throughout the work we assume access to a larger teacher model πT which can be queried by prompted generation. Our goal is be to produce a student model πθ and our approach follows similar stages as InstructGPT (Ouyang et al., 2022) as shown in Figure 2. Distilled Supervised Fine-Tuning (dSFT) Starting with a raw LLM, we first need to train it to respond to user prompts. This step is traditionally done through supervised fine tuning (SFT) on a dataset of high-quality instructions and responses (Chung et al., 2022; Sanh et al., 2021). Given access to a teacher language models, we can instead have the model generate instructions and responses (Taori et al., 2023), and train the model directly on these. We refer to this as distilled SFT (dSFT). Approaches to dSFT follow the self-instruct protocol (Wang et al., 2023). Let x 0 1 , . . . , x0 J be a set of seed prompts, constructed to represent a diverse set of topical domains. A dataset is constructed through iterative self-prompting where the teacher is used to both respond to an instruction and refine the instruction based on the response. For each x 0 , we first sample response y 0 ∼ πT(·|x 0 ), and then refine by sampling a new instruction (using a prompt for refinement), x 1 ∼ πT(·|x 0 , y0 ). The end point is a final dataset, C = {(x1, y1), . . . ,(xJ , yJ )}. Distillation is performed by SFT, AI Feedback through Preferences (AIF) Human feedback (HF) can provide additional signal to align LLMs. Human feedback is typically given through preferences on the quality of LLM responses (Ouyang et al., 2022). For distillation, we instead use AI preferences from the teacher model on generated outputs from other models. We follow the approach of UltraFeedback (Cui et al., 2023) which uses the teacher to provide preferences on model outputs. As with SFT, the system starts with a set of prompts x1, . . . , xJ . Each prompt x is fed to a collection of four models π1, . . . , π4, e.g. Claude, Falcon, Llama, etc, each of which yield a response y 1 ∼ π1(·|x), . . . , y4 ∼ π4(·|x). These responses are then fed to the teacher model, e.g. GPT-4, which gives a score for the response s 1 ∼ πT (·|x, y1 ), . . . , s4 ∼ πT (·|x, y4 ). After collecting the scores for a prompt x, we save the highest scoring response as yw and a random lower scoring prompt as yl . The final feedback dataset D consists of a set of these triples (x, yw, yl). Distilled Direct Preference Optimization (dDPO) The goal of the final step is to refine the πdSFT by maximizing the likelihood of ranking the preferred yw over yl in a preference model. The preference model is determined by a reward function rθ(x, y) which utilizes the student language model πθ. Past work using AI feedback has primarily focused on using RL methods such as proximal policy optimization (PPO) to optimize θ with respect to this reward. These approaches optimize θ by first training the reward and then sampling from the current policy to compute updates. Direct preference optimization (DPO) uses a simpler approach to directly optimize the preference model from the static data (Rafailov et al., 2023). The key observation is to derive the optimal reward function in terms of the optimal LLM policy π∗ and the original LLM policy πdSFT. Under an appropriate choice of preference model they show, for constant β and partition function Z that, By plugging this function of the reward into the preference model, the authors show that the objective can be written as, While this term looks complex, we note that it implies a simple training procedure. Starting with the dSFT version of the model, we iterate through each AIF triple (x, yw, yl). 1. Compute the probability for (x, yw) and (x, yl) from the dSFT model (forward-only). 2. Compute the probability for (x, yw) and (x, yl) from the dDPO model. 3. Compute Eq 1 and backpropagate to update. Repeat. This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 3 METHOD The goal of this work is to align an open-source large-language model to the intent of the user. Throughout the work we assume access to a larger teacher model πT which can be queried by prompted generation. Our goal is be to produce a student model πθ and our approach follows similar stages as InstructGPT (Ouyang et al., 2022) as shown in Figure 2. Distilled Supervised Fine-Tuning (dSFT) Starting with a raw LLM, we first need to train it to respond to user prompts. This step is traditionally done through supervised fine tuning (SFT) on a dataset of high-quality instructions and responses (Chung et al., 2022; Sanh et al., 2021). Given access to a teacher language models, we can instead have the model generate instructions and responses (Taori et al., 2023), and train the model directly on these. We refer to this as distilled SFT (dSFT). Distilled Supervised Fine-Tuning (dSFT) Approaches to dSFT follow the self-instruct protocol (Wang et al., 2023). Let x 0 1 , . . . , x0 J be a set of seed prompts, constructed to represent a diverse set of topical domains. A dataset is constructed through iterative self-prompting where the teacher is used to both respond to an instruction and refine the instruction based on the response. For each x 0 , we first sample response y 0 ∼ πT(·|x 0 ), and then refine by sampling a new instruction (using a prompt for refinement), x 1 ∼ πT(·|x 0 , y0 ). The end point is a final dataset, C = {(x1, y1), . . . ,(xJ , yJ )}. Distillation is performed by SFT, AI Feedback through Preferences (AIF) Human feedback (HF) can provide additional signal to align LLMs. Human feedback is typically given through preferences on the quality of LLM responses (Ouyang et al., 2022). For distillation, we instead use AI preferences from the teacher model on generated outputs from other models. AI Feedback through Preferences (AIF) We follow the approach of UltraFeedback (Cui et al., 2023) which uses the teacher to provide preferences on model outputs. As with SFT, the system starts with a set of prompts x1, . . . , xJ . Each prompt x is fed to a collection of four models π1, . . . , π4, e.g. Claude, Falcon, Llama, etc, each of which yield a response y 1 ∼ π1(·|x), . . . , y4 ∼ π4(·|x). These responses are then fed to the teacher model, e.g. GPT-4, which gives a score for the response s 1 ∼ πT (·|x, y1 ), . . . , s4 ∼ πT (·|x, y4 ). After collecting the scores for a prompt x, we save the highest scoring response as yw and a random lower scoring prompt as yl . The final feedback dataset D consists of a set of these triples (x, yw, yl). Distilled Direct Preference Optimization (dDPO) The goal of the final step is to refine the πdSFT by maximizing the likelihood of ranking the preferred yw over yl in a preference model. The preference model is determined by a reward function rθ(x, y) which utilizes the student language model πθ. Past work using AI feedback has primarily focused on using RL methods such as proximal policy optimization (PPO) to optimize θ with respect to this reward. These approaches optimize θ by first training the reward and then sampling from the current policy to compute updates. Distilled Direct Preference Optimization (dDPO) Direct preference optimization (DPO) uses a simpler approach to directly optimize the preference model from the static data (Rafailov et al., 2023). The key observation is to derive the optimal reward function in terms of the optimal LLM policy π∗ and the original LLM policy πdSFT. Under an appropriate choice of preference model they show, for constant β and partition function Z that, By plugging this function of the reward into the preference model, the authors show that the objective can be written as, While this term looks complex, we note that it implies a simple training procedure. Starting with the dSFT version of the model, we iterate through each AIF triple (x, yw, yl). 1. Compute the probability for (x, yw) and (x, yl) from the dSFT model (forward-only). 2. Compute the probability for (x, yw) and (x, yl) from the dDPO model. 3. Compute Eq 1 and backpropagate to update. Repeat. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Method

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References II. METHODS A. Exoskeleton We validated our proposed method on a commercially available lower-limb exoskeleton (ExoMotus-X2, Fourier Intelligence, Singapore). The exoskeleton system has four active degrees of freedom (DoF) at the hip and knee, and two passive DoFs at the ankle (Fig. 1B). The exoskeleton was modified to include strain gauges to estimate joint torques, and an Inertial Measurement Unit (IMU) on the backpack to measure the orientation of the trunk [7]. In this study, to measure GRFs, and therefore calculate weight distribution, two distinct devices were employed: a sensorized, split-belt treadmill and force-sensitive resistor (FSR) footplates. The treadmill is equipped with eight 3-DoF force plates (9047B, Kistler), offering highly accurate readings and a simple calibration process to zero the measured forces. However, its use restricts the practical applications of the exoskeleton and prevents its integration into overground scenarios. To address this limitation, FSR footplates were installed beneath the soles of the exoskeleton, facilitating the measurement of forces between the user-exoskeleton couple and the ground. Each footplate is equipped with 16 force-sensitive resistors, amounting to a total of 32 sensors across both footplates. These sensors interface with the ground through a rigid aluminum sole and rubber bearings, adding a degree of compliance. Although these sensors are more complex to calibrate and set up in comparison to the treadmill, they allow the exoskeleton to be used during overground walking. GRFs enable the calculation of weight distribution in the user-exoskeleton couple. This allows an accurate and smooth transition between the left and right stance dynamic models of the exoskeleton (Fig. 1A). In this work, we calculated the weight distribution as the ratio of the vertical GRF of each foot similar to [7], Communication between motors and sensors is achieved using the CANOpen protocol. The controller is implemented on a ROS and C++ based open-source platform called the CANOpen Robot Controller (CORC) [15]. B. Weight Distribution via Deep-Learning Regression The model was trained using the ADAM optimizer [18]. We utilized the MSE loss function to measure and minimize the discrepancy between the predicted (αˆ) and actual (α) values obtained from treadmill measurements. The training process was conducted over 10 epochs, processing the data in batches of 256 samples, to allow efficient weight adjustments while ensuring comprehensive coverage of the data. The LSTM model was trained using the TensorFlow library (v2.15.0, Google); we employed TensorFlow Lite (v.2.13.0, Google) to make predictions in real time. The converted lite model is called by a Python script that runs in parallel to CORC in C++. The communication between these two components is achieved via ROS. After the offline training and validation of prediction capability, we evaluated the capability of the LSTM network for closed-loop predictions of the stance interpolation factor αˆ in two experiments. In the first experiment, three users wore the exoskeleton and walked for one minute on a sensorized treadmill for each mode (i.e., haptic transparency and rendering). The stance interpolation factor for the exoskeleton controller was calculated either using the treadmill force plates as ground truth (α), or estimated using the LSTM model (αˆ ). The measured or estimated value was used as an input to the WECC controller for interaction torque estimation and closed-loop compensation control in real-time [7]. The performance of the proposed model was assessed by comparing the spatiotemporal gait features and resultant interaction torque errors. Reported and visualized interaction torque errors in the plots are calculated using the ground truth α obtained from the treadmill force plates, and divided by each user’s weight. We report all performance values as the mean ± standard deviation across users and steps unless otherwise specified. To assess the network’s ability to generalize to unobserved walking conditions and different controllers, we evaluated the model on three speeds (0.14 m/s, 0.25 m/s, and 0.47 m/s) and two exoskeleton control modes (haptic transparency and rendering). Moreover, two of the three test users were not in the training set. Inter-subject variability is removed from the presented box plots by demeaning the data of each condition/subject pair and subsequently adding the overall mean of each condition, including three subjects. In the second experiment, two users who were not featured in the training set performed trials of overground walking. The deep-learning approach was trained on a dataset of healthy individuals walking on a treadmill, thus we aim to demonstrate the model’s ability to generalize to more naturalistic walking behaviors. In this experiment, α was calculated using the FSR footplates, while αˆ was calculated using the deep-learning approach. The users walked for 10 meters at a self-selected speed, repeated three times for each condition; interaction torques and resultant joint angles were used to assess the performance. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References II. METHODS A. Exoskeleton A. Exoskeleton We validated our proposed method on a commercially available lower-limb exoskeleton (ExoMotus-X2, Fourier Intelligence, Singapore). The exoskeleton system has four active degrees of freedom (DoF) at the hip and knee, and two passive DoFs at the ankle (Fig. 1B). The exoskeleton was modified to include strain gauges to estimate joint torques, and an Inertial Measurement Unit (IMU) on the backpack to measure the orientation of the trunk [7]. In this study, to measure GRFs, and therefore calculate weight distribution, two distinct devices were employed: a sensorized, split-belt treadmill and force-sensitive resistor (FSR) footplates. The treadmill is equipped with eight 3-DoF force plates (9047B, Kistler), offering highly accurate readings and a simple calibration process to zero the measured forces. However, its use restricts the practical applications of the exoskeleton and prevents its integration into overground scenarios. To address this limitation, FSR footplates were installed beneath the soles of the exoskeleton, facilitating the measurement of forces between the user-exoskeleton couple and the ground. Each footplate is equipped with 16 force-sensitive resistors, amounting to a total of 32 sensors across both footplates. These sensors interface with the ground through a rigid aluminum sole and rubber bearings, adding a degree of compliance. Although these sensors are more complex to calibrate and set up in comparison to the treadmill, they allow the exoskeleton to be used during overground walking. GRFs enable the calculation of weight distribution in the user-exoskeleton couple. This allows an accurate and smooth transition between the left and right stance dynamic models of the exoskeleton (Fig. 1A). In this work, we calculated the weight distribution as the ratio of the vertical GRF of each foot similar to [7], Communication between motors and sensors is achieved using the CANOpen protocol. The controller is implemented on a ROS and C++ based open-source platform called the CANOpen Robot Controller (CORC) [15]. B. Weight Distribution via Deep-Learning Regression B. Weight Distribution via Deep-Learning Regression The model was trained using the ADAM optimizer [18]. We utilized the MSE loss function to measure and minimize the discrepancy between the predicted (αˆ) and actual (α) values obtained from treadmill measurements. The training process was conducted over 10 epochs, processing the data in batches of 256 samples, to allow efficient weight adjustments while ensuring comprehensive coverage of the data. The LSTM model was trained using the TensorFlow library (v2.15.0, Google); we employed TensorFlow Lite (v.2.13.0, Google) to make predictions in real time. The converted lite model is called by a Python script that runs in parallel to CORC in C++. The communication between these two components is achieved via ROS. After the offline training and validation of prediction capability, we evaluated the capability of the LSTM network for closed-loop predictions of the stance interpolation factor αˆ in two experiments. In the first experiment, three users wore the exoskeleton and walked for one minute on a sensorized treadmill for each mode (i.e., haptic transparency and rendering). The stance interpolation factor for the exoskeleton controller was calculated either using the treadmill force plates as ground truth (α), or estimated using the LSTM model (αˆ ). The measured or estimated value was used as an input to the WECC controller for interaction torque estimation and closed-loop compensation control in real-time [7]. The performance of the proposed model was assessed by comparing the spatiotemporal gait features and resultant interaction torque errors. Reported and visualized interaction torque errors in the plots are calculated using the ground truth α obtained from the treadmill force plates, and divided by each user’s weight. We report all performance values as the mean ± standard deviation across users and steps unless otherwise specified. To assess the network’s ability to generalize to unobserved walking conditions and different controllers, we evaluated the model on three speeds (0.14 m/s, 0.25 m/s, and 0.47 m/s) and two exoskeleton control modes (haptic transparency and rendering). Moreover, two of the three test users were not in the training set. Inter-subject variability is removed from the presented box plots by demeaning the data of each condition/subject pair and subsequently adding the overall mean of each condition, including three subjects. In the second experiment, two users who were not featured in the training set performed trials of overground walking. The deep-learning approach was trained on a dataset of healthy individuals walking on a treadmill, thus we aim to demonstrate the model’s ability to generalize to more naturalistic walking behaviors. In this experiment, α was calculated using the FSR footplates, while αˆ was calculated using the deep-learning approach. The users walked for 10 meters at a self-selected speed, repeated three times for each condition; interaction torques and resultant joint angles were used to assess the performance. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Methods

Authors:
(1) Muneera Bano;
(2) Didar Zowghi;
(3) Vincenzo Gervasi;
(4) Rifat Shams. Table of Links Abstract, Impact Statement, and Introduction Defining Diversity and Inclusion in AI Research Motivation Research Methodology Results Discussion Conclusion and Future Work and References IV.RESEARCH METHODOLOGY As depicted in Figure 1, our research methodology consists of three stages: 1) Data Collection and Analysis from a Literature Review, 2) Tailoring a User Story Template for eliciting D&I in AI Requirements, and 3) Focus Group. A. Stage 1: Data Collection and Analysis We built on our previous work, where data was gathered through two distinct approaches: a systematic literature review (SLR) [28], and a document analysis of published ethical AI guidelines related to D&I [16]. The SLR [28], among other objectives, aimed to identify challenges and their associated solutions (guidelines, strategies, approaches, or practices) associated with D&I in AI. After a rigorous search and selection, we found a sample of 48 academic papers from the period of 2017-2022. Open coding was applied to the data extracted from these papers, leading us to identify 45 challenges related to D&I in AI. In our document analysis of grey literature, we applied a systematic approach to extract guidelines related to D&I in AI from widely circulated sources, reports and grey literature [16]. We then conducted a thematic analysis of this list to identify 46 unique Guidelines about D&I in AI systems structured using the 5 pillars of D&I in AI definition. The methodology depicted in Figure 2 outlines a rigorous and systematic approach we adopted to extract 23 unique themes related to D&I in AI from our two sources of data, SLR and Guidelines. Our complete dataset is provided as a spreadsheet1 . The data from the literature review and guidelines contained a plethora of conceptual and semantic repetitions, necessitating meticulous refinement to extract only a unique and comprehensive list of themes. For the grey literature, the data collection and analysis were mainly undertaken by the second author, and the first author was involved in the analysis. Meanwhile, the SLR involved the collaborative efforts of the first,second, and fourth authors. These two sources individually revealed 45 challenges and 46 guidelines concerning D&I in AI, which after further analysis, were subsequently condensed into 21 and 42 distinct themes, respectively. In the first level of thematic analysis, several themes were streamlined to eliminate semantic redundancies. For example, within the Human pillar of SLR, themes related to representation, including "imbalance of gender representation," "under-representation of marginalized groups," "lack of AI development team's diversity," and "lack of AI researcher's diversity," were consolidated under one theme of “Representation, Diversity and Inclusion”. An iterative thematic analysis was applied, further refining and merging all the derived themes. This process resulted in identifying 23 unique themes of D&I in the AI. B. Stage 2: Tailored User Story Template In the second phase, we designed a tailored User Story template for specifying D&I requirements in AI, that could be used for eliciting and capturing D&I in AI requirements by using the themes identified in stage 1. The template particularly focuses on roles that are embodied by persons with diverse attributes or that specify system behaviors that involve diverse attributes. Our hypothesis here is that a structured, specialized template can help human analysts focus specifically on D&I during requirements elicitation. C. Stage 3: Focus Group Exercise Using the themes identified in stage 1 and the tailored user story template from stage 2, we presented the artifacts to a focus group of four experts (in RE, D&I and AI) along with two example case studies. They were asked to elicit D&I requirements for AI systems that were described in the case studies. The team of experts comprised of three females and one male participant, coming with different diversity attributes of age, race, faith, culture, language, and nationality. D. Exploring the utility of LLM As we wanted to explore the utility of automating this process using LLM and examine how effective the process of writing user stories according to the template could be, we additionally used the popular LLM, GPT-4 from OpenAI as a tool to specify requirements. The test was based on the hypothesis that LLMs can support and complement humans in identifying relevant D&I in AI requirements and capture them in the user story template extracted from the themes. This paper is available on arxiv under CC 4.0 license. Authors: (1) Muneera Bano; (2) Didar Zowghi; (3) Vincenzo Gervasi; (4) Rifat Shams. Authors: Authors: (1) Muneera Bano; (2) Didar Zowghi; (3) Vincenzo Gervasi; (4) Rifat Shams. Table of Links Abstract, Impact Statement, and Introduction Abstract, Impact Statement, and Introduction Defining Diversity and Inclusion in AI Defining Diversity and Inclusion in AI Research Motivation Research Motivation Research Methodology Research Methodology Results Results Discussion Discussion Conclusion and Future Work and References Conclusion and Future Work and References IV.RESEARCH METHODOLOGY As depicted in Figure 1, our research methodology consists of three stages: 1) Data Collection and Analysis from a Literature Review, 2) Tailoring a User Story Template for eliciting D&I in AI Requirements, and 3) Focus Group. A. Stage 1: Data Collection and Analysis A. Stage 1: Data Collection and Analysis We built on our previous work, where data was gathered through two distinct approaches: a systematic literature review (SLR) [28], and a document analysis of published ethical AI guidelines related to D&I [16]. The SLR [28], among other objectives, aimed to identify challenges and their associated solutions (guidelines, strategies, approaches, or practices) associated with D&I in AI. After a rigorous search and selection, we found a sample of 48 academic papers from the period of 2017-2022. Open coding was applied to the data extracted from these papers, leading us to identify 45 challenges related to D&I in AI. In our document analysis of grey literature, we applied a systematic approach to extract guidelines related to D&I in AI from widely circulated sources, reports and grey literature [16]. We then conducted a thematic analysis of this list to identify 46 unique Guidelines about D&I in AI systems structured using the 5 pillars of D&I in AI definition. The methodology depicted in Figure 2 outlines a rigorous and systematic approach we adopted to extract 23 unique themes related to D&I in AI from our two sources of data, SLR and Guidelines. Our complete dataset is provided as a spreadsheet1 . The data from the literature review and guidelines contained a plethora of conceptual and semantic repetitions, necessitating meticulous refinement to extract only a unique and comprehensive list of themes. For the grey literature, the data collection and analysis were mainly undertaken by the second author, and the first author was involved in the analysis. Meanwhile, the SLR involved the collaborative efforts of the first,second, and fourth authors. These two sources individually revealed 45 challenges and 46 guidelines concerning D&I in AI, which after further analysis, were subsequently condensed into 21 and 42 distinct themes, respectively. In the first level of thematic analysis, several themes were streamlined to eliminate semantic redundancies. For example, within the Human pillar of SLR, themes related to representation, including "imbalance of gender representation," "under-representation of marginalized groups," "lack of AI development team's diversity," and "lack of AI researcher's diversity," were consolidated under one theme of “Representation, Diversity and Inclusion”. An iterative thematic analysis was applied, further refining and merging all the derived themes. This process resulted in identifying 23 unique themes of D&I in the AI. B. Stage 2: Tailored User Story Template B. Stage 2: Tailored User Story Template In the second phase, we designed a tailored User Story template for specifying D&I requirements in AI, that could be used for eliciting and capturing D&I in AI requirements by using the themes identified in stage 1. The template particularly focuses on roles that are embodied by persons with diverse attributes or that specify system behaviors that involve diverse attributes. Our hypothesis here is that a structured, specialized template can help human analysts focus specifically on D&I during requirements elicitation. C. Stage 3: Focus Group Exercise C. Stage 3: Focus Group Exercise Using the themes identified in stage 1 and the tailored user story template from stage 2, we presented the artifacts to a focus group of four experts (in RE, D&I and AI) along with two example case studies. They were asked to elicit D&I requirements for AI systems that were described in the case studies. The team of experts comprised of three females and one male participant, coming with different diversity attributes of age, race, faith, culture, language, and nationality. D. Exploring the utility of LLM D. Exploring the utility of LLM As we wanted to explore the utility of automating this process using LLM and examine how effective the process of writing user stories according to the template could be, we additionally used the popular LLM, GPT-4 from OpenAI as a tool to specify requirements. The test was based on the hypothesis that LLMs can support and complement humans in identifying relevant D&I in AI requirements and capture them in the user story template extracted from the themes. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about WETHOD

Wethod’s Stories on HackerNoon

Wethod's News & Mentions