AI-powered music creation&licensing platform

Mubert Inc

Meet the Winners of Startups in North and Central America

88 Stories To Learn About Music

The role of music for business and user experience as well as ways to make the best of it, discussed by me, Alex Kochetkov, founder and CEO of Mubert , the first generative music streaming service. The impact of music in the life of human beings cannot be overrated. People would rather do without sports, newspapers, movies, internet and smartphones. But music is important. When we listen to masterpieces of Debussy or Wagner it is truly astounding how such beauty can be created. The essence of talent is a mystery. We all have been to live shows at least once, and this feeling evoked by music is unexplainable. According to numerous scientific studies, music makes people think of themselves, distract from the outside world and forget about reality, calm down and acquire additional meaning to life. It is all about subconsciousness. Music is magic and music has power. Prominent businesses leverage this knowledge and it bears fruit. “Music gives a soul to the universe, wings to the mind, flight to the imagination and life to everything.” ― Plato. I think the great Athenian philosopher will not mind if I add “music gives profit to business owners”. In a world where companies fight for unparalleled user experience, sound appears to be a decisive competitive differentiator. The secret is in the ‘right’ music being played. Again, the relevance matters a lot when we speak of personalized UX. Customers will leave public places in the case of the music not suiting their brand identity. On the contrary, they are more likely to stay longer, come back and buy more if they like what they hear. The trick is to find the balance and satisfy both brand profile and customer tastes. There is more to it. Customers are concerned about the moral aspects of music accompaniment in public places. They want things to be right, and musicians should be appreciated and paid for their music being used. And, if people find out businesses act unethically and illegally, their attitude towards a business will change to negative. A watershed between companies and their clientele implies a conflict of interests. While customers want justice, every business owner cares about cutting costs. Paying royalties is a hit to the wallet. Streaming services are popular, however, they cannot provide a solution to cover all needs at once. The development of music recommender systems or MRS is the subproduct of streaming services which have mushroomed in recent years. Thanks to services such as Apple Music, Spotify, and Pandora,  people acquire access to immense catalogs. In the old days it took quite an effort to find the music you want. Today, millions of tracks and artists can be found in a matter of seconds. People hate having multiple choices. In this respect, MRS are used to filter and narrow down offered music based on the specific preferences of the user. Basically, highly relevant recommendations are the exact feature listeners want to see in those online streaming services. Though, they have a number of serious flaws. Music recommender systems: far from perfect MRS vulnerability boils down to one thing: they fail to take into account a vast number of factors that influence decision-making in music. Current recommendation systems waltz around the user-item interaction approach and descriptors based on content. In reality, music choices at a specific point in time are driven by internal and external factors as well as context. Emotional state, plus weather conditions, plus social surrounding form this particular narrow  halfway we want to walk through here and now. It is does not mean we will necessarily go back there again. Nowadays, MRS are unable to provide relevant recommendations for newcomers; the ‘cold start’ issue as we know it — not enough data. Also, when we manage our playlists, it’s not the order of tracks that matters: we pay attention to the entire ensemble and the transition from one track to another. Adding one more track to the existing playlist, so that it fits perfectly, is another challenge. Recommender systems, in fact, represent one side of the problem with streaming services. Streaming services: the big picture The debate on the sustainability of streaming services revolve around money and ethics. To what extent are these platforms transparent, fair in terms of artists recognition? Record labels have been holding a grudge for a long time. In the case of Spotify, the platform did not hesitate to make deals with artists directly cutting out the record labels — this is one of the practices that questions morality. Armies of paying subscribers across the globe do not seem to have saved the day. Despite its growing customer base Spotify, for instance, used to be unprofitable for years since the day of its inception. The price of subscription is one of the major motivators for the illegal consumption of music. Streaming services — the largest of them — may be different in design; they provide the same catalog with approximately 300 million tracks. Here we cannot talk about differentiation as compared to video streaming platforms that have original content. Embarrassing news on data security appeared at the beginning of the year. There has been a lot of speculation on how Spotify : there is a lot happening in terms of online marketing and data sharing you would never think of. The giants of the online streaming market are the ones that are targeted by constant GDPR complaints. They refer to user rights structural violations: streaming platforms have automated response systems for access requests. Users do not get sufficient data, for instance, who their data is shared with. In addition, little protection is provided to user privacy. sells user data You have to face it: the future is here. Artificial intelligence changes companies, businesses, industries; it has changed the way we interact with the world. At first, AI approached the area of figures and computation. When neural networks entered the world of art painters, musicians, and artists all over the world started getting nervous. In the context of art and AI there are a lot of debates on what makes music human, whether there is anything an algorithm cannot learn. The big question is — will AI replace or help? Let’s put the alarmist paradigm aside and look in the direction of the opportunities. How music met AI From a business perspective, sound is an important element of the UX. By providing personalized experiences through music, a company creates its own unique atmosphere and differentiates itself. At the same time, businesses in horeca domain are interested in functional music: it’s not meant to distract by vivid emotional coloring. French restaurant clients, for instance, would hardly enjoy their glass of wine listening to Bjork or Puccini’s Tosca. So, functional music must provide a light background, a supplement to an environment. Driven by this idea, we elaborated the framework to tackle all the existing challenges and provide an on-demand solution. Mubert AI-based generative music streaming services platform, kills two, or even three birds with one stone. The service broadcasts non-stop music via API; is capable of giving businesses exactly what they want —personalization, cost-cutting, differentiation, unique UX for their customers. Algorithms write music in real-time mode: they compile never-ending streams of music that do not need to be managed. AI accesses a large database of patterns written by human composers. Neuronets fulfil the task of conductor: it takes into account the current circumstances and manages tiny musical pieces accordingly. Mubert’s innovative approach comes with ease of use. The platform’s technology does not have all stereotypical functions that other streaming services have — the appropriate streams are in one-click proximity. Moreover, the platform is adaptive to external commands or device sensors which makes it a suitable solution for smart homes and voice assistants. Ultimately, the service provides businesses with copyright protection. Mubert addresses legal issues via copyright-free music — no contracts, no royalties. What separates a business from personalized music accompaniment? Only the purchase of subscription. That’s it. The flexibility of generative music has been successfully tested in a number of industries, and gaming is one of them. Historically, gaming streamers were banned from services for  illegally using music in the background. The problem with copyright holders is not new, and , with its 1 million monthly users, reaped the benefits of generative music. The partnership resulted in around 66 thousand unique streamers creating approximately 780 hours of unique content within the first month. Omlet Arcade Personalized user experience literally means getting personal with customers: their lives are much more than content, they go far beyond that. The audience is sophisticated. Everyone needs to feel privileged, special, and have a sense of being considered important. Music appeals both emotionally and intangibly, which further pays off with rather tangible sums. We are all customers and when it comes down to what we like — nobody can deceive us. We know when a business cares and tries its best to please us. Leveraging cutting-edge tech for unparalleled UX is a win-win option every business owner should consider.

Music and tech: the perfect mixture for business purposes

This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Dihia LANASRI, OMDENA, New York, USA; (2) Juan OLANO, OMDENA, New York, USA; (3) Sifal KLIOUI, OMDENA, New York, USA; (4) Sin Liang Lee, OMDENA, New York, USA; (5) Lamia SEKKAI, OMDENA, New York, USA. Table of Links Abstract & Introduction Background Related Work Our Methodology Experiments and Results Conclusion, Acknowledgments & References 3 Related Work A restricted number of works have been published in the context of hate speech detection dealing with Arabic dialects. This section will analyze the most important NLP approaches dedicated to (1) the Algerian dialect and (2) Other Arabic dialects like Iraqi, Egyptian, Syrian, and Tunisian. This analysis helps us to identify the used approaches, models, and corpora. 3.1 Hate speech detection in Algerian dialect Guellil et al. [2022, 2021] Developed the first approach and corpus in Algerian dialect for hate speech detection against women in Arabic community on social media. This corpus contains more than 373K YouTube comments. Two different algorithms for feature extraction were used: Word2vec with machine learning models (GaussianNB, LogisticRegression, RandomForset, SGDClassifier, and LinearSVC) and FastText with Deep learning models (deep Convolutional Neural Network (CNN), long short-term memory (LSTM) network and Bi-directional LSTM (BiLSTM) network). Simulation results demonstrated the best performance of the CNN model with FastText. Boucherit and Abainia [2022] addressed the problem of detecting offensive and abusive content in Facebook comments. The corpus contains 8.7K comments in Algerian dialect written in Arabic and Latin characters, manually annotated as usual, abusive, and offensive. They used BiLSTM, CNN, FastText, SVM, and Multinomial Naive Bayes (NB) as classifiers. The experimental results showed that SVM and Multinomial NB classifiers outperformed all the other classifiers. Abainia et al. [2022] addressed the offensive language detection in the Amazigh language, which is one of the underresourced languages. They were interested in the Kabyle dialect. A new corpus of offensive Amazigh language is proposed containing 6.2K documents collected from Facebook and manually annotated as usual or offensive. A new lexicon of offensive and abusive Amazigh words with 12.6k entries is also developed. Many models have been evaluated, like SVM and Multinomial Naive Bayes classifiers tested with tf-idf. FastText was tested with deep learning models CNN and BiLSTM. The naive statistical classifier based on lexicon checking was the winner classifier. Mazari and Kheddar [2023] introduced a new dataset for Algerian dialect toxic text detection. An annotated multi-label dataset is built, containing around 14K comments extracted from Facebook, YouTube, and Twitter and labeled as hate speech, offensive language, and cyberbullying. Several tests have been conducted using many classification models of traditional machine learning: Random Forest, Naıve Bayes, Linear Support Vector (LSV), Stochastic Gradient Descent (SGD), and Logistic Regression. Furthermore, several assessments have been conducted using Deep Learning models such as CNN, LSTM, Gated Recurrent Unit (GRU), BiLSTM and Bidirectional-GRU (Bi-GRU). Results demonstrated the best performance of LSV, BiLSTM, and MLP when associated with the SGD model. Guellil et al. [2020] proposed a system for detecting hateful speech in Arabic political debates. The approach was evaluated against a hateful corpus concerning Algerian political debates. It contains 5K YouTube comments in MSA and Algerian dialects, written in both Arabic and Latin characters. Both classical algorithms of classification (Gaussian NB, Logistic Regression, Random Forest, SGD Classifier, and Linear SVC(LSVC)) and deep learning algorithms (CNN, multilayer perceptron (MLP), LSTM, and BiLSTM) are tested. For extracting features, the authors use Word2vec and FastText with their two implementations, namely, Skip Gram and CBOW. Simulation results demonstrate the best performance of LSVC, BiLSTM and MLP. Mohdeb et al. [2022] proposed an approach for analysis and the detection of dialectal Arabic hate speech that targeted African refugees and illegal migrants on the YouTube Algerian space. The corpus contains more than 4K comments annotated as Incitement, Hate, Refusing with non-hateful words, Sympathetic, and Comment. The transfer learning approach has been exploited for classification. The experiments show that the AraBERT monolingual transformer outperforms the mono-dialectal transformer DziriBERT and the cross-lingual transformers mBERT and XLM-R. 3.2 Hate speech detection in other Arabic dialects Various datasets or corpora were published in different dialects, which can be used for different purposes like hate speech, racism, violence, etc. detection. ALBayari and Abdallah [2022] is the first work to propose a corpus built from Instagram comments. This corpus contains 198K comments, written in MSA and three different dialects: Egyptian, Gulf, and Levantine. The comments were annotated as neutral, toxic, and Bullying. Al-Ajlan and Ykhlef [2018] and Haidar et al. [2019] datasets are collected from Twitter containing respectively 20K and 34K multi-dialectal Arabic tweets annotated as bullying and non-bullying labels. These tweets were from various dialects (Lebanon, Egypt, and the Gulf area). Moreover, two other datasets were proposed by Mubarak et al. [2017]. The first one with 1.1K tweets in different dialects and the second dataset contains 32K inappropriate comments collected from a famous Arabic news site and annotated as obscene, offensive, or clean. Albadi et al. [2018] proposed the religious hate speech detection where a multi-dialectal dataset of 6.6K tweets was introduced. It included an identification of the religious groups targeted by hate speech. Alakrot et al. [2018] also provided a dataset of 16K Egyptian, Iraqi, and Libyan comments collected from YouTube. The comments were annotated as either offensive, inoffensive, or neutral. T-HSAB Haddad et al. [2019] and L-HSAB Mulki et al. [2019] are two publicly available corpora for abusive hate speech detection. The first one is in the Tunisian dialect, combining 6K comments. The second one is in Levantine dialect (Syrian, Lebanese, Palestinian, and Jordanian dialects) containing around 6K tweets. These documents are labeled as Abusive, Hate, or Normal. Mubarak et al. [2020] looked at MSA and four major dialects (Egyptian, Levantine, Maghrebi, and Gulf). It presented a systematic method for building an Arabic offensive language tweet dataset that does not favor specific dialects, topics, or genres with 10K tweets. For tweet labeling, they used the count of positive and negative terms based on a polarity lexicon. FastText and Skip-Gram (AraVec skip-gram, Mazajak skip-gram); and deep contextual embeddings, namely BERTbase-multilingual and AraBERT are used. They evaluated different models: SVM, AdaBoost, and Logistic regression. Mulki and Ghanem [2021] introduced the first Arabic Levantine Twitter dataset for Misogynistic language (LeT-Mi) to be a benchmark dataset for automatic detection of online misogyny written in the Arabic and Levantine dialect. The proposed dataset consists of 6.5K tweets annotated either as neutral (misogynistic-free) or as one of seven misogyny categories: discredit, dominance, cursing/damning, sexual harassment, stereotyping and objectification, derailing, and threat of violence. They used BOW + TF-IDF, SOTA, LSTM, BERT, and Majority class as classifiers. Duwairi et al. [2021] investigated the ability of CNN, CNN-LSTM, and BiLSTM-CNN deep learning networks to classify or discover hateful content posted on social media. These deep networks were trained and tested using the ArHS dataset, which consists of around 10K tweets that were annotated to suit hateful speech detection in Arabic. Three types of experiments are reported: first, the binary classification of tweets into Hate or Normal. Ternary classification of tweets into (Hate, Abusive, or Normal), and multi-class classification of tweets into (Misogyny, Racism, Religious Discrimination, Abusive, and Normal). Aldjanabi et al. [2021] have built an offensive and hate speech detection system using a multi-task learning (MTL) model built on top of a pre-trained Arabic language model. The Arabic MTL model was experimented with two different language models to cover MSA and dialect Arabic. They evaluated a new pre-trained model ’MarBERT’ to classify both dialect and MSA tweets. They propose a model to explore multi-corpus-based learning using Arabic LMs and MTL to improve the classification performance. Haidar et al. [2017] presented a solution for the issue of cyberbullying in both Arabic and English languages. The proposed solution is based on machine learning algorithms using a dataset from Lebanon, Syria, the Gulf Area, and Egypt. That dataset contained 35K Arabic texts. In this research, Naïve Bayes and SVM models were chosen to classify the text. The SVM model achieved greater precision. Abdelali et al. [2016] The authors built a large dataset that consists of offensive Arabic words from different dialects and topics. The tweets were labeled into one of these categories: offensive, vulgar, hate speech, or clean. Since the offensive tweets involve implicit insults, the hate speech category was the tweets that contain racism, religious, and ethnic words. Different classifiers were employed in this study; the SVM model with a radial function kernel was mainly used with lexical features and pre-trained static embedding, while Adaptive Boosting and Logistic regression classifiers were employed when using Mazajak embedding. SVM gave the best precision. According to this literature analysis, we detect that the topic of hate speech detection in the Algerian dialect is not widely considered, and only few works deal with this problem. Furthermore, a lack of Algerian datasets prepared for hate speech is found. All these findings motivate our proposal.

Hate Speech Detection in Algerian Dialect Using Deep Learning: Related Work

This is paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Leah Chrestien, Czech Technical University in Prague; (2) Tomá˘s Pevný, Czech Technical University in Prague, and Gen Digital, Inc.; (3) Stefan Edelkamp, Czech Technical University in Prague; (4) Antonín Komenda, Czech Technical University in Prague. Table of Links Abstract & Introduction Preliminaries Conditions on strictly optimally efficient heuristic Loss functions Related Work Experiments Conclusion and References Appendix 7 Conclusion The motivation of this paper stemmed from the observation that even the cost-to-goal, considered to be an optimal heuristic, fails to guarantee a strictly optimally efficient search. Since a large body of existing research optimizes this quantity, we are effectively lost with respect to what should be optimized. To fill this gap, we have stated the necessary and sufficient conditions guaranteeing the forward search to be strictly optimally efficient. These conditions show that the absolute value of the heuristic is not important, but that the ranking of states in the Open list is what controls the efficiency. Ranking can be optimized by minimizing the ranking loss functions, but its concrete implementation needs to correspond to a variant of the forward search. In case of mismatch, the resulting heuristic can perform poorly, which has been demonstrated when the heuristic optimized for BGFS search was used with A* search. The other benefit of ranking losses is that from the point of view of statistical learning theory, they solve a simpler problem than ubiquitous regression in estimating the cost-to-goal. We do not question the existence of a strictly optimally efficient heuristic. Given our experiments, we believe that if the space of heuristic functions over which the loss is optimized is sufficiently rich, the result will be sufficiently close to the optimal for the needs of the search. 7.1 Acknowledgements This work has been supported by project numbers 22-32620S and 22-30043S from Czech Science Foundation and OP VVV project CZ.02.1.01/0.0/0.0/16_019/0000765 "Research Center for Informatics". This work has also received funding from the European Union’s Horizon Europe Research and Innovation program under the grant agreement TUPLES No 101070149. References [1] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik’s cube with deep reinforcement learning and search. Nature Machine Intelligence, 1(8):356–363, 2019. [2] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5360–5370, 2017. [3] Shahab Jabbari Arfaee, Sandra Zilles, and Robert C Holte. Learning heuristic functions for large state spaces. Artificial Intelligence, 175(16-17):2075–2098, 2011. [4] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. [5] Mohak Bhardwaj, Sanjiban Choudhury, and Sebastian Scherer. Learning heuristic search via imitation. In Conference on Robot Learning, pages 271–280. PMLR, 2017. [6] Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence. 2001 Jun; 129 (1-2): 5-33., 2001. [7] Vladimir Cherkassky, Xuhui Shao, Filip M Mulier, and Vladimir N Vapnik. Model complexity control for regression using vc generalization bounds. IEEE transactions on Neural Networks, 10(5):1075–1089, 1999. [8] Leah Chrestien, Tomáš Pevný, Antonín Komenda, and Stefan Edelkamp. Heuristic search planning with deep neural network using imitation, attention and curriculum learning. In PRLIJCAI workshop, 2021. [9] Lukás Chrpa. Combining learning techniques for classical planning: Macro-operators and entanglements. In ICTAI, pages 79–86. IEEE Computer Society, 2010. [10] Rina Dechter and Judea Pearl. The optimality of A* revisited. In Michael R. Genesereth, editor, AAAI, pages 95–99. AAAI Press, 1983. [11] Edsger W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1:269–271, 1959. [12] Stefan Edelkamp. Automated creation of pattern database search heuristics. In Model Checking and Artificial Intelligence, pages 35–50, 2006. [13] Stefan Edelkamp and Stefan Schrödl. Heuristic Search - Theory and Applications. Academic Press, 2012. [14] Ariel Felner, Uzi Zahavi, Robert Holte, Jonathan Schaeffer, Nathan Sturtevant, and Zhifu Zhang. Inconsistent heuristics in theory and practice. Artificial Intelligence, 175(9):1570–1603, 2011. [15] Dieqiao Feng, Carla P. Gomes, and Bart Selman. Solving hard AI planning instances using curriculum-driven deep reinforcement learning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 2198–2205. ijcai.org, 2020. [16] Patrick Ferber, Malte Helmert, and Jörg Hoffmann. Neural network heuristics for classical planning: A study of hyperparameter space. In ECAI, pages 2346–2353. IOS Press, 2020. [17] Santiago Franco, Álvaro Torralba, Levi H. S. Lelis, and Mike Barley. On creating complementary pattern databases. In IJCAI, pages 4302–4309, 2017. [18] Caelan Reed Garrett, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Learning to rank for synthesizing planning heuristics. page 3089–3095, 2016. [19] Edward Groshev, Aviv Tamar, Maxwell Goldstein, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. 2018. [20] Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra, Rémi Munos, and David Silver. Learning to search with mctsnets. In International conference on machine learning, pages 1822–1831. PMLR, 2018. [21] Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. [22] Patrik Haslum, Adi Botea, Malte Helmert, Blai Bonet, and Sven Koenig. Domain-independent construction of pattern database heuristics for cost-optimal planning. In AAAI, pages 1007–1012, 2007. [23] Malte Helmert and Gabriele Röger. How good is almost perfect? In AAAI, pages 944–949. AAAI Press, 2008. [24] Robert C. Holte and Sandra Zilles. On the optimal efficiency of cost-algebraic A. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pages 2288–2295. AAAI Press, 2019. [25] Michael Katz, Shirin Sohrabi, Horst Samulowitz, and Silvan Sievers. Delfi: Online planner selection for cost-optimal planning. IPC-9 planner abstracts, pages 57–64, 2018. [26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014. [27] Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning. Artificial intelligence, 6(4):293–326, 1975. [28] Richard E. Korf. Iterative-deepening-A*: An optimal admissible tree search. In IJCAI, pages 1034–1036. Morgan Kaufmann, 1985. [29] Richard E Korf. Macro-operators: A weak method for learning. Artificial intelligence, 26(1):35–77, 1985. [30] Šimon Mandlík, Matej Ra ˇ cinsk ˇ y, Viliam Lis ` y, and Tomáš Pevn ` y. Jsongrinder. jl: automated ` differentiable neural architecture for embedding arbitrary json data. Journal of Machine Learning Research, 23(298):1–5, 2022. [31] Drew V. McDermott. A heuristic estimator for means-ends analysis in planning. In AIPS, pages 142–149. AAAI, 1996. [32] David Ménager, Dongkyu Choi, Mark Roberts, and David W. Aha. Learning planning operators from episodic traces. In 2018 AAAI Spring Symposia. AAAI Press, 2018. [33] Ionut Moraru, Stefan Edelkamp, Santiago Franco, and Moisés Martínez. Simplifying automated pattern selection for planning with symbolic pattern databases. In KI, pages 249–263, 2019. [34] Christian Muise. Collection of pddl files of classical problems. https://github.com/AI-Planning/classical-domains, 2023. [35] XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. On surrogate loss functions and f-divergences. The Annals of Statistics, pages 876–904, 2009. [36] Laurent Orseau, Levi Lelis, Tor Lattimore, and Théophane Weber. Single-agent policy tree search with guarantees. Advances in Neural Information Processing Systems, 31, 2018. [37] Laurent Orseau and Levi HS Lelis. Implementation of the algorithms described in "policyguided heuristic search with guarantees" by l. orseau and l. lelis, published at aaai’21. https://github.com/levilelis/h-levin, 2021. [38] Laurent Orseau and Levi HS Lelis. Policy-guided heuristic search with guarantees. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12382–12390, 2021. [39] Arthur L Samuel. Some studies in machine learning using the game of checkers. ii—recent progress. IBM Journal of research and development, 11(6):601–617, 1967. [40] Max-Philipp B. Schrader. gym-sokoban. https://github.com/mpSchrader/gym-sokoban, 2018. [41] Jendrik Seipp, Florian Pommerening, and Malte Helmert. New optimization functions for potential heuristics. In ICAPS, 2015. [42] Jendrik Seipp and Silvan Sievers. Fast downward stone soup 2023. [43] Jendrik Seipp, Álvaro Torralba, and Jörg Hoffmann. PDDL generators. https://doi.org/10.5281/zenodo.6382173, 2022. [44] William Shen. Strips-hgn. https://github.com/williamshen-nz/STRIPS-HGN, 2021. [45] William Shen, Felipe Trevizan, and Sylvie Thiébaux. Learning domain-independent planning heuristics with hypergraph networks. In ICAPS, volume 30, pages 574–584, 2020. [46] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. [47] Kristian Smith. Maze creator / solver. https://github.com/ravenkls/Maze-Generator-and Solver, 2022. [48] Gustav Sourek, Vojtech Aschenbrenner, Filip Zelezny, Steven Schockaert, and Ondrej Kuzelka. Lifted relational neural networks: Efficient learning of latent relational structures. Journal of Artificial Intelligence Research, 62:69–100, 2018. [49] Simon Ståhlberg, Blai Bonet, and Hector Geffner. Learning general optimal policies with graph neural networks: Expressive power, transparency, and limits. In Akshat Kumar, Sylvie Thiébaux, Pradeep Varakantham, and William Yeoh, editors, Proceedings of the Thirty-Second International Conference on Automated Planning and Scheduling, ICAPS 2022, Singapore (virtual), June 13-24, 2022, pages 629–637. AAAI Press, 2022. [50] Ingo Steinwart. Consistency of support vector machines and other regularized kernel classifiers. IEEE transactions on information theory, 51(1):128–142, 2005. [51] Takeshi Takahashi, He Sun, Dong Tian, and Yebin Wang. Learning heuristic functions for mobile robot path planning using deep neural networks. In ICAPS, volume 29, pages 764–772, 2019. [52] Alvaro Torralba, Vidal Alcázar, Daniel Borrajo, Peter Kissmann, and Stefan Edelkamp. SymBA*: A symbolic bidirectional A* planner. In International Planning Competition, pages 105–108, 2014. [53] Sam Toyer, Sylvie Thiébaux, Felipe Trevizan, and Lexing Xie. Asnets: Deep learning for generalised planning. Journal of Artificial Intelligence Research, 68:1–68, 2020. [54] Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991. [55] Marin Vlastelica, Michal Rolinek, and Georg Martius. Neuro-algorithmic policies enable fast combinatorial generalization. In International Conference on Machine Learning, pages 10575– 10585. PMLR, 2021. [56] Xuemei Wang. Learning planning operators by observation and practice. In Kristian J. Hammond, editor, AIPS, pages 335–340. AAAI, 1994. [57] Christopher Wilt and Wheeler Ruml. When does weighted A* fail? In International Symposium on Combinatorial Search, volume 3, 2012. [58] Christopher Makoto Wilt and Wheeler Ruml. Effective heuristics for suboptimal best-first search. J. Artif. Intell. Res., 57:273–306, 2016. [59] Yuehua Xu, Alan Fern, and Sungwook Yoon. Learning linear ranking functions for beam search with application to planning. Journal of Machine Learning Research, 10(7), 2009. [60] Ryo Yonetani, Tatsunori Taniai, Mohammadamin Barekatain, Mai Nishimura, and Asako Kanezaki. Path planning using neural A* search. In International Conference on Machine Learning, pages 12029–12039. PMLR, 2021. [61] Ziqi Zhang and Florian Geißer. Extending graph neural networks for generalized stochastic planning. In Bridging the Gap Between AI Planning and Reinforcement Learning. 2021. [62] Tan Zhi-Xuan. PDDL. jl: An Extensible Interpreter and Compiler Interface for Fast and Flexible AI Planning. PhD thesis, Massachusetts Institute of Technology, 2022. [63] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795–18806, 2020.

Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goal: Conclusion and References

Generate and Pray: Using SALLMS to Evaluate the Security: Conclusion & References

Table of Contents A Deep Dive into Voice Cloning with SoftVC VITS and Bert-VITS2 Prepare Dataset Extract from a Song UVR Workflows Preparation for vocal recording Cheapskate’s Audio Equipment Audacity workflows audio-slicer Cleaning dataset Match loudness so-vits-svc Set up environment Initialization Download pre-trained models Dataset Preparation Edit Configs Training Inference Audio Editing so-vits-svc-fork Installation Preparation Training Inference DDSP-SVC Preparation Training Inference Bert-vits2-V2.3 Initialization Preparation Transcription Training and Inference vits-simple-api Tweaks Share models In the , I tried a little bit of and found it interesting. So, I decided to train a usable model with my own voice. previous post TTS Generation WebUI This voice cloning project explores both SVC for Voice Changing and for Text-to-Speech. There is no one tool that does all jobs. VITS I have tested several tools for this project. Many of the good guides, like , , and , are in Chinese. So, I thought it would be useful to post my notes in English. this this this Although has been archived for a few months, probably due to oppression, it is still the tool for the best result. so-vits-svc Other related tools such as , , , and provide either faster/liter optimization, more features, or better interfaces. so-vits-svc-fork so-vits-svc-5.0 DDSP-SVC RVC But with enough time and resources, none of these alternatives can compete with the superior result generated by the original so-vits-svc. For TTS, a new tool called works fantastically and has already matured with its final release last month. It has some very different use cases, for example, audio content creation. Bert-VITS2 Prepare Dataset The audio files of the dataset should be in WAV format, 44100 Hz, 16bit, mono, 1-2 hours ideally. Extract From a Song is the easiest tool for this job. There is a that explains everything in detail. Ultimate Vocal Remover thread UVR Workflows Remove and extract Instrumental Model: VR - UVR(4_HP-Vocal-UVR) Settings: 512 - 10 - GPU Output and unclean vocal Instrumental Remove and extract background vocals. Model: VR - UVR(5_HP-Karaoke-UVR) Settings: 512 - 10 - GPU Output and unclean main vocal background vocal Remove reverb and noise. Model: VR - UVR-DeEcho-DeReverb & UVR-DeNoise Settings: 512 - 10 - GPU - No Other Only Output clean main vocal (Optional) Using RipX (non-free) to perform a manual fine-cleaning. Preparation for Vocal Recording It’s better to record in a treated room with a condenser microphone, otherwise, use a directional or dynamic microphone to reduce noise. Cheapskate’s Audio Equipment The very first time I got into music was during high school, with the blue Sennheiser MX500 and Koss Porta Pro. I still remember the first time I was recording a song that was on a Sony VAIO with Cool Edit Pro. Nowadays, I still resist spending a lot of money on audio hardware as an amateur because it is literally a money-sucking black hole. Nonetheless, I really appreciate the reliability of those cheap production equipment. The core part of my setup is a Behringer UCA202, and it’s perfect for my use cases. I bought it for $10 during a price drop. It is a so-called “Audio Interface” but basically just a sound card with multiple ports. I used RCA to 3.5mm TRS cables for my headphones, a semi-open K240s for regular output, and a closed-back HD669/MDR7506 for monitor output. All three mentioned headphones are under $100 for the normal price. And there are clones from Samson, Tascam, Knox Gear, and more out there for less than $50. For the input device, I’m using a dynamic microphone for the sake of my environmental noises. It is an SM58 copy (Pyle) + a Tascam DR-05 recorder (as an amplifier). Other clones such as SL84c or wm58 would do it too. I use an XLR to 3.5mm TRS cable to connect the microphone to the MIC/External-input of the recorder, and then use an AUX cable to connect between the line-out of the recorder and the input of the UCA202. It’s not recommended to buy an “audio interface” and a dedicated amplifier to replicate my setup. A $10 c-media USB sound card should be good enough. The Syba model that I owned is capable of “pre-amping” dynamic microphones directly and even some lower-end phantom-powered microphones. The setup can go extremely cheap ($40~60), but with UCA202 and DR-05, the sound is much cleaner. And I really like the physical controls, versatility, and portability of my old good digital recorder. Audacity Workflows Although when I was getting paid as a designer, I was pretty happy with Audition. But for personal use on a fun project, Audacity is the way to avoid the chaotic evil of Adobe. Noise Reduction Dereverb Truncate Silence Normalize audio-slicer Use or to slice the audio file into small pieces for later use. audio-slicer audio-slicer (gui) The default setting works great. Cleaning Dataset Remove those very short ones and re-slice those that are still over 10 seconds. In case of a large dataset, remove all that are less than 4 sec. In the case of a small dataset, remove those only under 2 sec. If necessary, perform a manual inspection for every single file. Match Loudness Use Audacity again with ; 0db should do it. Loudness Normalization so-vits-svc Set Up the Environment Virtual environment is essential to run multiple Python tools inside one system. I used to use VMs and Docker, but now, I found that is way quicker and handier than the others. anaconda Create a new environment for so-vits-svc, and activate it. conda create -n so-vits-svc python=3.8
conda activate so-vits-svc Then, install requirements. git clone https://github.com/svc-develop-team/so-vits-svc
cd so-vits-svc

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

#for linux
pip install -r requirements.txt

#for windows
pip install -r requirements_win.txt
pip install --upgrade fastapi==0.84.0
pip install --upgrade gradio==3.41.2
pip install --upgrade pydantic==1.10.12
pip install fastapi uvicorn Initialization Download pre-trained models pre-train wget https://huggingface.co/WitchHuntTV/checkpoint_best_legacy_500.pt/resolve/main/checkpoint_best_legacy_500.pt wget https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/rmvpe.pt logs/44k wget https://huggingface.co/datasets/ms903/sovits4.0-768vec-layer12/resolve/main/sovits_768l12_pre_large_320k/clean_D_320000.pth wget https://huggingface.co/datasets/ms903/sovits4.0-768vec-layer12/resolve/main/sovits_768l12_pre_large_320k/clean_G_320000.pth logs/44k/diffusion wget https://huggingface.co/datasets/ms903/Diff-SVC-refactor-pre-trained-model/resolve/main/fix_pitch_add_vctk_600k/model_0.pt (Alternative) wget https://huggingface.co/datasets/ms903/DDSP-SVC-4.0/resolve/main/pre-trained-model/model_0.pt (Alternative) wget https://huggingface.co/datasets/ms903/Diff-SVC-refactor-pre-trained-model/blob/main/hubertsoft_fix_pitch_add_vctk_500k/model_0.pt pre-train/nsf_hifigan wget -P pretrain/ https://github.com/openvpi/vocoders/releases/download/nsf-hifigan-v1/nsf_hifigan_20221211.zip unzip -od pretrain/nsf_hifigan pretrain/nsf_hifigan_20221211.zip Dataset Preparation Put all Prepared audio.wav files into dataset_raw/character cd so-vits-svc
python resample.py --skip_loudnorm
python preprocess_flist_config.py --speech_encoder vec768l12 --vol_aug
python preprocess_hubert_f0.py --use_diff Edit Configs The file is located at configs/config.json : the frequency of printing log : the frequency of saving checkpoints : total steps : numbers of saved checkpoints, 0 for unlimited. : fp32 In my case, : the smaller the faster (rougher), the larger the slower (better). log interval eval interval epochs keep ckpts half_type batch_size Recommended batch_size per VRAM: 4=6G；6=8G；10=12G；14=16G；20=24G Keep default for configs/diffusion.yaml Training python cluster/train_cluster.py --gpu
python train_index.py -c configs/config.json
python train.py -c configs/config.json -m 44k
python train_diff.py -c configs/diffusion.yaml On training steps: Use to train the main model; usually, 20k-30k would be usable, and 50k and up would be good enough. This can take a few days depending on the GPU speed. train.py Feel free to stop it by , and it will be continued training by re-run anytime. ctrl+c python train.py -c configs/config.json -m 44k Use to train the diffusion model; training steps is recommended at 1/3 of the main model. train_diff.py Be aware of over-training. Use to monitor the plots to see if it goes flat. tensorboard --logdir=./logs/44k Change the from 0.0001 to 0.00005 if necessary. learning rate When done, share/transport these files for inference. config/ config.json diffusion.yaml logs/44k feature_and_index.pkl kmeans_10000.pt model_0.pt G_xxxxx.pt Inference It’s time to try out the trained model. I’d prefer Webui for the convenience of tweaking the parameters. But before firing it up, edit the following lines in for LAN access: webUI.py os.system("start http://localhost:7860")
app.launch(server_name="0.0.0.0", server_port=7860) Run ; then access its from a web browser. python webUI.py ipaddress:7860 The webui has no English localization, but would be helpful. Immersive Translate Most parameters would work well with the default value. Refer to and to make changes. this this Upload these 5 files: and its main model.pt config.json and its diffusion model.pt diffusion.yaml Either cluster model for speaking or feature retrieval for singing. kmeans_10000.pt feature_and_index.pkl is for speaking only, not for singing. Recommend when using. F0 predictor RMVPE is useful when singing a feminine song using a model with a masculine voice, or vice versa. Pitch change is the way of controlling the tone. Use to get the clearest speech, and use to get the closest tone to the model. Clustering model/feature retrieval mixing ratio 0.1 0.9 should be set around , it enhances the result at steps. shallow diffusion steps 50 30-100 Audio Editing This procedure is optional. Just for the production of a better song. I won’t go into details about this since the audio editing software, or so-called DAW (digital audio workstation), that I’m using is non-free. I have no intention to advocate proprietary software even though the entire industry is paywalled and closed-source. Audacity supports multitrack, effects, and a lot more. It does load some advanced VST plugins as well. It’s not hard to find tutorials on mastering songs with Audacity. Typically, the mastering process should be mixing/balancing, EQ/compressing, reverb, and imaging. The more advanced the tool is, the easier the process will be. I’ll definitely spend more time on adopting Audacity for my mastering process in the future, and I recommend everyone do so. so-vits-svc-fork This is a so-vits-svc with real-time support, and the models are compatible. Easier to use but does not support the Diffusion model. For dedicated real-time voice changing, a is recommended. fork voice-changer Installation conda create -n so-vits-svc-fork python=3.10 pip
conda activate so-vits-svc-fork

git clone https://github.com/voicepaw/so-vits-svc-fork
cd so-vits-svc-fork

python -m pip install -U pip setuptools wheel
pip install -U torch torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -U so-vits-svc-fork
pip install click
sudo apt-get install libportaudio2 Preparation Put dataset .wav files into so-vits-svc-fork/dataset_raw svc pre-resample
svc pre-config Edit a in . This fork is a larger size than the original. batch_size configs/44k/config.json Training svc pre-hubert
svc train -t
svc train-cluster Inference Use GUI with . This requires a local desktop environment. svcg Or use CLI with for real time and for generating. svc vc svc infer -m "logs/44k/xxxxx.pth" -c "configs/config.json" raw/xxx.wav DDSP-SVC requires fewer hardware resources and runs faster than so-vits-svc. It supports both real-time and diffusion models (Diff-SVC). DDSP-SVC conda create -n DDSP-SVC python=3.8
conda activate DDSP-SVC

git clone https://github.com/yxlllc/DDSP-SVC
cd DDSP-SVC

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt Refer to the for the two files: Initialization section pretrain/rmvpe/model.pt
pretrain/contentvec/checkpoint_best_legacy_500.pt Preparation python draw.py
python preprocess.py -c configs/combsub.yaml
python preprocess.py -c configs/diffusion-new.yaml Edit configs/ batch_size: 32  (16 for diffusion)
cache_all_data: false
cache_device: 'cuda'
cache_fp16: false Training conda activate DDSP-SVC
python train.py -c configs/combsub.yaml
python train_diff.py -c configs/diffusion-new.yaml

tensorboard --logdir=exp Inference It’s recommended to use since it includes both DDSP and diffusion model. main_diff.py python main_diff.py -i "input.wav" -diff "model_xxxxxx.pt" -o "output.wav" Real-time GUI for voice cloning: python gui_diff.py Bert-vits2-V2.3 This is a TTS tool that is completely different from everything above. By using it, I have already created several audiobooks with my voice for my parents, and they really enjoy it. Instead of using the , I used the fork by for an easier setup. original v3u Initialization conda create -n bert-vits2 python=3.9
conda activate bert-vits2

git clone https://github.com/v3ucn/Bert-vits2-V2.3.git
cd Bert-vits2-V2.3

pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt Download pre-trained models (includes Chinese, Japanese, and English): wget -P slm/wavlm-base-plus/ https://huggingface.co/microsoft/wavlm-base-plus/resolve/main/pytorch_model.bin
wget -P emotional/clap-htsat-fused/ https://huggingface.co/laion/clap-htsat-fused/resolve/main/pytorch_model.bin
wget -P emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/ https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim/resolve/main/pytorch_model.bin
wget -P bert/chinese-roberta-wwm-ext-large/ https://huggingface.co/hfl/chinese-roberta-wwm-ext-large/resolve/main/pytorch_model.bin
wget -P bert/bert-base-japanese-v3/ https://huggingface.co/cl-tohoku/bert-base-japanese-v3/resolve/main/pytorch_model.bin
wget -P bert/deberta-v3-large/ https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.bin
wget -P bert/deberta-v3-large/ https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.generator.bin
wget -P bert/deberta-v2-large-japanese/ https://huggingface.co/ku-nlp/deberta-v2-large-japanese/resolve/main/pytorch_model.bin Create a character model folder mkdir -p Data/xxx/models/ Download base models: !wget -P Data/xxx/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/DUR_0.pth
!wget -P Data/xxx/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/D_0.pth
!wget -P Data/xxx/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/G_0.pth
!wget -P Data/xxx/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/WD_0.pth

#More options
https://openi.pcl.ac.cn/Stardust_minus/Bert-VITS2/modelmanage/model_filelist_tmpl?name=Bert-VITS2_2.3%E5%BA%95%E6%A8%A1
https://huggingface.co/Erythrocyte/bert-vits2_base_model/tree/main
https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/tree/main Edit by replacing all to train_ms.py bfloat16 float16 Edit for LAN access: webui.py webbrowser.open(f"start http://localhost:7860")
app.launch(server_name="0.0.0.0", server_port=7860) Edit for and Data/xxx/config.json batch_size spk2id Preparation Similar workflow as in the . previous section Remove noise and silence, normalization, then put the un-sliced WAV file into . Data/xxx/raw Edit for , , and . config.yml dataset_path num_workers keep_ckpts Run to slice the WAV file. python3 audio_slicer.py Clean the dataset ( ) by removing small files that are under 2 sec. Data/xxx/raw Transcription Install whisper pip install git+https://github.com/openai/whisper.git To turn off language auto-detection, set it to English only, and use model; edit as below: large short_audio_transcribe.py # set the spoken language to english
    print('language: en')
    lang = 'en'
    options = whisper.DecodingOptions(language='en')
    result = whisper.decode(model, mel, options)
	
    # set to use large model
    parser.add_argument("--whisper_size", default="large")

    #Solve error "Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead" while using large model
    mel = whisper.log_mel_spectrogram(audio,n_mels = 128).to(model.device) Run to start the transcription. python3 short_audio_transcribe.py Re-sample the sliced dataset: python3 resample.py --sr 44100 --in_dir ./Data/zizek/raw/ --out_dir ./Data/zizek/wavs/ Preprocess transcription: python3 preprocess_text.py --transcription-path ./Data/zizek/esd.list Generate BERT feature config: python3 bert_gen.py --config-path ./Data/zizek/configs/config.json Training and Inference Run to start training python3 train_ms.py Edit for model path: config.yml model: "models/G_20900.pth" Run to start webui for inference python3 webui.py vits-simple-api is a web frontend for using trained models. I use this mainly for its long text support which the original project doesn’t have. vits-simple-api git clone https://github.com/Artrajz/vits-simple-api
git pull https://github.com/Artrajz/vits-simple-api
cd vits-simple-api

conda create -n vits-simple-api python=3.10 pip
conda activate vits-simple-api && 

pip install -r requirements.txt (Optional) Copy pre-trained model files from to Bert-vits2-V2.3/ vits-simple-api/bert_vits2/ Copy and to Bert-vits2-V2.3/Data/xxx/models/G_xxxxx.pth Bert-vits2-V2.3/Data/xxx/config.json vits-simple-api/Model/xxx/ Edit for and as preferred config.py MODEL_LIST Default parameter Edit as below: Model/xxx/config.json "data": {
    "training_files": "Data/train.list",
    "validation_files": "Data/val.list",
	
  "version": "2.3" Check/Edit in as model_list config.yml [xxx/G_xxxxx.pth, xxx/config.json] Run python app.py Tweaks for tone, for randomness, for pronunciation, and for speed. and are self-explanatory SDP Ratio Noise Noise_W Length emotion style Share Models In its , there are a lot of VITS models shared by others. You can try it out first, and then download the desired models from . Hugging Face repo Files The is widely used in some content creation communities because of its high quality. It contains hundreds of characters, although only Chinese and Japanese are supported. Genshin model In , there are a lot of Bert-vits2 models that are made from popular Chinese streamers and VTubers. another repo There are already projects making AI Vtubers like and . I’m looking forward to how this technology can change the industry in the near future. this this https://techshinobi.org/posts/voice-vits/https://techshinobi.org/posts/voice-vits/

How to Voice Clone With SoftVC VITS and Bert-VITS2: A Deep Dive

SoftVC VITS এবং Bert-VITS2 দিয়ে কীভাবে ভয়েস ক্লোন করবেন: একটি গভীর ডুব

So klonen Sie Stimmen mit SoftVC VITS und Bert-VITS2: Ein tiefer Einblick

Cómo clonar por voz con SoftVC VITS y Bert-VITS2: una inmersión profunda

Comment cloner la voix avec SoftVC VITS et Bert-VITS2 : une plongée approfondie

SoftVC VITS और Bert-VITS2 के साथ वॉयस क्लोन कैसे करें: एक गहन जानकारी

SoftVC VITS および Bert-VITS2 を使用して音声クローンを作成する方法: 詳細

SoftVC VITS 및 Bert-VITS2를 사용한 음성 복제 방법: 심층 분석

Como clonar voz com SoftVC VITS e Bert-VITS2: um mergulho profundo

Как озвучить клон с помощью SoftVC VITS и Bert-VITS2: глубокое погружение

SoftVC VITS ve Bert-VITS2 ile Ses Klonlama: Derinlemesine Bir İnceleme

Cách sao chép giọng nói với SoftVC VITS và Bert-VITS2: Tìm hiểu sâu

如何使用 SoftVC VITS 和 Bert-VITS2 进行语音克隆：深入探讨

This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Shanshan Han & Qifan Zhang, UCI; (2) Wenxuan Wu, Texas A&M University; (3) Baturalp Buyukates, Yuhang Yao & Weizhao Jin, USC; (4) Salman Avestimehr, USC & FedML. Table of Links Abstract and Introduction Problem Setting The Proposed Two-Stages Anomaly Detection Verifiable Anomaly Detection using ZKP Evaluations Related Works Conclusion & References 7 CONCLUSION We present a cutting-edge anomaly detection technique specifically designed for the real-world FL systems. Our approach utilizes an early cross-round check that activates subsequent anomaly detection exclusively in the presence of attacks. When attacks happen, our approach removes anomaly client models efficiently, ensuring that the local models submitted by benign clients remain unaffected. Further, by leveraging ZKPs, participating clients get to verify the integrity of the anomaly detection and removal performed by the server. The practical design and the inherent efficiency of our approach make it well-suited for real-world FL systems. Our plans for future works include extending our approach to asynchronous FL and vertical FL scenarios. REFERENCES T. Aoki. On the stability of the linear transformation in banach spaces. Journal of the Mathematical Society of Japan, 2(1-2):64–66, 1950. E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov. How to backdoor federated learning. In AISTATS, 2020a. Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938–2948. PMLR, 2020b. Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634– 643. PMLR, 2019. P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In NeurIPS, 2017. David Byrd and Antigoni Polychroniadou. Differentially private secure multi-party computation for federated learning in financial applications. In Proceedings of the First ACM International Conference on AI in Finance, pp. 1–9, 2020. Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konecnˇ y, H Brendan McMa- ` han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018. Xiaoyu Cao and Neil Zhenqiang Gong. Mpaf: Model poisoning attacks to federated learning based on fake clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3396–3404, 2022. Xiaoyu Cao and Neil Zhenqiang Gong. Mpaf: Model poisoning attacks to federated learning based on fake clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3396–3404, 2022. Xiaoyu Cao, Zaixi Zhang, Jinyuan Jia, and Neil Zhenqiang Gong. Flcert: Provably secure federated learning against poisoning attacks. IEEE Transactions on Information Forensics and Security, 17: 3691–3705, 2022. Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Franc¸oise Beaufays. Federated learning of outof-vocabulary words. arXiv preprint arXiv:1903.10635, 2019. Y. Chen, L. Su, and J. Xu. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. ACM on Measurement and Analysis of Computing Systems, 1(2):1–25, 2017. Alexander Chowdhury, Hasan Kassem, Nicolas Padoy, Renato Umeton, and Alexandros Karargyris. A review of medical federated learning: Applications in oncology and cancer research. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 7th International Workshop, BrainLes 2021, Held in Conjunction with MICCAI 2021, Virtual Event, September 27, 2021, Revised Selected Papers, Part I, pp. 3–24. Springer, 2022. M. Fang, X. Cao, J. Jia, and N. Gong. Local model poisoning attacks to Byzantine-robust federated learning. In USENIX Security, 2020. B. Feng, L. Qin, Z. Zhang, Y. Ding, and S. Chu. ZEN: An optimizing compiler for verifiable, zero-knowledge neural network inferences. 2021. Cryptology ePrint Archive. Grant S Fletcher. Clinical epidemiology: the essentials. Lippincott Williams & Wilkins, 2019. R. Freivalds. Probabilistic machines can use less running time. In IFIP Congress, 1977. Shuhao Fu, Chulin Xie, Bo Li, and Qifeng Chen. Attack-resistant federated learning with residualbased reweighting. arXiv preprint arXiv:1912.11464, 2019. Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. The limitations of federated learning in sybil settings. In RAID, pp. 301–316, 2020. S. Goldwasser, S. Micali, and C. Rackoff. The knowledge complexity of interactive proof systems. SIAM Jour. on Comp., 18(1):186–208, 1989. J. Groth. On the size of pairing-based non-interactive arguments. In Eurocrypt, 2016. Rachid Guerraoui, Sebastien Rouault, et al. The hidden vulnerability of distributed learning in ´ byzantium. In International Conference on Machine Learning, pp. 3521–3530. PMLR, 2018. S. Han, H. Wang, J. Wan, and J. Li. An iterative scheme for leverage-based approximate aggregation. In IEEE ICDE, 2019. Shanshan Han, Baturalp Buyukates, Zijian Hu, Han Jin, Weizhao Jin, Lichao Sun, Xiaoyang Wang, Chulin Xie, Kai Zhang, Qifan Zhang, et al. Fedmlsecurity: A benchmark for attacks and defenses in federated learning and federated llms. arXiv preprint arXiv:2306.04959, 2023. Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Franc¸oise Beaufays, Sean Augenstein, Hubert Eichner, Chloe Kiddon, and Daniel Ramage. Federated learning for mobile ´ keyboard prediction. arXiv preprint arXiv:1811.03604, 2018. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016. L. He, S. P. Karimireddy, and M. Jaggi. Byzantine-robust decentralized learning via self-centered clipping. 2022. Available on arXiv:2202.01545. Xiao Jin, Pin-Yu Chen, Chia-Yi Hsu, Chia-Mu Yu, and Tianyi Chen. Cafe: Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems, 34:994–1006, 2021. Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. arXiv preprint arXiv:2006.09365, 2020. Sanjay Kariyappa, Chuan Guo, Kiwan Maeng, Wenjie Xiong, G. Edward Suh, Moinuddin K. Qureshi, and Hsien-Hsin S. Lee. Cocktail party attack: Breaking aggregation-based privacy in federated learning using independent component analysis. In International Conference on Machine Learning, 2022. URL https://api.semanticscholar.org/CorpusID: 252211968. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. Maximilian Lam, Gu-Yeon Wei, David Brooks, Vijay apa Reddi, and Michael Mitzenmacher. Gradient disaggregation: Breaking privacy in federated learning by reconstructing the user participant matrix. In International Conference on Machine Learning, pp. 5959–5968. PMLR, 2021. S. Lee, H. Ko, J. Kim, and H. Oh. vCNN: Verifiable convolutional neural network based on zksnarks. 2020. Cryptology ePrint Archive. David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. Federated learning for keyword spotting. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6341–6345, 2019. Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. arXiv preprint arXiv:2002.00211, 2020. T. Liu, X. Xie, and Y. Zhang. ZkCNN: Zero knowledge proofs for convolutional neural network predictions and accuracy. In ACM CCS, 2021. A. Lyon. Why are normal distributions normal? The British Journal for the Philosophy of Science, 65(3):621–649, 2014. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017a. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017b. J. Osborne. Improving your data transformations: Applying the Box-Cox transformation. Practical Assessment, Research, and Evaluation, 15(1):12, 2010. Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia R Gel. Defending against backdoors in federated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9268–9276, 2021. Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142–1154, 2022. Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Franc¸oise Beaufays. Federated learning for emoji prediction in a mobile keyboard. arXiv preprint arXiv:1906.04329, 2019. M. Rosenblatt. A central limit theorem and a strong mixing condition. National Academy of Sciences, 42(1):43–47, 1956. R. M. Sakia. The Box-Cox transformation technique: A review. Journal of the Royal Statistical Society: Series D, 41(2):169–178, 1992. E. B. Sasson, A. Chiesa, C. Garman, M. Green, I. Miers, E. Tromer, and M. Virza. Zerocash: Decentralized anonymous payments from bitcoin. In IEEE S&P, 2014. Jingwei Sun, Ang Li, Louis DiValentin, Amin Hassanzadeh, Yiran Chen, and Hai Li. Fl-wbc: Enhancing robustness against model poisoning attacks in federated learning from a client perspective. Advances in Neural Information Processing Systems, 34:12613–12624, 2021. Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.07963, 2019. Circom Contributors. Circom zkSNARK ecosystem, 2022. https://github.com/iden3/circom. Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Security, pp. 480– 501. Springer, 2020. Richard Tomsett, Kevin Chan, and Supriyo Chakraborty. Model poisoning attacks against distributed machine learning systems. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, pp. 481–489. SPIE, 2019. H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J. Sohn, K. Lee, and D. Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In NeurIPS, 2020. Jianhua Wang. Pass: Parameters audit-based secure and fair federated learning scheme against free rider. arXiv preprint arXiv:2207.07292, 2022. S. Weisberg. Yeo-Johnson power transformations. 2001. Available at https://www.stat.umn.edu/arc/yjpower.pdf. Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. SLSGD: Secure and Efficient Distributed Ondevice Machine Learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 213–228. Springer, 2020. H. Yang, X. Zhang, M. Fang, and J. Liu. Byzantine-resilient stochastic gradient descent for distributed learning: A Lipschitz-inspired coordinate-wise median approach. In IEEE CDC, 2019. Dong Yin, Yudong Chen, Kannan Ramchandran, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650–5659. PMLR, 2018. Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Implementation of fldetector. https://github.com/zaixizhang/FLDetector, 2022a. Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545–2555, 2022b. Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael W. Mahoney, Joseph Gonzalez, Kannan Ramchandran, and Prateek Mittal. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, 2022c. URL https: //api.semanticscholar.org/CorpusID:249889464. A APPENDIX A.1 PROOF OF THE RANGE OF PPV B ZKP IMPLEMENTATION B.1 CHOICE OF THE ZKP SYSTEM In our implementation, we use the Groth16 (Groth, 2016) zkSNARK scheme implemented in the Circom library (Circom Contributors, 2022) for all the computations described above. We chose this ZKP scheme because its construction ensures constant proof size (128 bytes) and constant verification time. Because of this, Groth16 is popular for blockchain application due to small on-chain computation. There are other ZKP schemes based on different construction that can achieve faster prover time (Liu et al., 2021), but the proof size is too big and verification time is not constant, which is a problem if verifier lacks computational power. The construction of a ZKP scheme that is efficient for both prover and verifier is still an open research direction. B.2 SUPPLEMENTARY EXPERIMENTS FOR IMPORTANCE LAYER

MUBERT INC

mubert.com

-- employees

Since n.d.

COMPANY RANKING

MUBERT INC

EVERGREEN INDEX #6856

Representing this company? Click here to claim and customize this page!

Mubert Inc WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

MUBERT INC

mubert.com

-- employees

Since n.d.

COMPANY RANKING

MUBERT INC

EVERGREEN INDEX #6856

RELATED COMPANIESRANK

STORIES ABOUT MUBERT INC

Representing this company? Click here to claim and customize this page!

Mubert Inc WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!