Anthony Chen’s quiet character study follows a traumatized Liberian woman (Cynthia Erivo) on a Greek island who befriends an American tour guide (Alia Shawkat).

‘Drift’ Review: Cynthia Erivo Keeps a Breakdown at Bay - The New York Times

Pod of orcas in north Japan freed themselves as gaps between the ice grew, officials say

Killer whales trapped in drift ice off Japan’s Hokkaido coast escape | Japan | The Guardian

The pod of about a dozen killer whales were first spotted stranded on Monday north of Hokkaido.

Trapped orcas escape from drift ice near Japan

An image captured off the Norwegian archipelago of Svalbard shows a young polar bear drifting to sleep on a small iceberg after carving a bed in the ice.

Polar bear sleeping on tiny iceberg drifting in Arctic sea captured in heartbreaking photo | Live Science

A group of more than a dozen orcas has disappeared after being trapped in freezing Japanese waters for about a day – it isn’t known if they escaped or have died

Orca pod disappears after being trapped in drifting ice off Japan | New Scientist

Video posted on social media shows dozens of trapped orca whales escaping after being encircled by drift ice.

Video Shows Dozens Of Trapped Whales Attempt To Escape Drift Ice | The Daily Caller

A pod of killer whales that was trapped in drift ice off Japan's northern main island of Hokkaido, prompting concern from environmental groups, has apparently safely escaped, officials said Wednesday. The killer whales, also known as orcas, were initially spotted by a local fisherman who reported ...

A pod of killer whales trapped in drift ice off northern Japan has apparently safely escaped

The coast guard is unable to rescue them as the ice is too thick, an official told local media.

Orcas gasp for air whilst trapped in drift ice off the coast of Japan

The coast guard is unable to rescue them as the ice is too thick, an official tells local media.

Orcas gasp for air whilst trapped in drift ice off the coast of Japan - BBC News

The struggling orcas appeared to be sticking their heads out of the water to breathe. - Videos from The Weather Channel | weather.com

Drone Footage Captures Killer Whales Trapped in Drift Ice off Japan's Coast; Thick Ice Hampers Rescue Efforts - Videos from The Weather Channel

Driff

driff

Authors:
(1) Anton Razzhigaev, AIRI and Skoltech;
(2) Arseniy Shakhmatov, Sber AI;
(3) Anastasia Maltseva, Sber AI;
(4) Vladimir Arkhipkin, Sber AI;
(5) Igor Pavlov, Sber AI;
(6) Ilya Ryabov, Sber AI;
(7) Angelina Kuts, Sber AI;
(8) Alexander Panchenko, AIRI and Skoltech;
(9) Andrey Kuznetsov, AIRI and Sber AI;
(10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 1 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction
Related Work
Demo System
Kandinsky Architecture
Experiments
Results
Conclusion & Limitations
Ethical Considerations, Acknowledgements and References Abstract Text-to-image generation is a significant domain in modern computer vision and has achieved substantial improvements through the evolution of generative architectures. Among these, there are diffusion-based models that have demonstrated essential quality enhancements. These models are generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky[1], a novel exploration of latent diffusion architecture, combining the principles of the image prior models with latent diffusion techniques. The image prior model is trained separately to map text embeddings to image embeddings of CLIP. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall, the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation, and text-guided inpainting/outpainting. Additionally, we released the source code and checkpoints for the Kandinsky models. Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K dataset, marking our model as the top open-source performer in terms of measurable image generation quality. 1 Introduction In quite a short period of time, generative abilities of text-to-image models have improved substantially, providing users with photorealistic quality, near real-time inference speed, a great number of applications and features, including simple easy-to-use web-based platforms and sophisticated AI graphics editors. This paper presents our unique investigation of latent diffusion architecture design, offering a fresh and innovative perspective on this dynamic field of study. First, we describe the new architecture of Kandinsky and its details. The demo system with implemented features of the model is also described. Second, we show the experiments, carried out in terms of image generation quality and come up with the highest FID score among existing open-source models. Additionally, we present the rigorous ablation study of prior setups that we conducted, enabling us to carefully analyze and evaluate various configurations to arrive at the most effective and refined model design. Our contributions are as follows: • We present the first text-to-image architecture designed using a combination of image prior and latent diffusion. • We demonstrate experimental results comparable to the state-of-the-art (SotA) models such as Stable Diffusion, IF, and DALL-E 2, in terms of FID metric and achieve the SotA score among all existing open source models. • We provide a software implementation of the proposed state-of-the-art method for textto-image generation, and release pre-trained models, which is unique among the topperforming methods. Apache 2.0 license makes it possible to use the model for both non-commercial and commercial purposes.2 3 • We create a web image editor application that can be used for interactive generation of images by text prompts (English and Russian languages are supported) on the basis of the proposed method, and provides inpainting/outpainting functionality.4 The video demonstration is available on YouTube.5 This paper is available on arxiv under CC BY 4.0 DEED license. [1] The system is named after Wassily Kandinsky, a famous painter and an art theorist. [2] https://github.com/ai-forever/Kandinsky-2 [3] https://huggingface.co/kandinsky-community [4] https://fusionbrain.ai/en/editor [5] https://www.youtube.com/watch?v=c7zHPc59cWU Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Authors: Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 1 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 1 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 1 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction Related Work Demo System Kandinsky Architecture Experiments Results Conclusion & Limitations Ethical Considerations, Acknowledgements and References Abstract and Introduction Abstract and Introduction Related Work Related Work Demo System Demo System Kandinsky Architecture Kandinsky Architecture Experiments Experiments Results Results Conclusion & Limitations Conclusion & Limitations Ethical Considerations, Acknowledgements and References Ethical Considerations, Acknowledgements and References Abstract Text-to-image generation is a significant domain in modern computer vision and has achieved substantial improvements through the evolution of generative architectures. Among these, there are diffusion-based models that have demonstrated essential quality enhancements. These models are generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky[1], a novel exploration of latent diffusion architecture, combining the principles of the image prior models with latent diffusion techniques. The image prior model is trained separately to map text embeddings to image embeddings of CLIP. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall, the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation, and text-guided inpainting/outpainting. Additionally, we released the source code and checkpoints for the Kandinsky models. Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K dataset, marking our model as the top open-source performer in terms of measurable image generation quality. 1 Introduction In quite a short period of time, generative abilities of text-to-image models have improved substantially, providing users with photorealistic quality, near real-time inference speed, a great number of applications and features, including simple easy-to-use web-based platforms and sophisticated AI graphics editors. This paper presents our unique investigation of latent diffusion architecture design, offering a fresh and innovative perspective on this dynamic field of study. First, we describe the new architecture of Kandinsky and its details. The demo system with implemented features of the model is also described. Second, we show the experiments, carried out in terms of image generation quality and come up with the highest FID score among existing open-source models. Additionally, we present the rigorous ablation study of prior setups that we conducted, enabling us to carefully analyze and evaluate various configurations to arrive at the most effective and refined model design. Our contributions are as follows: contributions • We present the first text-to-image architecture designed using a combination of image prior and latent diffusion. • We demonstrate experimental results comparable to the state-of-the-art (SotA) models such as Stable Diffusion, IF, and DALL-E 2, in terms of FID metric and achieve the SotA score among all existing open source models. • We provide a software implementation of the proposed state-of-the-art method for textto-image generation, and release pre-trained models, which is unique among the topperforming methods. Apache 2.0 license makes it possible to use the model for both non-commercial and commercial purposes.2 3 • We create a web image editor application that can be used for interactive generation of images by text prompts (English and Russian languages are supported) on the basis of the proposed method, and provides inpainting/outpainting functionality.4 The video demonstration is available on YouTube.5 This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] The system is named after Wassily Kandinsky, a famous painter and an art theorist. [2] https://github.com/ai-forever/Kandinsky-2 [3] https://huggingface.co/kandinsky-community [4] https://fusionbrain.ai/en/editor [5] https://www.youtube.com/watch?v=c7zHPc59cWU

Russian Scientists Develop First Text-to-Image Architecture Using Image Prior, Latent Diffusion

Rusia markan cientificonakax Nayrïr Arquitectura Texto-Imagen ukar uñstayapxi, Image Prior, Difusión Latente ukamp apnaqasa

Rus alimləri Şəkildən əvvəl, gizli diffuziyadan istifadə edərək ilk mətndən şəkilə arxitektura hazırlayıblar

Ruski naučnici razvili prvu arhitekturu teksta u sliku koristeći prethodnu, latentnu difuziju slike

Científicos rusos desarrollan la primera arquitectura de texto a imagen utilizando difusión latente previa a la imagen

دانشمندان روسی اولین معماری متن به تصویر را با استفاده از تصویر قبلی، انتشار پنهان توسعه دادند.

Ilmuwan Rusia Kembangkan Arsitektur Teks-ke-Gambar Pertama Menggunakan Prioritas Gambar, Difusi Laten

ロシアの科学者が、画像の事前分布と潜在的拡散を利用した初のテキストから画像へのアーキテクチャを開発

Орус илимпоздору сүрөттүн алдындагы, жашыруун диффузияны колдонуу менен биринчи тексттен сүрөткө архитектурасын иштеп чыгышты

Saintis Rusia Membangunkan Seni Bina Teks-ke-Imej Pertama Menggunakan Imej Terdahulu, Resapan Terpendam

रूसी वैज्ञानिकहरूले छवि पूर्व, अव्यक्त प्रसार प्रयोग गरेर पहिलो पाठ-देखि-छवि वास्तुकला विकास गर्छन्

Abashakashatsi b'Abarusiya Batezimbere Umwandiko-Kuri-Ishusho Ubwubatsi Ukoresheje Ishusho Mbere, Gutandukana Bitinze

Ruskí vedci vyvinuli prvú architektúru textu na obrázok pomocou latentnej difúzie obrazu

This Open-Source Image Generator From Russian Scientists Uses A Novel Approach to Diffusion

Authors:
(1) Anton Razzhigaev, AIRI and Skoltech;
(2) Arseniy Shakhmatov, Sber AI;
(3) Anastasia Maltseva, Sber AI;
(4) Vladimir Arkhipkin, Sber AI;
(5) Igor Pavlov, Sber AI;
(6) Ilya Ryabov, Sber AI;
(7) Angelina Kuts, Sber AI;
(8) Alexander Panchenko, AIRI and Skoltech;
(9) Andrey Kuznetsov, AIRI and Sber AI;
(10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction
Related Work
Demo System
Kandinsky Architecture
Experiments
Results
Conclusion & Limitations
Ethical Considerations, Acknowledgements and References 7 Conclusion We presented Kandinsky, a system for various image generation and processing tasks based on a novel latent diffusion model. Our model yielded the SotA results among open-sourced systems. Additionally, we provided an extensive ablation study of an image prior to design choices. Our system is equipped with free-to-use interfaces in the form of Web application and Telegram messenger bot. The pre-trained models are available on Hugging Face, and the source code is released under a permissive license enabling various, including commercial, applications of the developed technology. In future research, our goal is to investigate the potential of the latest image encoders. We plan to explore the development of more efficient UNet architectures for text-to-image tasks and focus on improving the understanding of textual prompts. Additionally, we aim to experiment with generating images at higher resolutions and to investigate new features extending the model: local image editing by a text prompt, attention reweighting, physics-based generation control, etc. The robustness against generating abusive content remains a crucial concern, warranting the exploration of real-time moderation layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive, outputs. 8 Limitations The current system produces images that appear natural, however, additional research can be conducted to (1) enhance the semantic coherence between the input text and the generated image, and (2) to improve the absolute values of FID and image quality based on human evaluations. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Authors: Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction Related Work Demo System Kandinsky Architecture Experiments Results Conclusion & Limitations Ethical Considerations, Acknowledgements and References Abstract and Introduction Abstract and Introduction Related Work Related Work Demo System Demo System Kandinsky Architecture Kandinsky Architecture Experiments Experiments Results Results Conclusion & Limitations Conclusion & Limitations Ethical Considerations, Acknowledgements and References Ethical Considerations, Acknowledgements and References 7 Conclusion We presented Kandinsky, a system for various image generation and processing tasks based on a novel latent diffusion model. Our model yielded the SotA results among open-sourced systems. Additionally, we provided an extensive ablation study of an image prior to design choices. Our system is equipped with free-to-use interfaces in the form of Web application and Telegram messenger bot. The pre-trained models are available on Hugging Face, and the source code is released under a permissive license enabling various, including commercial, applications of the developed technology. In future research, our goal is to investigate the potential of the latest image encoders. We plan to explore the development of more efficient UNet architectures for text-to-image tasks and focus on improving the understanding of textual prompts. Additionally, we aim to experiment with generating images at higher resolutions and to investigate new features extending the model: local image editing by a text prompt, attention reweighting, physics-based generation control, etc. The robustness against generating abusive content remains a crucial concern, warranting the exploration of real-time moderation layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive, outputs. 8 Limitations The current system produces images that appear natural, however, additional research can be conducted to (1) enhance the semantic coherence between the input text and the generated image, and (2) to improve the absolute values of FID and image quality based on human evaluations. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Russian Scientists Unveil Open-Source Image Generator With a Groundbreaking Diffusion Method

A Reference List to Learn More About Image Editing, Video Editing, and Diffusion Models

Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Method 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 6 Discussion 7 Acknowledgement and References A Implementation Details 3 PRELIMINARIES This paper is available on arxiv under CC BY 4.0 DEED DEED license. Authors:
(1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution;
(2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution;
(3) Shai Bagon, Weizmann Institute of Science;
(4) Tali Dekel, Weizmann Institute of Science. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Method 4 Method 4.1 Key Sample and Joint Editing 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 4.2 Edit Propagation Via TokenFlow 5 Results 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 5.3 Ablation Study 6 Discussion 6 Discussion 7 Acknowledgement and References 7 Acknowledgement and References A Implementation Details A Implementation Details 3 PRELIMINARIES This paper is available on arxiv under CC BY 4.0 DEED DEED license. This paper is available on arxiv under CC BY 4.0 DEED DEED license. available on arxiv Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science. Authors: Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science.

Diffusion Models and Stable Diffusion Explained

Table of Links Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 2.3 Diffusion Models Diffusion models have also demonstrated their powerful generative performances in speech synthesis. Grad-TTS [63] first introduced a score-based decoder to generate a Melspectrogram, and Diff-VC demonstrated the high-adaptation performance of diffusion models in zero-shot voice conversion scenarios. DiffSinger achieved a state-of-the-art performance in SVS task by generating a high-quality singing voice with a powerful adaptation performance. DDDM-VC [10] significantly improved speech representation disentangle [3] and voice conversion performance by a disentnalged denoising diffusion model and prior mixup. Diff-HierVC [11] introduced a hierarchical voice style transfer frameworks that generates pitch contour and voice hierarchically based on diffusion models. Guided-TTS [33] and Guided-TTS 2 [38] have also shown good speaker adaptation performance for TTS. UnitSpeech [32] introduced a unit-based speech synthesis with diffusion models. Furthermore, recent studies utilized a diffusion model in latent representation. Naturalspeech 2 [70] and HiddenSinger [24] utilized the acoustic representation of an audio autoencoder as a latent representation, and developed a conditional latent diffusion model for speech or singing voice synthesis. StyleTTS 2 [54] proposed a style latent diffusion for style adaptation. Although all the above models have shown powerful adaptation performance, they have a slow inference speed for their iterative generation manner. To reduce the inference speed, CoMoSpeech [87] and Multi-GradSpeech [83] adopted a consistency model for a diffusion-based TTS model. Recently, VoiceBox [43] and P-Flow [39] utilized flow matching with optimal transport for fast sampling. However, these models still have a traininginference mismatch problem that arises from two-stage speech synthesis frameworks and they are vulnerable to noisy target voice prompt. 2.4 Zero-shot Voice Cloning Zero-shot learning [81] for voice cloning is a task to synthesize speech with a novel speaker, which has not been previously observed during training. A majority of the studies on voice cloning [25], [74] focused on cloning the voice styles, such as timbre and environment, and speaking styles, such as prosody and pronunciation. [72] presented a reference encoder for prosody modeling, and GST [80] utilized a learnable token for style modeling from the reference speech or manually control. [51] proposed a fine-grained prosody control for expressive speech synthesis from reference speech. Multi-SpectroGAN [50] utilized adversarial feedback and a mixup strategy for an expressive and diverse zero-shot TTS. Meta-StyleSpeech [59] introduced meta-learning for style modeling, and GenerSpeech [22] utilized a mix-style layer normalization for better generalization on the out-ofdomain style transfer. PVAE-TTS [44] utilized a progressive style adaptation for high-quality zero-shot TTS. AdaSpeech [8] introduced adaptive layer normalization for adaptive speech synthesis. YourTTS [7] trained the VITS [35] with a speaker encoder and Grad-StyleSpeech [29] utilized a styleconditioned prior on a score-based Mel-spectrogram decoder for better adaptive TTS. Built upon VQTTS [17], TN-VQTTS [18] introduce a timbre-normalized vector-quantized acoustic features for speaking style and timbre transfer. Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [19], [52], [84]. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors:
(1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.4 Speech Super-resolution 3.5 Model Architecture 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.4 Evaluation Metrics 5.5 Ablation Study 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 7 Conclusion, Acknowledgement and References 2.3 Diffusion Models Diffusion models have also demonstrated their powerful generative performances in speech synthesis. Grad-TTS [63] first introduced a score-based decoder to generate a Melspectrogram, and Diff-VC demonstrated the high-adaptation performance of diffusion models in zero-shot voice conversion scenarios. DiffSinger achieved a state-of-the-art performance in SVS task by generating a high-quality singing voice with a powerful adaptation performance. DDDM-VC [10] significantly improved speech representation disentangle [3] and voice conversion performance by a disentnalged denoising diffusion model and prior mixup. Diff-HierVC [11] introduced a hierarchical voice style transfer frameworks that generates pitch contour and voice hierarchically based on diffusion models. Guided-TTS [33] and Guided-TTS 2 [38] have also shown good speaker adaptation performance for TTS. UnitSpeech [32] introduced a unit-based speech synthesis with diffusion models. Furthermore, recent studies utilized a diffusion model in latent representation. Naturalspeech 2 [70] and HiddenSinger [24] utilized the acoustic representation of an audio autoencoder as a latent representation, and developed a conditional latent diffusion model for speech or singing voice synthesis. StyleTTS 2 [54] proposed a style latent diffusion for style adaptation. Although all the above models have shown powerful adaptation performance, they have a slow inference speed for their iterative generation manner. To reduce the inference speed, CoMoSpeech [87] and Multi-GradSpeech [83] adopted a consistency model for a diffusion-based TTS model. Recently, VoiceBox [43] and P-Flow [39] utilized flow matching with optimal transport for fast sampling. However, these models still have a traininginference mismatch problem that arises from two-stage speech synthesis frameworks and they are vulnerable to noisy target voice prompt. 2.4 Zero-shot Voice Cloning Zero-shot learning [81] for voice cloning is a task to synthesize speech with a novel speaker, which has not been previously observed during training. A majority of the studies on voice cloning [25], [74] focused on cloning the voice styles, such as timbre and environment, and speaking styles, such as prosody and pronunciation. [72] presented a reference encoder for prosody modeling, and GST [80] utilized a learnable token for style modeling from the reference speech or manually control. [51] proposed a fine-grained prosody control for expressive speech synthesis from reference speech. Multi-SpectroGAN [50] utilized adversarial feedback and a mixup strategy for an expressive and diverse zero-shot TTS. Meta-StyleSpeech [59] introduced meta-learning for style modeling, and GenerSpeech [22] utilized a mix-style layer normalization for better generalization on the out-ofdomain style transfer. PVAE-TTS [44] utilized a progressive style adaptation for high-quality zero-shot TTS. AdaSpeech [8] introduced adaptive layer normalization for adaptive speech synthesis. YourTTS [7] trained the VITS [35] with a speaker encoder and Grad-StyleSpeech [29] utilized a styleconditioned prior on a score-based Mel-spectrogram decoder for better adaptive TTS. Built upon VQTTS [17], TN-VQTTS [18] introduce a timbre-normalized vector-quantized acoustic features for speaking style and timbre transfer. Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [19], [52], [84]. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Authors: Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author.

Diffusion Models and Zero-shot Voice Cloning in Speech Synthesis: How Do They Fare?

Authors:
(1) Albert Gu, Machine Learning Department, Carnegie Mellon University and with equal contribution;
(2) Tri Dao, Department of Computer Science, Princeton University and with equal contribution. Table of Links Abstract and 1 Introduction 2 State Space Models 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3.2 Improving SSMs with Selection 3.3 Efficient Implementation of Selective SSMs 3.4 A Simplified SSM Architecture 3.5 Properties of Selection Mechanisms 3.6 Additional Model Details 4 Empirical Evaluation and 4.1 Synthetic Tasks 4.2 Language Modeling 4.3 DNA Modeling 4.4 Audio Modeling and Generation 4.5 Speed and Memory Benchmarks 4.6 Model Ablations 5 Discussion 6 Conclusion and References A Discussion: Selection Mechanism B Related Work C Mechanics of Selective SSMs D Hardware-aware Algorithm For Selective SSMs E Experimental Details and Additional Results 3.6 Additional Model Details Real vs. Complex. Most prior SSMs use complex numbers in their state ℎ, which is necessary for strong performance on many tasks (Gu, Goel, and Ré 2022). However, it has been empirically observed that completely real-valued SSMs seem to work fine, and possibly even better, in some settings (Ma et al. 2023). We use real values as the default, which work well for all but one of our tasks; we hypothesize that the complex-real tradeoff is related to the continuous-discrete spectrum in data modalities, where complex numbers are helpful for continuous modalities (e.g. audio, video) but not discrete (e.g. text, DNA). Initialization. Most prior SSMs also suggest special initializations, particularly in the complex-valued case, which can help in several settings such as low-data regimes. Our default initialization for the complex case is S4D-Lin and for the real case is S4D-Real (Gu, Gupta, et al. 2022), which is based on the HIPPO theory (Gu, Dao, et al. 2020). These define the n-th element of A as −1∕2 + ni and −(n + 1) respectively. However, we expect many initializations to work fine, particularly in the large-data and real-valued SSM regimes; some ablations are considered in Section 4.6. Remark 3.1. For brevity in our experimental results, we sometimes abbreviate selective SSMs as S6 models, because they are S4 models with a selection mechanism and computed with a scan. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Albert Gu, Machine Learning Department, Carnegie Mellon University and with equal contribution; (2) Tri Dao, Department of Computer Science, Princeton University and with equal contribution. Authors: Authors: (1) Albert Gu, Machine Learning Department, Carnegie Mellon University and with equal contribution; (2) Tri Dao, Department of Computer Science, Princeton University and with equal contribution. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 State Space Models 2 State Space Models 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3.2 Improving SSMs with Selection 3.2 Improving SSMs with Selection 3.3 Efficient Implementation of Selective SSMs 3.3 Efficient Implementation of Selective SSMs 3.4 A Simplified SSM Architecture 3.4 A Simplified SSM Architecture 3.5 Properties of Selection Mechanisms 3.5 Properties of Selection Mechanisms 3.6 Additional Model Details 3.6 Additional Model Details 4 Empirical Evaluation and 4.1 Synthetic Tasks 4 Empirical Evaluation and 4.1 Synthetic Tasks 4.2 Language Modeling 4.2 Language Modeling 4.3 DNA Modeling 4.3 DNA Modeling 4.4 Audio Modeling and Generation 4.4 Audio Modeling and Generation 4.5 Speed and Memory Benchmarks 4.5 Speed and Memory Benchmarks 4.6 Model Ablations 4.6 Model Ablations 5 Discussion 5 Discussion 6 Conclusion and References 6 Conclusion and References A Discussion: Selection Mechanism A Discussion: Selection Mechanism B Related Work B Related Work C Mechanics of Selective SSMs C Mechanics of Selective SSMs D Hardware-aware Algorithm For Selective SSMs D Hardware-aware Algorithm For Selective SSMs E Experimental Details and Additional Results E Experimental Details and Additional Results 3.6 Additional Model Details Real vs. Complex. Most prior SSMs use complex numbers in their state ℎ, which is necessary for strong performance on many tasks (Gu, Goel, and Ré 2022). However, it has been empirically observed that completely real-valued SSMs seem to work fine, and possibly even better, in some settings (Ma et al. 2023). We use real values as the default, which work well for all but one of our tasks; we hypothesize that the complex-real tradeoff is related to the continuous-discrete spectrum in data modalities, where complex numbers are helpful for continuous modalities (e.g. audio, video) but not discrete (e.g. text, DNA). Real vs. Complex. Initialization . Most prior SSMs also suggest special initializations, particularly in the complex-valued case, which can help in several settings such as low-data regimes. Our default initialization for the complex case is S4D-Lin and for the real case is S4D-Real (Gu, Gupta, et al. 2022), which is based on the HIPPO theory (Gu, Dao, et al. 2020). These define the n-th element of A as −1∕2 + ni and −(n + 1) respectively. However, we expect many initializations to work fine, particularly in the large-data and real-valued SSM regimes; some ablations are considered in Section 4.6. Initialization Remark 3.1. For brevity in our experimental results, we sometimes abbreviate selective SSMs as S6 models, because they are S4 models with a selection mechanism and computed with a scan. Remark 3.1. For brevity in our experimental results, we sometimes abbreviate selective SSMs as S6 models, because they are S4 models with a selection mechanism and computed with a scan. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

The Key Differences Between Real and Complex-Valued State Space Models

Table of Links Abstract and 1. Introduction Background and Related Work


Threat Model


Robust Style Mimicry


Experimental Setup


Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis


Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples B. Robust Mimicry Generations C. Detailed Results D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 F. Findings on Mist v2 G. Methods for Style Mimicry H. Existing Style Mimicry Protections I. Robust Mimicry Methods J. Experimental Setup K. User Study L. Compute Resources J Experimental Setup This section describes our general experimental setup and specifies the settings and hyperparameters of the methods we use. When possible, we use default values from the machine learning literature. For implementation details see our official repository: https://github.com/ethz-spylab/ robust-style-mimicry J.1 Style Mimicry Experimental Details As described in Section 3, our threat model considers style mimicry with a latent diffusion text-toimage model M that is finetuned on a set of images X in a style S. This section specifies our choices for model M, images X, style S, the hyperparameters for finetuning M, and the hyperparameters for generating images with the finetuned model. Where possible, we try to replicate the style mimicry setup used by Shan et al. to evaluate Glaze, and highlight any differences. Model We use Stable Diffusion version 2.1 (Stability AI, 2022), the same model used to optimize the protections we evaluate (Shan et al., 2023a; Liang et al., 2023; Van Le et al., 2023). J.2 Protections Experimental Details Next, we describe specific hyperparameters we use to reproduce each of the protections. J.2.1 Anti-DreamBooth J.2.2 Mistϕ J.2.3 Glaze The Glaze authors were unable to share a codebase upon request. We thus use their publicly released Windows application binary. We use the latest available version of Glaze, v1.1.1. We set Intensity to High and Render Quality to Slowest, to obtain the strongest protections. Appendix E includes qualitative results on an updated version released after we concluded our user study. J.3 Robust Mimicry Methods Experimental Details J.3.1 Gaussian noising We manually tune the Gaussian noising strength to σ2 = 0.05. J.3.2 DiffPure We use conditional DiffPure with the best-performing publicly available image generation diffusion model, Stable Diffusion XL 1.0 (SDXL) (Podell et al., 2023). We implement conditional DiffPure using the HuggingFace AutoPipelineForImage2Image pipeline. We use classifier-free guidance scale guidance scale = 7.5 with prompt P = Cx for image x. We manually tune the number of diffusion timesteps t via the strength pipeline argument to strength = 0.2. J.3.3 IMPRESS++ Authors:
(1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch);
(2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch);
(3) Nicholas Carlini, Google DeepMind;
(4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. [8] www.artstation.com [9] @nulevoy is the first ArtStation artist that we experimented with. In our experiments, we found “nulevoy” a suitable choice for the special word w∗ and use it for all artists. We check that all of nulevoy’s images are published after the release date of LAION-5B to ensure that SD 2.1 has no prior knowledge about nulevoy’s style. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Threat Model Robust Style Mimicry Experimental Setup Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Background and Related Work Background and Related Work Background and Related Work Threat Model Threat Model Threat Model Robust Style Mimicry Robust Style Mimicry Robust Style Mimicry Experimental Setup Experimental Setup Experimental Setup Results 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis Results Results 6.1 Main Findings: All Protections are Easily Circumvented 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis 6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples A. Detailed Art Examples B. Robust Mimicry Generations B. Robust Mimicry Generations C. Detailed Results C. Detailed Results D. Differences with Glaze Finetuning D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 E. Findings on Glaze 2.0 F. Findings on Mist v2 F. Findings on Mist v2 G. Methods for Style Mimicry G. Methods for Style Mimicry H. Existing Style Mimicry Protections H. Existing Style Mimicry Protections I. Robust Mimicry Methods I. Robust Mimicry Methods J. Experimental Setup J. Experimental Setup K. User Study K. User Study L. Compute Resources L. Compute Resources J Experimental Setup This section describes our general experimental setup and specifies the settings and hyperparameters of the methods we use. When possible, we use default values from the machine learning literature. For implementation details see our official repository: https://github.com/ethz-spylab/ robust-style-mimicry J.1 Style Mimicry Experimental Details As described in Section 3, our threat model considers style mimicry with a latent diffusion text-toimage model M that is finetuned on a set of images X in a style S. This section specifies our choices for model M, images X, style S, the hyperparameters for finetuning M, and the hyperparameters for generating images with the finetuned model. Where possible, we try to replicate the style mimicry setup used by Shan et al. to evaluate Glaze, and highlight any differences. Model We use Stable Diffusion version 2.1 (Stability AI, 2022), the same model used to optimize the protections we evaluate (Shan et al., 2023a; Liang et al., 2023; Van Le et al., 2023). Model J.2 Protections Experimental Details Next, we describe specific hyperparameters we use to reproduce each of the protections. J.2.1 Anti-DreamBooth J.2.1 Anti-DreamBooth J.2.2 Mistϕ J.2.2 Mistϕ J.2.3 Glaze J.2.3 Glaze The Glaze authors were unable to share a codebase upon request. We thus use their publicly released Windows application binary. We use the latest available version of Glaze, v1.1.1. We set Intensity to High and Render Quality to Slowest, to obtain the strongest protections. Appendix E includes qualitative results on an updated version released after we concluded our user study. J.3 Robust Mimicry Methods Experimental Details J.3.1 Gaussian noising J.3.1 Gaussian noising We manually tune the Gaussian noising strength to σ2 = 0.05. J.3.2 DiffPure J.3.2 DiffPure We use conditional DiffPure with the best-performing publicly available image generation diffusion model, Stable Diffusion XL 1.0 (SDXL) (Podell et al., 2023). We implement conditional DiffPure using the HuggingFace AutoPipelineForImage2Image pipeline. We use classifier-free guidance scale guidance scale = 7.5 with prompt P = Cx for image x. We manually tune the number of diffusion timesteps t via the strength pipeline argument to strength = 0.2. J.3.3 IMPRESS++ J.3.3 IMPRESS++ Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). Authors: Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv [8] www.artstation.com [9] @nulevoy is the first ArtStation artist that we experimented with. In our experiments, we found “nulevoy” a suitable choice for the special word w∗ and use it for all artists. We check that all of nulevoy’s images are published after the release date of LAION-5B to ensure that SD 2.1 has no prior knowledge about nulevoy’s style.

How We Evaluated Different AI Techniques for Mimicking Artistic Styles

In the fast-paced world of blockchain technology, I often find myself wearing two distinct hats – that of a Smart Contract Researcher/Developer and a Technical Blogger sharing insights with the community. This unique position has led me to some interesting revelations about how we adapt to modern tools while maintaining authenticity in our craft. As I delve deep into smart contract development each day, my mind is consumed by the intricate details of secure code implementation. Every line of code demands unwavering attention, every function requires thorough testing, and each deployment must be preceded by rigorous security audits. It's a world where precision isn't just a virtue – it's an absolute necessity. My recent work with Vyper on both Rootstock and Hedera platforms perfectly illustrates this reality. These projects weren't just about writing functional code; they required an intimate understanding of the platforms' nuances, careful consideration of gas optimization, and multiple rounds of security checks before deployment. The stakes in smart contract development are incredibly high – a single overlooked vulnerability could lead to substantial financial losses. Yet, parallel to this technical journey, I feel a strong responsibility to share my experiences and insights with the broader blockchain community. This is where an interesting dilemma emerges. After spending a decent amount of time ensuring my smart contracts are secure and efficient, I often find myself mentally drained when it comes to documenting and sharing these experiences through blog posts. This is where I made a decision that might raise some eyebrows in the development community – I chose to embrace AI as my writing partner. Not because I lack the ability to articulate my thoughts, but because I recognized an opportunity to optimize my workflow. While I pour my energy into the technical aspects of smart contract development, AI helps me transform my technical notes and documentation into engaging blog posts. The authenticity of this approach might be questioned by some, but I view it through a practical lens. Every technical detail, every implementation decision, and every learning comes directly from my hands-on experience. AI simply helps me structure these insights into readable content, much like how we use code formatters and linters to enhance our development workflow. Think about it – when you're deep in the trenches of smart contract development, juggling security considerations, gas optimization, and cross-chain compatibility, would you rather spend hours crafting the perfect blog post, or focus on ensuring your smart contracts are bulletproof? For me, the answer became clear when I realized that AI could help me maintain a consistent blogging schedule without compromising the quality of my technical work. My experience writing about Vyper implementations on Rootstock and Hedera serves as a perfect example. While I invested significant time in the technical implementation, ensuring every aspect of the smart contracts met the highest standards of security and efficiency, AI helped me transform this technical work into accessible blog posts that could benefit the wider community. The beauty of this approach lies in its efficiency. The technical knowledge, the practical insights, and the real-world experiences are all mine. AI simply serves as a writing assistant, helping me communicate these insights more effectively while allowing me to maintain my focus on what I do best – developing secure, efficient smart contracts. So here's my truth as a Smart Contract Researcher and Developer in the age of AI: I choose to leverage modern tools to share my knowledge more efficiently, just as I leverage development tools to write better code. The authenticity doesn't lie in whether every word was manually typed, but in the genuine experiences and insights being shared. I leave it to you, my readers, to judge this approach. What matters more – the time spent crafting each sentence, or the value and accuracy of the technical insights being shared? How do you balance the demands of technical excellence with the need to contribute to our community's collective knowledge? Share your thoughts and experiences – I'm curious to hear your perspective on this modern developer's dilemma. You can find these 2 blog posts here In the fast-paced world of blockchain technology, I often find myself wearing two distinct hats – that of a Smart Contract Researcher/Developer and a Technical Blogger sharing insights with the community. This unique position has led me to some interesting revelations about how we adapt to modern tools while maintaining authenticity in our craft. As I delve deep into smart contract development each day, my mind is consumed by the intricate details of secure code implementation. Every line of code demands unwavering attention, every function requires thorough testing, and each deployment must be preceded by rigorous security audits. It's a world where precision isn't just a virtue – it's an absolute necessity. My recent work with Vyper on both Rootstock and Hedera platforms perfectly illustrates this reality. These projects weren't just about writing functional code; they required an intimate understanding of the platforms' nuances, careful consideration of gas optimization, and multiple rounds of security checks before deployment. The stakes in smart contract development are incredibly high – a single overlooked vulnerability could lead to substantial financial losses. Yet, parallel to this technical journey, I feel a strong responsibility to share my experiences and insights with the broader blockchain community. This is where an interesting dilemma emerges. After spending a decent amount of time ensuring my smart contracts are secure and efficient, I often find myself mentally drained when it comes to documenting and sharing these experiences through blog posts. This is where I made a decision that might raise some eyebrows in the development community – I chose to embrace AI as my writing partner. Not because I lack the ability to articulate my thoughts, but because I recognized an opportunity to optimize my workflow. While I pour my energy into the technical aspects of smart contract development, AI helps me transform my technical notes and documentation into engaging blog posts. The authenticity of this approach might be questioned by some, but I view it through a practical lens. Every technical detail, every implementation decision, and every learning comes directly from my hands-on experience. AI simply helps me structure these insights into readable content, much like how we use code formatters and linters to enhance our development workflow. Think about it – when you're deep in the trenches of smart contract development, juggling security considerations, gas optimization, and cross-chain compatibility, would you rather spend hours crafting the perfect blog post, or focus on ensuring your smart contracts are bulletproof? For me, the answer became clear when I realized that AI could help me maintain a consistent blogging schedule without compromising the quality of my technical work. My experience writing about Vyper implementations on Rootstock and Hedera serves as a perfect example. While I invested significant time in the technical implementation, ensuring every aspect of the smart contracts met the highest standards of security and efficiency, AI helped me transform this technical work into accessible blog posts that could benefit the wider community. The beauty of this approach lies in its efficiency. The technical knowledge, the practical insights, and the real-world experiences are all mine. AI simply serves as a writing assistant, helping me communicate these insights more effectively while allowing me to maintain my focus on what I do best – developing secure, efficient smart contracts. So here's my truth as a Smart Contract Researcher and Developer in the age of AI: I choose to leverage modern tools to share my knowledge more efficiently, just as I leverage development tools to write better code. The authenticity doesn't lie in whether every word was manually typed, but in the genuine experiences and insights being shared. I leave it to you, my readers, to judge this approach. What matters more – the time spent crafting each sentence, or the value and accuracy of the technical insights being shared? How do you balance the demands of technical excellence with the need to contribute to our community's collective knowledge? Share your thoughts and experiences – I'm curious to hear your perspective on this modern developer's dilemma. You can find these 2 blog posts here You can find these 2 blog posts here

Developers Shouldn't Rely on AI for Code, But Blogging Is a Whole Different Matter

Authors:
(1) Florian Cafiero (ORCID 0000-0002-1951-6942), Sciences Po, Medialab;
(2) Jean-Baptiste Camps (ORCID 0000-0003-0385-7037), Ecole nationale des chartes, Universite Paris, Sciences & Lettres. Table of Links Abstract and Introduction Why work on QAnon? Specificities and social impact Who is Q? The theories put to test Authorship attribution Results Discussion Corpus constitution Quotes of authors outside of the corpus have been Definition of two subcorpus: dealing with generic difference and an imbalanced dataset The genre of “Q drops”: a methodological challenge Detecting style changes: rolling stylometry Ethical statement, Acknowledgements, and References Definition of two subcorpus: dealing with generic difference and an imbalanced dataset The difficulties in data collection for a variety of individual and profiles, as well as the ubiquity of deleted content, not recoverable to us, forces us to adopt a a dual approach, and to build two corpora: large corpus in which we include the larger number of candidates, whatever the number of samples available to us and the genre of said samples. It contains everything described in the contains everything described in the Corpus constitution section above; controlled corpus in which we removed authors for which only too cross-genre and/or too few samples are available, and do not include training material that is too different from the rest (in particular, books). It is the same as the previous one, minus • interviews transcripts (Michael F., Paul F.); • a book by Paul F.; • the small amount of available data for Courtney T. and Roger S. In both cases, due to the limitations in data collection and available material, the quantity of training material is imbalanced between authors, a potential problem in machine learning. To counter this effect, we used class weights during training, where errors for a given class are penalised not always by one, but by a specific weight inversely proportional to class size, where the weight for class i is computed as where N is the total number of samples, and C the total number of unique classes and ni the number of samples in class i. Xe used the sklearn ‘balance’ implementation (Pedregosa et al., 2011). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Florian Cafiero (ORCID 0000-0002-1951-6942), Sciences Po, Medialab; (2) Jean-Baptiste Camps (ORCID 0000-0003-0385-7037), Ecole nationale des chartes, Universite Paris, Sciences & Lettres. Authors: Authors: (1) Florian Cafiero (ORCID 0000-0002-1951-6942), Sciences Po, Medialab; (2) Jean-Baptiste Camps (ORCID 0000-0003-0385-7037), Ecole nationale des chartes, Universite Paris, Sciences & Lettres. Table of Links Abstract and Introduction Abstract and Introduction Why work on QAnon? Specificities and social impact Why work on QAnon? Specificities and social impact Who is Q? The theories put to test Who is Q? The theories put to test Authorship attribution Authorship attribution Results Results Discussion Discussion Corpus constitution Corpus constitution Quotes of authors outside of the corpus have been Quotes of authors outside of the corpus have been Definition of two subcorpus: dealing with generic difference and an imbalanced dataset Definition of two subcorpus: dealing with generic difference and an imbalanced dataset The genre of “Q drops”: a methodological challenge The genre of “Q drops”: a methodological challenge Detecting style changes: rolling stylometry Detecting style changes: rolling stylometry Ethical statement, Acknowledgements, and References Ethical statement, Acknowledgements, and References Definition of two subcorpus: dealing with generic difference and an imbalanced dataset The difficulties in data collection for a variety of individual and profiles, as well as the ubiquity of deleted content, not recoverable to us, forces us to adopt a a dual approach, and to build two corpora: large corpus in which we include the larger number of candidates, whatever the number of samples available to us and the genre of said samples. It contains everything described in the contains everything described in the Corpus constitution section above; large corpus controlled corpus in which we removed authors for which only too cross-genre and/or too few samples are available, and do not include training material that is too different from the rest (in particular, books). It is the same as the previous one, minus controlled corpus • interviews transcripts (Michael F., Paul F.); • a book by Paul F.; • the small amount of available data for Courtney T. and Roger S. In both cases, due to the limitations in data collection and available material, the quantity of training material is imbalanced between authors, a potential problem in machine learning. To counter this effect, we used class weights during training, where errors for a given class are penalised not always by one, but by a specific weight inversely proportional to class size, where the weight for class i is computed as where N is the total number of samples, and C the total number of unique classes and ni the number of samples in class i. Xe used the sklearn ‘balance’ implementation (Pedregosa et al., 2011). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about DRIFF

Driff’s Stories on HackerNoon

Driff's News & Mentions