Nearshore talent network matching devs with jobs

Torc

torc

How to Improve the Parallelization of Torch Dataloaders Using Torch.multiprocessing

কিভাবে Torch.multiprocessing ব্যবহার করে টর্চ ডেটালোডারের সমান্তরালকরণ উন্নত করা যায়

So verbessern Sie die Parallelisierung von Torch-Dataloadern mit Torch.multiprocessing

Cómo mejorar la paralelización de los cargadores de datos de Torch mediante el multiprocesamiento de Torch.

Comment améliorer la parallélisation des chargeurs de données Torch à l'aide de Torch.multiprocessing

Torch.multiprocessing का उपयोग करके Torch डेटालोडर्स के समानांतरीकरण में सुधार कैसे करें

Torch.multiprocessing を使用して Torch データローダーの並列化を改善する方法

Torch.multiprocessing을 사용하여 Torch 데이터 로더의 병렬화를 개선하는 방법

Como melhorar a paralelização de dataloaders Torch usando Torch.multiprocessing

Как улучшить распараллеливание загрузчиков данных Torch с помощью Torch.multiprocessing

Torch.multiprocessing Kullanılarak Torch Veri Yükleyicilerinin Paralelleştirilmesi Nasıl Geliştirilir?

Cách cải thiện tính năng song song hóa của bộ tải dữ liệu Torch bằng Torch.multiprocessing

如何使用 Torch.multiprocessing 提高 Torch 数据加载器的并行化

Computer vision is a field experiencing significant growth and has numerous practical applications, ranging from self-driving cars to facial recognition systems. However, one of the major challenges in this field is obtaining high-quality datasets to train machine learning models. To address this challenge, torchvision provides access to pre-built datasets, models and transforms specifically designed for computer vision tasks. Torchvision also supports both CPU and GPU acceleration, making it a flexible and powerful tool for developing computer vision applications. What are “Torchvision Datasets”? Torchvision datasets are collections of popular datasets commonly used in computer vision for developing and testing machine learning models. With torchvision datasets, developers can train and test their machine learning models on a range of tasks, such as image classification, object detection, and segmentation. The datasets are also preprocessed, labeled and organized into formats that can be easily loaded and used. List of the Torchvision Datasets MNIST CIFAR-10 CIFAR-100 ImageNet COCO Fashion-MNIST SVHN STL-10 CelebA Pascal VOC Places365 1. MNIST This torchvision dataset is popular and widely used in the fields of machine learning and computer vision. It consists of 70,000 grayscale images of handwritten digits 0–9, with 60,000 images for training and 10,000 for testing. Each image is 28x28 pixels in size and has a corresponding label denoting which digits it represents. To access this dataset, you can download it directly from or load the dataset using torchvision: Kaggle import torchvision.datasets as datasets
# Load the training dataset
train_dataset = datasets.MNIST(root='data/', train=True, transform=None, download=True)
# Load the testing dataset
test_dataset = datasets.MNIST(root='data/', train=False, transform=None, download=True) Code for loading MNIST dataset using PyTorch torchvision package. Retrieved from . on 20/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST 2. CIFAR-10 The CIFAR-10 dataset consists of 60,000 32x32 colour images in 10 classes, with 6,000 images per class. It has a total of 50,000 training images and 10,000 test images which is further divided into five training batches and one test batch, each with 10,000 images. This dataset can be downloaded from , or loaded here using torchvision: Kaggle import torch
import torchvision
import torchvision.transforms as transforms
 
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
 
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                 	                   download=True, transform=transform)
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
 
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2) Note that you can adjust the batch size and number of worker processes for the data loaders as needed. Code for loading CIFAR-10 dataset using PyTorch torchvision package. Retrieved from . on 20/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.CIFAR10.html#torchvision.datasets.CIFAR10 3. CIFAR-100 The CIFAR-100 dataset has 60,000(50,000 training images and 10,000 test images) 32x32 colour images in 100 classes, with 600 images per class. The 100 classes are grouped into 20 super-classes, with a fine label to denote its class and a coarse label to represent the super-class that it belongs to. To download the torchvision dataset from Kaggle, please visit the Kaggle and follow the instructions provided there. Alternatively, if you prefer to load the dataset using the torchvision library, you can do this: website import torchvision.datasets as datasets
import torchvision.transforms as transforms
# Define transform to normalize data
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])
 
# Load CIFAR-100 train and test datasets
trainset = datasets.CIFAR100(root='./data', train=True, download=True, transform=transform)
testset = datasets.CIFAR100(root='./data', train=False, download=True, transform=transform)
 
# Create data loaders for train and test datasets
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False) Code for loading CIFAR-100 dataset using PyTorch torchvision package. Retrieved from . on 20/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.CIFAR100.html#torchvision.datasets.CIFAR100 4. ImageNet The ImageNet dataset in torchvision contains approximately 1.2 million training images, 50,000 validation images and 100,000 test images. Each image in the dataset is labeled with one of the 1,000 categories such as "cat," "dog," "car,", "airplane" etc. To download this torchvision dataset, you have to visit the or load in torchvision: website import torchvision.datasets as datasets
import torchvision.transforms as transforms
 
# Set the path to the ImageNet dataset on your machine
data_path = "/path/to/imagenet"
 
# Create the ImageNet dataset object with custom options
imagenet_train = datasets.ImageNet(
	root=data_path,
	split='train',
    transform=transforms.Compose([
        transforms.Resize(256),
        transforms.RandomCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225])
	]),
	download=False
)
 
imagenet_val = datasets.ImageNet(
	root=data_path,
	split='val',
    transform=transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225])
	]),
	download=False
)
 
# Print the number of images in the training and validation sets
print("Number of images in the training set:", len(imagenet_train))
print("Number of images in the validation set:", len(imagenet_val)) Code for loading ImageNet dataset using PyTorch torchvision package. Retrieved from . on 21/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageNet.html#torchvision.datasets.ImageNet 5. MS Coco The Microsoft Common Objects in Context(MS Coco) dataset contains 328,000 high-quality visual images of everyday objects and humans, often used as a standard to compare the performance of algorithms in real-time object detection. To download this torchvision dataset, please visit the or load in torchvision: website import torch
from torchvision import datasets, transforms
# Define transformation
transform = transforms.Compose([
   transforms.Resize(256),
   transforms.CenterCrop(224),
   transforms.ToTensor(),
   transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
])
 
# Load training dataset
train_dataset = datasets.CocoDetection(root='/path/to/dataset/train2017',
                                       annFile='/path/to/dataset/annotations/instances_train2017.json',
                                       transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
 
# Load validation dataset
val_dataset = datasets.CocoDetection(root='/path/to/dataset/val2017',
                                     annFile='/path/to/dataset/annotations/instances_val2017.json',
                                     transform=transform)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=32, shuffle=False) Make sure to replace the /path/to/dataset placeholders with the actual path to your dataset directory. Also, adjust the batch_size parameter to fit your needs. Code for loading MS Coco dataset using PyTorch torchvision package. Retrieved from . https://pytorch.org/vision/stable/generated/torchvision.datasets.CocoDetection.html#torchvision.datasets.CocoDetection on 21/3/2023 6. Fashion-MNIST The Fashion MNIST dataset was created by Zalando Research as a replacement for the original MNIST dataset. The Fashion MNIST dataset consists of 70,000 grayscale images (training set of 60,000 and a test set of 10,000) of clothing items. The images are 28x28 pixels in size and represent 10 different classes of clothing items, including ­T-shirts/tops, trousers, pullovers, dresses, coats, sandals, shirts, sneakers, bags, and ankle boots. It is similar to the original MNIST dataset, but with more challenging classification tasks due to the greater complexity and variety of the clothing items. This torchvision dataset can be downloaded from , or loaded using this code: Kaggle import torch
import torchvision
import torchvision.transforms as transforms

# Define transformations
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5,), (0.5,))])
# Load the dataset
trainset = torchvision.datasets.FashionMNIST(root='./data', train=True,
                                        download=True, transform=transform)

testset = torchvision.datasets.FashionMNIST(root='./data', train=False,
                                       download=True, transform=transform)

# Create data loaders
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2) Code for loading Fashion-MNIST dataset using PyTorch torchvision package. Retrieved from . on 21/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.FashionMNIST.html#torchvision.datasets.FashionMNIST 7. SVHN The SVHN (Street View House Numbers) dataset is an image dataset derived from Google’s Street View imagery, which consists of cropped images of house numbers taken from street-level images. It is available in a full format with all the house numbers and their bounding boxes and a cropped format with just the house numbers alone. The full format is often used for object detection tasks, while the cropped format is commonly used for classification tasks. The SVHN dataset is also included in the torchvision package and it contains 73,257 images for training, 26,032 images for testing and 531,131 additional images for extra training data. To download this torchvision dataset, you can go to or you can load it here: Kaggle import torchvision
import torch

# Load the train and test sets
train_set = torchvision.datasets.SVHN(root='./data', split='train', download=True, transform=torchvision.transforms.ToTensor())
test_set = torchvision.datasets.SVHN(root='./data', split='test', download=True, transform=torchvision.transforms.ToTensor())

# Create data loaders
train_loader = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_set, batch_size=64, shuffle=False) Code for loading SVHN dataset using PyTorch torchvision package. Retrieved from . on 22/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.SVHN.html#torchvision.datasets.SVHN 8. STL-10 The STL-10 dataset is an image recognition dataset that consists of 10 classes, with a total of about 6,000+ images. The STL-10 stands for “Standard Training and Test Set for Image Recognition-10 classes” and the 10 classes in the dataset are: Airplane Bird Car Cat Deer Dog Horse Monkey Ship Truck To access this dataset, you can download it directly from or load the dataset using torchvision: Kaggle import torchvision.datasets as datasets
import torchvision.transforms as transforms

# Define the transformation to apply to the data
transform = transforms.Compose([
    transforms.ToTensor(),   

# Convert PIL image to PyTorch tensor
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))   # Normalize the data
])

# Load the STL-10 dataset
train_dataset = datasets.STL10(root='./data', split='train', download=True, transform=transform)
test_dataset = datasets.STL10(root='./data', split='test', download=True, transform=transform) Code for loading STL-10 dataset using PyTorch torchvision package. Retrieved from . on 22/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.STL10.html#torchvision.datasets.STL10 9. CelebA This torchvision dataset is a popular large-scale face attributes dataset which comprises over 200,000 celebrity images. It was first released by researchers at the Chinese University of Hong Kong in 2015. An image in the CelebA consists of 40 facial attributes such as age, hair colour, facial expression and gender. Also, these images were retrieved from the internet and cover a wide range of facial appearances, including different races, ages and genders. Bounding box annotations for the location of the face in each image, as well as 5 landmark points for the eyes, nose and mouth. You can download this dataset on or load it using this code: Kaggle import torchvision.datasets as datasets
import torchvision.transforms as transforms

transform = transforms.Compose([
    transforms.CenterCrop(178),
    transforms.Resize(128),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
celeba_dataset = datasets.CelebA(root='./data', split='train', transform=transform, download=True) Code for loading CelebA dataset using PyTorch torchvision package. Retrieved from . on 22/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.CelebA.html#torchvision.datasets.CelebA 10. PASCAL VOC The VOC dataset (Visual Object Classes) was first introduced in 2005 as part of the PASCAL VOC Challenge, which aimed at advancing the state of the art in visual recognition. It consists of images of 20 different object categories, including animals, vehicles, and common household objects. Each of these images is annotated with the locations and classifications of objects within the image. The annotations include both bounding boxes and pixel-level segmentation masks. The dataset is split into two main sets: the training and validation sets. The training set contains approximately 5,000 images with annotations, while the validation set contains around 5,000 images without annotations. In addition, the dataset also includes a test set with approximately 10,000 images, but the annotations for this set are not publicly available. To access the recent dataset, you can download from the , or load it in torchvision: website Kaggle import torch
import torchvision
from torchvision import transforms

# Define transformations to apply to the images
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Load the train and validation datasets
train_dataset = torchvision.datasets.VOCDetection(root='./data', year='2007', image_set='train', transform=transform)
val_dataset = torchvision.datasets.VOCDetection(root='./data', year='2007', image_set='val', transform=transform)
# Create data loaders
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=32, shuffle=False) Code for loading PASCAL VOC dataset using PyTorch torchvision package. Retrieved from . on 22/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.VOCDetection.html#torchvision.datasets.VOCDetection 11. Places365 The Places365 dataset is a large-scale scene recognition dataset with more than 1.8 million images covering 365 scene categories. The Places365 Standard dataset consists of around 1.8 million images, while the Places365-Challenge dataset contains 50,000 additional validation images that are more challenging for recognition models. To access this dataset, you can use or load in torchvision here: Kaggle import torch
import torchvision
from torchvision import transforms

# Define transformations to apply to the images
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Load the train and validation datasets
train_dataset = torchvision.datasets.Places365(root='./data', split='train-standard', transform=transform)
val_dataset = torchvision.datasets.Places365(root='./data', split='val', transform=transform)
# Create data loaders
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=32, shuffle=False) Code for loading Places365 dataset using PyTorch torchvision package. Retrieved from . on 22/3/2023 https://pytorch.org/vision/stable/generated/torchvision.datasets.Places365.html#torchvision.datasets.Places365 Common Use Cases for Torchvision Datasets - This dataset is commonly used for image classification tasks, particularly for handwritten digit recognition. MNIST - These datasets are commonly used for image classification tasks, particularly for object recognition. and CIFAR-10 CIFAR-100 - This torchvision dataset contains millions of images of various objects and animals and is commonly used for object recognition tasks. ImageNet - This dataset is commonly used for object detection, segmentation, and captioning tasks. MS COCO - Similar to MNIST, this dataset is commonly used for image classification tasks, particularly for fashion item recognition. Fashion-MNIST - This dataset is commonly used for digit recognition tasks. It is particularly useful for developing and testing algorithms for recognizing numbers in real-world settings. SVHN - This torchvision dataset is commonly used for image classification tasks, particularly for object recognition. STL-10 - This dataset is commonly used for facial recognition and attribute classification tasks. It consists of images of various facial expressions, poses and backgrounds. CelebA - This dataset is commonly used for object detection and segmentation tasks. Pascal VOC - This dataset is commonly used for scene recognition tasks. It consists of images of various indoor and outdoor scenes such as bedrooms, beaches, and offices. Places365 Final Thoughts Torchvision datasets are often used for training and evaluating machine learning models such as convolutional neural networks (CNNs), which are commonly used in computer vision applications. They are also available for anyone to download and use freely. The lead image of this article was generated via model using the prompt 'thousands of images organized together in small frames'. HackerNoon's AI Stable Diffusion More Dataset Listicles: Excel Datasets Keras Datasets R Datasets PyTorch Datasets Hugging Datasets

11 Torchvision Datasets for Computer Vision You Need to Know

কম্পিউটার ভিশনের জন্য 11 টর্চভিশন ডেটাসেট আপনার জানা দরকার

11 Torchvision-Datensätze für Computer Vision, die Sie kennen müssen

11 conjuntos de datos de Torchvision para visión por computadora que necesita saber

11 ensembles de données Torchvision pour la vision par ordinateur que vous devez connaître

कंप्यूटर विजन के लिए 11 टॉर्चविजन डेटासेट आपको जानना जरूरी है

知っておくべきコンピューター ビジョン用の 11 の Torchvision データセット

당신이 알아야 할 11가지 컴퓨터 비전용 Torchvision 데이터 세트

11 conjuntos de dados Torchvision para visão computacional que você precisa conhecer

Bilgisayarlı Görme için Bilmeniz Gereken 11 Torchvision Veri Kümesi

11 bộ dữ liệu Torchvision cho thị giác máy tính bạn cần biết

你需要知道的 11 个用于计算机视觉的 Torchvision 数据集

Singapore, July 8, 2022-Torches Finance, a lending protocol on KCC, announced that it has received seed funding from KuCoin Ventures. KuCoin Ventures and KCC will jointly incubate and empower Torches Finance to build the infrastructure foundation for KCC. KuCoin Ventures is a leading investment arm of KuCoin, the world's leading crypto trading platform, aiming to discover and incubate high-quality projects in various fields such as DeFi, Web 3.0, and infrastructure. Torches Finance is the underlying lending protocol on KCC, which means users can supply or borrow assets on Torches. It helps users obtain passive income while maximizing asset utilization. Torches now has supported supply and borrow services of BTC, ETH, KCS, USDT, and USDC, and will officially start Torches Point Mining in Q3. Users can earn Torches Points by supplying or borrowing assets. Torches Points can be converted into Torches tokens ($TOR) according to certain coefficients after the token launch. It is reported that Torches will leverage the seed funding in product development, technology innovation, security, marketing, and other aspects to increase protocol adoption and market share. For example, in terms of products, Torches will further improve accessibility, optimize the user’s experience, upgrade product functions such as leveraged mining and NFTs, and continuously expand the richness of Torches services. In terms of technology, Torches will continue to improve the transparency of the protocol and work with KCC to increase protocol adoption. Joe, CEO of Torches said, "Receiving seed funding from KuCoin Ventures and the incubation from  KCC is a great integration of Torches with KCS ecosystem. Torches will maximize KCC's marketing, technology, and liquidity resources, and invest the funding into key areas such as product development and technology innovation. With the improvement of products and the launch of point mining, we hope to build Torches as the backbone of KCC’s DeFi ecosystem, and strive to become one of the most important infrastructures in the KCC ecosystem.” The Torches "Supply & Borrow to Win" campaign is still on, and users who provide liquidity (supply or borrow) to a certain amount will have the opportunity to win the whitelist of Torches Genesis NFTs - Torchlight. Torchlight will grant the owner multiple benefits, including but not limited to boosting buffs at Torches Point Mining, IDO allocation…etc. About Torches Finance Torches Finance is a decentralized lending protocol based on KCC, supporting mainstream crypto assets such as BTC, ETH, etc. Users can earn TOR tokens by depositing and lending assets on Torches Finance. Visit Torches Finance: https://www.torches.finance/en About KuCoin Ventures KuCoin Ventures is a leading investment arm of KuCoin Exchange that aims to invest in the most disruptive cryptocurrency and blockchain projects in Web3.0 era. With the commitment of empowering Crypto/Web 3.0 Builders with Deep Insights and Global Resources, it is also a community-friendly and research-driven full-stage corporate venture that focuses on DeFi, Game-Fi, Web 3.0, infrastructure, working closely with its portfolio projects throughout the journey of entrepreneurship.

KuCoin Ventures invests in Torches Finance, a Lending Protocol Based on KCC

Singapore, June 27, 2022 - Torches Finance, a decentralized lending protocol based on KuCoin Community Chian (KCC), announced the mainnet launch on KCC. Torches Finance has enabled supplying and borrowing services for BTC, ETH, KCS, USDT, and USDC. As a lending protocol, Torches Finance wants to achieve a win-win with KCC and leveraged assets on KCC. With the advantages of the low gas fee and gradually thriving KCC ecosystem, Torches Finance can further reduce the impermanent loss and increase the utilization of user assets. According to the official news, Torches Finance has made innovations in lendable assets, lending incentives, liquidation rules, risk control, and other aspects to differentiate Torches from other lending products in the market. Also, Torches Finance was negotiating with other projects of the KCC ecosystem, which will provide investors with more opportunities in the future. Torches have a unique economic incentive model where users can receive TOR tokens whenever they supply and borrow assets. It also revealed Torches Points Mining activities to motivate early users to contribute to Torches Finance. Users can gain a certain amount of Torches Points (TOP) by supplying, borrowing, and liquidating. As the contribution proof, TOP will be able to redeem to TOR after the official release of the TOR token at a specific ratio. The details will be announced at the token launch. Eric, the CMO of Torches Finance, said, “Torches' short-term goal is to become a DeFi project on KCC with massive hype. Torches Finance will meet the needs of asset management on the KCC. And it will work with KCC to expand the market space for DeFi to another level. ” With Torches Finance’s mainnet launch, it will hold a series of activities, including the Genesis NFT Collection, Torchlight, and Free Mint. Torches Finance will give out whitelists to communities. Users can follow Torches’ official Twitter and Discord to discover more. About Torches Finance Torches Finance is a decentralized lending protocol based on KCC, supporting mainstream crypto assets such as BTC, ETH, etc. Users can earn TOR tokens by both depositing and lending assets on Torches Finance. Visit Torches Finance: https://www.torches.finance/en

Torches Finance: A Decentralized Lending Protocol Launched on KCC

Torches Finance: un protocolo de préstamo descentralizado lanzado en KCC

Hi there! If you’re new to this series, I laid down some ground rules , and explain the idea for my first project . I document, a little every day, what I’m working on. in day zero on day one I’ve realized, especially with today’s update where I’m discussing torching some money on social advertising and Nginx boilerplate, that I sometimes throw a weird hodgepodge of stuff at you. I almost edited this post down to one topic, but I think this mix of random thoughts is the most honest representation of my day-to-day during this challenge. So let’s get into it. Image credit: CC0 https://pixabay.com/en/users/Intellectual-4717896/ Burning some cash, like the Joker but on a scale my wife won’t kick me out of bed for First, let me lead up to I’m burning a bit of cash on ad experiments. Today has been pretty lively for a Sunday: why Chat on is working. I’ve had amazing live chats today with lovely people. Now this may not sound like much. But this is a B2B SaaS landing page that’s been live for less than 24 hours and has no advertising or backlinks other than those related to this little series. bystander.io 3 I have new signups for the early access form on the landing page today. These exclude the few signups I’ve been able to battle for via cold outreach. 7 Traffic to the landing page and stats to this blog continue to grow. While the stats for this series may not mean much in terms of validating Bystander, it sure does mean a lot to me and keeps me incredibly motivated (seriously, thank you to everyone reading and messaging me) For those curious how this little series documenting my journey has grown on Medium from day zero: Moving the series into Hacker Noon has been great Now, back on topic the topic of throwing away some cash. This budding interest in Bystander gives me the confidence to move a bit farther. A big goal over the next two days is setting up an option to preorder. Preordering will save businesses 50% over the period they purchase. I’ve never tried the preordering method of validation, and I’d like to see if we can make it work here. My assumptions now, are, a business who makes the leap for a preorder is someone I’m in active communication with. I don’t expect a landing page alone to be able to convert someone into prepaying for a B2B SaaS solution. So why ads? They’re a cheap way for me to test my copy and assumptions at a larger scale than I currently can organically. For trying to get folks to preorder I want to have the best copy and value proposition I can possibly have with my current resources and traffic. Now, first time ad accounts usually offer bonuses that are quite generous for the scale were working with. I want to keep my spend below 200 dollars here. But , for example, will give you $50 credit to start with, taking advantage of this and similar bonuses stretches that small testing budget quite a bit farther. LinkedIn Each campaign on both and LinkedIn . Facebook are both running 3 different ads With each add set, I’m testing two different things: My headlines & subheadlines My audience’s by job title: user experience professionals VS. support leaders VS. project managers Here’s an example LinkedIn ad for the User Experience camp: I don’t expect conversions in this small run at advertising. The scale of my spend is tiny, it’s my first go ad advertising Bystander, and my primary goal is to see if I can pin down a measurable difference on my audience and headlines to help focus my early efforts. Plus, I can’t learn about advertising by just reading, I need to just start. And at the very least, I’m spending a bit of money for your entertainment 😂 Going into some details and small changes in Bystander’s stack Some folks have kindly expressed an interest in the stack I’m working with for . I touched on this briefly , but I thought it may interest some folks on a more detailed look at the stack and the tools I use on a daily basis. Bystander.io on day 2 I come from a front-end development background originally. But most recently spent time as a Senior Engineer in Customer Support. Support engineering is a strange beast, you get really wide knowledge on an unbelievable number of technologies and customer stacks. But it’s like exploring an ankle deep ocean of tech — you find things fast, in any setting, and fix them. You don’t generally have a need to further explore code (or the time!) Going solo on a project, I mostly stick to what I know, and may toss _just one_new piece into the stack that I’m interested in trying or learning. You do have to stay interested and excited after all! The hours are long when you try something new. I’ve made the mistake of trying to build a project with all the new cool things a few times before, and progress ends up so slow I lose motivation and abandon ship. So! The stack for this … Bystander Backend: latest stable with . Node Express with MongoDB Mongoose for authentication and access control, authentication is something that’s been solved a million times. But working on a solo project, authentication-as-a-service saves me a ton of time. And it let’s me be confident in an otherwise scary space. Their content team is top notch too, if any of that interests you, check out . Auth0 their blog Express+Mongoose+Auth0 let me make APIs with protected endpoints incredibly fast. On the frontend: , which requires . I find Typescript an absolute joy to work with. It’s a superset of JS that let’s you develop with JS as a strongly typed language. So, working with Angular in and IDE like gives you typeahead on all the things and lets you catch compile errors. Angular 5 Typescript Webstorm For a design and component framework, my original plan was to go with , but the screenshots I currently made available actually leverage a , with components I’ve already sliced down into Angular components/services for a previous project. I’m still tempted by Nebular as it feels more modern, but really, I need to double down and focus on function rather than form. Nebular Bootstrap official theme The new (to me!) tech I’m playing with for this project is , although in the form of (Angular’s flavor of Redux) NgRx/Redux put your application’s state into a single object. So you get some reallllyyy cool benefits when it comes to debugging. is worth a watch. Plus, even without having an interest in Redux or NgRx, this is the best format I’ve ever seen for two developers presenting a talk. Brandon and Mike did an awesome job on this. Redux NgRx This talk by Ngrx maintainers Hosting Related: droplets: one for marketing+blog, one for the app’s node instance. Scalability comes when I actually have customers. Digital ocean Nginx . Cause it’s easy to use and their free services have a real impact. Cloudflare (Thanks to for recommending them over MLab) MongoDB’s own database-as-a-service a kind reader Applications I use daily: , still hard to beat once you get it set up with the plugins needed for your project. Although, because of Angular 5 I’m spending less time in there and more time in… Sublime Text Webstorm which I touched on earlier. makes it so your terminal is awesome. If you don’t already use iTerm2, I’d Google around a bit to see how other folks set it up. That will save you some time on getting the visual side going. I LOVE tabs and color coding so I can see where I have different services running locally: iTerm2 Red, Blue two different running stances of an app on different ports so I can test two branches live. Which are my active tabs where I actually need to work in the console. to take screenshots and annotate them for documenting this journey. It’s free and amazing. Skitch when I need to quickly record a Gif of part of my screen. Also free and super simple to use. LICEcap My own Slack (free) workspace to pipe notifications and data to me. Checking different emails, chat messages from Bystander.io, keeping up with Tweets, and follows on medium… Pretty much anything I ever need to tab over to, to check any sort of notification, I just yank that into a slack channel to avoid distractions and save a ton of time. Plus I don’t have room for new tabs when I’m working: I know I need to close some tabs when the icons go away, or my laptop’s fan is giving me a headache Spotify to listen to weird stuff. And I work until 11pm/midnight, so I keep Night Shift turned on, it definitely helps me. For any screen that isn’t iOS based, I use for the same purpose. Flux Reduce some blue light to help me get to sleep after a late night working And that’s about everything! What tools are essential in your workflow? Is there anything you use that might save me some time? ’Cause I could certainly do with a bit more time! Tomorrow, Day 10! Begin hooking up Stripe to add a pre-order option. Lots and lots of follow up. I need to touch base with new subscribers, folks I’m cold emailing, and the few people from live chat. More cold outreach (it never stops) Rollover from today’s list, to get Bystander’s Ghost blog into a template I like 5 million more things that Ican’tfitintotheday Thanks for reading, this turned out quite long! And of course, feel free to email me with any questions or feedback! where this was originally published. And I’ll see you all tomorrow! My email is on my homepage : Forward to day 9: Let’s talk pricing Back to day 8 Please rip my landing page to shreds

[Day 9] Zero to MVP in 30 Days — Torching some cash

ATLAS congratulates Torchlight Energy on a successful Annual Shareholder Meeting I recently had the pleasure of attending Torchlight Energy’s . With Torchlight Energy being a large portfolio holding and with Torchlight being a multi-year investment for ATLAS — a considerable time, opportunity, capital, and — the was an important checkpoint. Very rarely are executive management, the Board of Directors, and key investors in such an intimate and real-time response environment. Torchlight’s offered a unique opportunity to update our investment thesis; as well as to consider optimization strategies, potential value-add scenarios, and exit timelines. Annual Shareholder Meeting reputation commitment Annual Shareholder Meeting Annual Shareholder Meeting We’re happy to report that we were more than satisfied with the and that we continue to believe in the disruptive potential of Torchlight Energy’s technology-first approach. Never before have we been more confident that Torchlight has the as well as the underlying to be able to execute its investment thesis. While Torchlight very much so still has venture level execution risk, we believe that it has the operating strategy to execute — even given the historic degree of difficulty it faces in attacking its assets. While every potentially industry disrupting endeavor has unique risks and unique uncertainties, post- we’re confident that Torchlight’s wildcatting attempts in both its Hazel Project and its Orogrande Project will yield productive results. Annual Shareholder Meeting; technology geology Orogrande Project Annual Shareholder Meeting With Torchlight being just months away from a drilling and completion program launch — Torchlight has already launched in the Hazel Project, it’s paramount that Torchlight be sure of its operational program. In hosting numerous multi-hour meetings with both the executive management team and the Board of Directors, I’m confident that the execution strategy is optimized. multi-play Torchlight management, impressively so, presented a point-to-point, logical, and decisive technical-strategy for attacking both the as well as the [the first horizontal Well attempts in the Orogrande Project and Hazel Project, respectively]. In meeting with Torchlight’s operations and technical team, I was impressed with the degree of granularity that each technical risk had been assessed and that each technical risk had been resolved . Torchlight, in staying to its core competency, has found ways to deploy modern Frac Tech to tailor a scheme-based solution for each anticipated complexity. University Founders A25 #1 Well Flying B Ranch #3H Well through technology-first, engineered solutions It’s in these engineered solutions that I derive my conviction in [1] the operational approach being deployed and [2] the probability of ultimate success. Seemingly, Torchlight has the architecture, design, landing capacity, and technology to be able to execute. Torchlight’s ultimate success or failure will be entirely dependent on its ability to maximize the utility of current Frac Tech; effectively, the Torchlight investment thesis and company “DNA”. “ aving done very detailed mapping and avoiding areas with intense faulting, volcanism, high clay contents, and which were overly mature [thermally], Torchlight Energy zoomed in on a 168,000 net acres within the oil window in Texas” — Wunderlich Securities [New Permutations to the Permian Theme: The Alpine High and Orogrande Basin] H But Torchlight isn’t simply an operational story, it’s also very much so a story. Torchlight’s geology — its underlying acreage, collected through a unique combination of relationships as well as a willingness to attack difficult complexities, is absolutely sound; a belief that was reinforced in multiple presentations at the Torchlight has collected libraries of data — data which it has been able to refine, cleanse, model, and analyze in ways that have never been applied to either the Hazel Project or the Orogrande Project. Post- I’m convinced that Torchlight has an unprecedented geological insight into its two resource plays; one that it believes has helped it identify productive acreage within undeveloped Projects. Quite possibly Torchlight’s most valuable asset might be its geological expertise — the collective geological intellectual capacity of its combined management team, Board of Directors, and consulting geologist. geology Annual Shareholder Meeting. Annual Shareholder Meeting Torchlight’s unique ability to read and assess complex geology — including subsets of geology within larger Projects or plays— has allowed it to aggregate a uniquely large acreage position for an E&P its size. We challenged Torchlight on [i] , [ii] , [iii] , [iv] , [v] , [vi] , [vii] , [viii] , [ix] , and [x] . In every instance we were met with more than adequate rebuttal. Geology, in our opinion, was the single largest risk to the investment thesis at Torchlight — given that geology can’t be altered, iterated, or changed [as nearly all other parts of the operational process can]. Having performed multiple interviews and sat through multiple presentations, we believe Torchlight’s geology to be greatly derisked to our prior evaluations; and it should be noted that we were more than satisfied to be invested at “prior evaluations”. thickness of reservoir porosity permeability brittleness clay content TOC thermal maturity matrix permeability pore throat sizing overall complexity translating to operational risk orchlight Energy is looking to challenge the unique geological complexities of the Orogrande Basin — one of its two major Projects; to challenge the extraordinary Well-design difficulty presented by the Orogrande Basin, and to challenge the nearly limitless down hole variables it is sure to encounter upon launching its drilling and exploration program T Having completed initial data-collection and analysis efforts, Torchlight Energy is proceeding with first horizontal-Well deployment; including the first horizontal effort in the history of the Orogrande Basin. With these horizontal efforts, Torchlight moves beyond vertical Well-testing and vertical Well data collection and into thesis prove-out. With these horizontal efforts, Torchlight will be able to deploy the full war chest of its technology, its geological, geophysical, and Well-design learnings, as well as its [vertical-Well tested] operating learnings. Torchlight Energy is ready to prove concept. We’ve never been more confident in Torchlight or in the strategy for attacking both the Hazel Project and the Orogrande Project; and we’re extremely excited that Torchlight has moved forward with horizontal Well deployment. Through each stage of maturation and through each stage of optimization we’ve been proud to work with Torchlight Energy management and we’ve been proud to support the E&P’s development efforts. We continue to believe in the team at Torchlight Energy and we continue to believe in the disruptive potential of its technology-first approach to revolutionizing the Energy Complex. DISCLOSURE: [1] Torchlight Energy Resources, Inc. was not involved in any aspect of the above note preparation or in any aspect of the preparation of any data or information presented [2] I have not received compensation from Torchlight Energy Resources, Inc. or any other party for the writing of this note; and/or any other note written prior [regarding Torchlight Energy Resources, Inc.] [3] I may or may not have received compensation for providing research and/or consulting regarding Torchlight Energy Resources, Inc. from third-parties at some point prior to writing this note and/or I may receive compensation for providing research and/or consulting regarding Torchlight Energy Resources, Inc. at a later date. This includes providing subscription-based research and/or consulting services to both institutional and retail-based capital market participants. [4] Dallas Salazar: I own, or my family or company owns, or I have influence over some amount of the outstanding shares of the following companies mentioned in this note: Torchlight Energy Resources, Inc. [5] The article does not constitute investment advice. Each reader is encouraged to consult with his or her individual financial professional and any action a reader takes as a result of information presented here is his or her own responsibility. [6] Torchlight Energy Resources, Inc. holds the license to certain images included in the note above; to which the company has allowed me a one-time publishing right. I did not pay for this right nor will I be providing a fee of any sort for this right into the future.

Torchlight Energy: A Billion Barrel Vision Nears Proof of Concept

ATLAS congratulates the Torchlight Energy team on permitting its first horizontal Well in the Orogrande Project Nearly two years ago, into the heart of what would eventually prove out to be the longest and most catastrophic commodity crisis in the history of Energy, we at ATLAS made a data-based bet on a pre-production oil and gas E&P [i.e., “exploration and production company”]. At the time, prior to an aggressive and historic asset restructuring effort, the E&P we had bet on had substantially one asset we were interested in: its [based in the — a completely unproven West Texas resource play]. Orogrande Project Orogrande Basin The Orogrande Basin, effectively leased in its entirety by Torchlight Energy, closely resembled its tethered-E&P — it had no established production and it presented its own set of unique stress points. This is why Torchlight had the opportunity to lease the Orogrande Basin acreage [1] in its entirety and [2] at economics that made sense to a pre-production, stressed-E&P. Nobody else wanted the Orogrande Basin. The Orogrande Basin had a long, established, and something of infamous track record of turning back E&Ps — of leaving them wanting. Of course, it only turned these E&Ps back after the E&Ps had spent millions in trying to develop what could potentially be a billion barrel resource play — the reason many an E&P had tried their hand at wildcatting the play in the first place [the potential payload locked within the rocks]. The Orogrande Basin had become a pariah; one that not only beat its challengers — but embarrassed them. Littered with geological complexities thanks to intense geological-deformations caused by millions of years of uplifts, faulting, igneous intrusions and volcanism — looking for prospective pockets is infinitely harder in the Orogrande Basin than, say, the Midland Basin; which is an eastern located, much friendlier, much more famous peer of the Orogrande Basin But Torchlight wasn’t coming to the Orogrande Basin — which houses its Orogrande Project — looking to replicate the efforts of its predecessors. Torchlight was optimizing an Orogrande Project-tailored Well-strategy that would see it deploy twenty years of macro-technology evolution; of translated evolution that it believed gave it unprecedented geological and Well-design insight as well as unprecedented technical prowess. Torchlight was looking to challenge the unique geological complexities of the Orogrande Basin, to challenge the extraordinary Well-design difficulty presented by the Orogrande Basin, and to challenge the nearly limitless down hole variables it was sure to encounter upon launching its drilling and exploration program. Frac Tech We believed that Torchlight, led into the Orogrande Basin by famed prospector-geologist Rich Masterson, could execute — if it were able to collect enough data prior to launching a wildcatting program — and we believed that Torchlight had the geological, geophysical, and reservoir engineering team in place to do what had never been done before. In short, we believed that Torchlight could by successfully proving out its Orogrande Project efforts. Torchlight, with its last vertical Well-test, has gathered the necessary data to move the Orogrande Project thesis forward. disrupt the global Energy Complex Nearly two years later, Torchlight Energy has permitted the first horizontal Well for its Orogrande Project; the first horizontal effort in the history of the Orogrande Basin. With this horizontal effort, Torchlight moves beyond vertical Well-testing and vertical Well data collection and into thesis prove-out. With this horizontal effort, Torchlight will be able to deploy the full war chest of its technology, its geological, geophysical, and Well-design learnings, as well as its vertical-Well tested operating-learnings. Torchlight, more so than any of its predecessors and more so than any E&P in the history of the Orogrande Basin, is ready to prove concept; it’s ready to unlock its Orogrande Project. We met with the Torchlight team at Torchlight HQ just two weeks ago to review overall strategy, Well-location, and the geological complexities of the Orogrande Project. We’ve never been more confident in Torchlight or in the strategy for attacking the Orogrande Project; and we’re extremely excited that Torchlight has moved forward with a horizontal permit. “Having done very detailed mapping and avoiding areas with intense faulting, volcanism, high clay contents, and which were overly mature [thermally], Torchlight Energy zoomed in on a 168,000 net acres within the oil window in Texas” — Wunderlich Securities [New Permutations to the Permian Theme: The Alpine High and Orogrande Basin] Through each stage of maturation and through each stage of optimization we’ve been proud to work with Torchlight management and we’ve been proud to support its development efforts. We continue to believe in the team at Torchlight and we continue to believe in the disruptive potential of its technology-first approach to revolutionizing the Energy Complex. Congratulations on first horizontal Well permit, Torchlight Energy. DISCLOSURE: [1] Torchlight Energy Resources, Inc. was not involved in any aspect of the above note preparation or in any aspect of the preparation of any data or information presented [2] I have not received compensation from Torchlight Energy Resources, Inc. or any other party for the writing of this note; and/or any other note written prior [regarding Torchlight Energy Resources, Inc.] [3] I may or may not have received compensation for providing research and/or consulting regarding Torchlight Energy Resources, Inc. from third-parties at some point prior to writing this note and/or I may receive compensation for providing research and/or consulting regarding Torchlight Energy Resources, Inc. at a later date. This includes providing subscription-based research and/or consulting services to both institutional and retail-based capital market participants. [4] Dallas Salazar: I own, or my family or company owns, or I have influence over some amount of the outstanding shares of the following companies mentioned in this note: Torchlight Energy Resources, Inc. [5] The article does not constitute investment advice. Each reader is encouraged to consult with his or her individual financial professional and any action a reader takes as a result of information presented here is his or her own responsibility. [6] Torchlight Energy Resources, Inc. holds the license to the title image included in the note title; to which the company has allowed me a one-time publishing right. I did not pay for this right nor will I be providing a fee of any sort for this right into the future.

Texas Wildcatter Torchlight Energy Takes Frac Tech to the Orogrande Project

The Crowded AI Spaces Still Have Room for One More Genius

জনাকীর্ণ এআই স্পেসগুলিতে এখনও আরও একটি প্রতিভা থাকার জায়গা রয়েছে

In den überfüllten KI-Räumen ist noch Platz für ein weiteres Genie

Los abarrotados espacios de IA todavía tienen espacio para un genio más

Les espaces IA bondés ont encore de la place pour un génie supplémentaire

भीड़-भाड़ वाले एआई स्पेस में अभी भी एक और प्रतिभा के लिए जगह है

混雑したAI空間にはまだもう一人の天才の余地がある

혼잡한 AI 공간에는 아직 천재가 한 명 더 들어갈 여지가 있습니다

Os lotados espaços de IA ainda têm espaço para mais um gênio

В переполненных пространствах ИИ все еще есть место для еще одного гения

Kalabalık Yapay Zeka Alanlarında Hala Bir Dahiye Daha Yer Var

Không gian AI đông đúc vẫn còn chỗ cho một thiên tài nữa

拥挤的人工智能领域仍有空间容纳更多天才

Day 1 and Day 2 Operations for Platform Engineers

Highlight: Check out my GPU VRAM Calculator I’ve always been curious about the GPU VRAM required for training and fine-tuning transformer-based language models. What factors influence VRAM consumption? How does it vary with different model settings? I dug into the topic and conducted my measurements. Other great resources include Stas Bekman’s section from his ML Engineering book, which is the core inspiration for Hugging Face’s model memory anatomy article. Also, check out Eleuther’s blog, which also covers compute costs. Quick note: This post doesn’t investigate the memory usage of quantized models and PEFT fine-tuning techniques like LoRA or QLoRA. Prerequisites for experiments When we talk about RAM, we often use GB (10^9 bytes) and GiB (2^30 bytes) interchangeably. But in reality, we’re dealing with GiB. Take the Nvidia 3090’s “24 GB VRAM” – it’s actually 24 GiB, or about 25.76 GB. To keep things clear, I’ll stick with MiB and GiB. To measure VRAM usage accurately, we need to delete the variable, run garbage collection, clear the CUDA cache, and then measure the VRAM difference. Here’s an example: x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB The ipyexperiments Python package automates this after each cell execution, which is pretty convenient. Before assessing memory usage, it’s important to perform warm-up steps, essentially running the same code twice, to load CUDA kernels that weren’t loaded during the initial setup. Also, we should disable the cache in the decoder, which is used during inference to prevent re-computation of hidden states [1]. Mixed precision training Understanding mixed precision training is key, as it’s commonly used in pretraining and finetuning. Normally, model parameters are stored in float32 format, taking up 4 bytes per parameter. Mixed precision training uses float16, halving the calculation time and reducing the size of activations. But why “mixed”? The training isn’t entirely in half precision. Lower precision can lead to imprecise weight updates or even gradients turning to zero. So, in mixed precision training, the master copy of the weights is kept and updated in fp32, and before each forward pass, these weights are copied into fp16 format. For a deeper dive into mixed precision, check out this fast.ai documentation, which includes a detailed illustration, and Aleksey Bilogur’s blog, which offers practical PyTorch code examples. Handling multi-GPU scenarios What if a model doesn’t fit on a single GPU? There are two scenarios: Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto". Learn more in the accelerate docs.
Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation. Breaking down the components Memory consumption consists of the following components: Train

Inference



CUDA Kernels

✅

✅



Parameters

✅

✅



Activations

✅

✅



Gradients

✅

❌



Optimizer States

✅

❌



Outputs

✅

✅ An interesting aspect of PyTorch is its approach to memory allocation. Essentially, PyTorch rarely releases memory once it’s been allocated. For instance, during the forward pass, activations are calculated and stored in memory. Even after these activations are no longer needed following the backward pass, the memory they occupy isn’t released. This strategy is adopted to avoid the overhead associated with frequent memory allocation calls [3]. CUDA Kernels Upon first using the GPU, CUDA kernels will allocate between 300 MiB to 2000 MiB. This can vary based on GPU, driver, and PyTorch versions. It could be measured by initializing any small tensor and moving it to GPU: x = torch.ones(1).cuda() Parameters When measuring the amount of memory that will be used by parameters, it is important to understand the difference between parameters and buffers. Parameters are the actual weights that are being trained and updated by the optimizer. They could be retrieved by calling model.parameters(). Apart from parameters, fixed tensors exist, which are needed in some computations but do not need to be updated. These are called buffers and may be retrieved by calling model.buffers(). One example of buffers is precomputed positional encodings [4]. So, in this section, under ‘parameters,’ I assume ‘parameters’ + ‘buffers.’ During inference, the memory needed for parameters is straightforward — it’s just the number of parameters multiplied by the number of bytes per parameter. You are specifying the number of bytes per parameter when loading a model like .from_pretrained(..., torch_dtype=torch.float16). For instance, a 7B-parameter model like Mistral, when loaded in half-precision (float16), would take 7.51 × 10**9 × 2 bytes, equating to 14324 MiB. When training as usual, in full precision, 4 bytes per parameter are occupied, mixed precision training is more common though, in this case, we have to maintain both half precision (for forward pass, 2 bytes per param) and full precision model weights (for applying updates to them, 4 bytes per param), so in total it takes 6 bytes per param. Activations ‘Activations’ refer to the intermediate outputs essential for backpropagation. They are usually the memory bottleneck in transformer training, especially since their size scales quadratically with sequence length (we have to store the output of a softmax(Q×K.T) which has Batch Size × Number of Attention Heads × Sequence Length ** 2 shape). There are good estimations of activations size per layer in the “Reducing Activation Recomputation in Large Transformer Models” paper in section 4.1 although for each model activations will differ. For example, in the mentioned paper they also count dropout masks whereas newer architectures like Llama don’t use dropout at all. During training, we store all layer activations for backprop, but in inference, we only keep the current (single) layer’s activations. We can reduce activation size on training in the cost of training speed (slowdown around 20%) by discarding the activations during the forward pass and recalculating them when needed during the backward pass; this is called gradient checkpointing. Gradients Gradients are always stored at full precision, taking 4 bytes per parameter. Optimizer states Optimizers like Adam and SGD have their own memory needs. SGD with momentum and Adam both store a moving average of gradients for each parameter in full precision. Additionally, Adam keeps a moving average of squared gradients. First Moments

Second Moments

Bytes per Param



SGD

❌

❌

0



SGD w momentum

✅

❌

4



ADAM

✅

✅

8 Outputs Finally, the output tensors (Batch Size × Sequence Length × Vocabulary Size) are almost always in float32. This remains true even if the model was loaded at a lower precision because the model itself casts outputs to float32 most of the time [5] [6]. While training, we also need to store probabilities F.softmax(logits, dim=-1) which are the same size as the output tensor. Problems In my experiments with measuring VRAM usage in the notebook, I am facing some persistent mismatch between what my experiments show and the calculated figures, particularly regarding the size of activations during the training’s forward pass. So, there is still something to figure out! Acknowledgments Thanks to Stas Bekman for helping me shape my understanding and Quentin Anthony’s Python gist for VRAM calculation. Highlight : Check out my GPU VRAM Calculator Highlight GPU VRAM Calculator I’ve always been curious about the GPU VRAM required for training and fine-tuning transformer-based language models. What factors influence VRAM consumption? How does it vary with different model settings? I dug into the topic and conducted my measurements. Other great resources include Stas Bekman’s section from his ML Engineering book , which is the core inspiration for Hugging Face’s model memory anatomy article . Also, check out Eleuther’s blog , which also covers compute costs. Stas Bekman’s section from his ML Engineering book Hugging Face’s model memory anatomy article Eleuther’s blog Quick note: This post doesn’t investigate the memory usage of quantized models and PEFT fine-tuning techniques like LoRA or QLoRA. Prerequisites for experiments When we talk about RAM, we often use GB (10^9 bytes) and GiB (2^30 bytes) interchangeably. But in reality, we’re dealing with GiB. Take the Nvidia 3090’s “24 GB VRAM” – it’s actually 24 GiB, or about 25.76 GB. To keep things clear, I’ll stick with MiB and GiB. To measure VRAM usage accurately, we need to delete the variable, run garbage collection, clear the CUDA cache, and then measure the VRAM difference. Here’s an example: x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB The ipyexperiments Python package automates this after each cell execution, which is pretty convenient. ipyexperiments Before assessing memory usage, it’s important to perform warm-up steps, essentially running the same code twice, to load CUDA kernels that weren’t loaded during the initial setup. Also, we should disable the cache in the decoder, which is used during inference to prevent re-computation of hidden states [1] . cache [1] Mixed precision training Understanding mixed precision training is key, as it’s commonly used in pretraining and finetuning. Normally, model parameters are stored in float32 format, taking up 4 bytes per parameter. Mixed precision training uses float16, halving the calculation time and reducing the size of activations. But why “mixed”? The training isn’t entirely in half precision. Lower precision can lead to imprecise weight updates or even gradients turning to zero. So, in mixed precision training, the master copy of the weights is kept and updated in fp32, and before each forward pass, these weights are copied into fp16 format. For a deeper dive into mixed precision, check out this fast.ai documentation , which includes a detailed illustration, and Aleksey Bilogur’s blog , which offers practical PyTorch code examples. fast.ai documentation Aleksey Bilogur’s blog Handling multi-GPU scenarios What if a model doesn’t fit on a single GPU? There are two scenarios: Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto". Learn more in the accelerate docs. Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation. Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto" . Learn more in the accelerate docs . device_map="auto" accelerate docs Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation . DeepSpeed ZeRO stages FSDP [2] Hugging Face’s documentation Breaking down the components Memory consumption consists of the following components: Train

Inference



CUDA Kernels

✅

✅



Parameters

✅

✅



Activations

✅

✅



Gradients

✅

❌



Optimizer States

✅

❌



Outputs

✅

✅ Train

Inference CUDA Kernels

✅

✅ Parameters

✅

✅ Activations

✅

✅ Gradients

✅

❌ Optimizer States

✅

❌ Outputs

✅

✅ Train Inference Train Train Inference Inference CUDA Kernels ✅ ✅ CUDA Kernels CUDA Kernels ✅ ✅ ✅ ✅ Parameters ✅ ✅ Parameters Parameters ✅ ✅ ✅ ✅ Activations ✅ ✅ Activations Activations ✅ ✅ ✅ ✅ Gradients ✅ ❌ Gradients Gradients ✅ ✅ ❌ ❌ Optimizer States ✅ ❌ Optimizer States Optimizer States ✅ ✅ ❌ ❌ Outputs ✅ ✅ Outputs Outputs ✅ ✅ ✅ ✅ An interesting aspect of PyTorch is its approach to memory allocation. Essentially, PyTorch rarely releases memory once it’s been allocated. For instance, during the forward pass, activations are calculated and stored in memory. Even after these activations are no longer needed following the backward pass, the memory they occupy isn’t released. This strategy is adopted to avoid the overhead associated with frequent memory allocation calls [3] . [3] CUDA Kernels Upon first using the GPU, CUDA kernels will allocate between 300 MiB to 2000 MiB. This can vary based on GPU, driver, and PyTorch versions. It could be measured by initializing any small tensor and moving it to GPU: x = torch.ones(1).cuda() x = torch.ones(1).cuda() Parameters When measuring the amount of memory that will be used by parameters, it is important to understand the difference between parameters and buffers. Parameters are the actual weights that are being trained and updated by the optimizer. They could be retrieved by calling model.parameters() . Apart from parameters, fixed tensors exist, which are needed in some computations but do not need to be updated. These are called buffers and may be retrieved by calling model.buffers() . One example of buffers is precomputed positional encodings [4] . So, in this section, under ‘parameters,’ I assume ‘parameters’ + ‘buffers.’ model.parameters() model.buffers() [4] During inference, the memory needed for parameters is straightforward — it’s just the number of parameters multiplied by the number of bytes per parameter. You are specifying the number of bytes per parameter when loading a model like .from_pretrained(..., torch_dtype=torch.float16) . For instance, a 7B-parameter model like Mistral, when loaded in half-precision (float16), would take 7.51 × 10**9 × 2 bytes, equating to 14324 MiB. .from_pretrained(..., torch_dtype=torch.float16) When training as usual, in full precision, 4 bytes per parameter are occupied, mixed precision training is more common though, in this case, we have to maintain both half precision (for forward pass, 2 bytes per param) and full precision model weights (for applying updates to them, 4 bytes per param), so in total it takes 6 bytes per param. Activations ‘Activations’ refer to the intermediate outputs essential for backpropagation. They are usually the memory bottleneck in transformer training, especially since their size scales quadratically with sequence length (we have to store the output of a softmax(Q×K.T) which has Batch Size × Number of Attention Heads × Sequence Length ** 2 shape). There are good estimations of activations size per layer in the “Reducing Activation Recomputation in Large Transformer Models” paper in section 4.1 although for each model activations will differ. For example, in the mentioned paper they also count dropout masks whereas newer architectures like Llama don’t use dropout at all. softmax(Q×K.T) “Reducing Activation Recomputation in Large Transformer Models” Llama During training, we store all layer activations for backprop, but in inference, we only keep the current (single) layer’s activations. We can reduce activation size on training in the cost of training speed (slowdown around 20%) by discarding the activations during the forward pass and recalculating them when needed during the backward pass; this is called gradient checkpointing . gradient checkpointing Gradients Gradients are always stored at full precision, taking 4 bytes per parameter. Optimizer states Optimizers like Adam and SGD have their own memory needs. SGD with momentum and Adam both store a moving average of gradients for each parameter in full precision. Additionally, Adam keeps a moving average of squared gradients. First Moments

Second Moments

Bytes per Param



SGD

❌

❌

0



SGD w momentum

✅

❌

4



ADAM

✅

✅

8 First Moments

Second Moments

Bytes per Param SGD

❌

❌

0 SGD w momentum

✅

❌

4 ADAM

✅

✅

8 First Moments Second Moments Bytes per Param First Moments First Moments Second Moments Second Moments Bytes per Param Bytes per Param SGD ❌ ❌ 0 SGD SGD ❌ ❌ ❌ ❌ 0 0 SGD w momentum ✅ ❌ 4 SGD w momentum SGD w momentum ✅ ✅ ❌ ❌ 4 4 ADAM ✅ ✅ 8 ADAM ADAM ✅ ✅ ✅ ✅ 8 8 Outputs Finally, the output tensors (Batch Size × Sequence Length × Vocabulary Size) are almost always in float32. This remains true even if the model was loaded at a lower precision because the model itself casts outputs to float32 most of the time [5] [6] . [5] [6] While training, we also need to store probabilities F.softmax(logits, dim=-1) which are the same size as the output tensor. F.softmax(logits, dim=-1) Problems In my experiments with measuring VRAM usage in the notebook , I am facing some persistent mismatch between what my experiments show and the calculated figures, particularly regarding the size of activations during the training’s forward pass. So, there is still something to figure out! in the notebook Acknowledgments Thanks to Stas Bekman for helping me shape my understanding and Quentin Anthony’s Python gist for VRAM calculation . Stas Bekman gist for VRAM calculation

Breaking down GPU VRAM consumption

On-Device AI Models and Core ML Tools: Insights From WWDC 2024

Developing large language models requires substantial investments in time and GPU resources, translating directly into high costs. The larger the model, the more pronounced these challenges become. Recently, Yandex has introduced a new solution: YaFSDP, an open-source tool that promises to revolutionize LLM training by significantly reducing GPU resource consumption and training time. In a pre-training scenario involving a model with 70 billion parameters, using YaFSDP can save the resources of approximately 150 GPUs. This translates to potential monthly savings of roughly $0.5 to $1.5 million, depending on the virtual GPU provider or platform. Yandex has made YaFSDP publicly available on GitHub. The Challenge of Distributed LLM Training Training LLMs across multiple GPUs involves complex operations that lead to inefficiencies and high memory consumption. One of the main issues is the need to send and receive massive amounts of data between GPUs. For instance, in a typical all_reduce operation, twice the amount of gradient data as there are network parameters must be communicated. In the case of a Llama 70B model, this means transferring 280 GB of data per iteration. Furthermore, weights, gradients, and optimizer states are duplicated across GPUs, leading to an enormous memory load. The Llama 70B model and the Adam optimizer require over 1 TB of memory, far exceeding the typical 80 GB memory capacity of most GPUs. This redundancy severely slows down the training process and often makes it impractical to fit even moderately sized models into GPU memory. Introducing YaFSDP Yandex’s YaFSDP offers a highly effective solution to these challenges. By focusing on optimizing memory consumption and eliminating communication bottlenecks, YaFSDP enhances the efficiency of LLM training. It works by sharding layers instead of individual parameters, maintaining efficient communications, and avoiding redundant operations. Additionally, YaFSDP pre-allocates buffers for all required data, ensuring that the Torch allocator does not introduce inefficiencies. YaFSDP operates by utilizing two buffers for intermediate weights and gradients, with odd layers using one buffer and even layers using the other. The weights from different layers are stored in the same memory. If the layers have the same structure, they will always be identical. It is crucial to ensure that when you need layer X, the buffer contains the weights for layer X. All parameters will be stored in the corresponding memory chunk within the buffer. Memory Consumption During training, the primary memory consumers are weights, gradients, optimizer states, buffers, and activations. YaFSDP significantly reduces memory consumption by optimizing how these elements are stored and accessed. Weights, Gradients, and Optimizer States: These depend on the number of processes, and their memory consumption tends to approach zero as the number of processes increases. By sharding these components across GPUs, YaFSDP minimizes duplication and thus reduces memory usage. Buffers consume a constant amount of memory and store intermediate values during computations. Activations depend on the model size and the number of tokens processed per GPU. Activation Checkpointing Activation checkpointing is a technique that stores only necessary activations during the forward pass and recomputes them during the backward pass. This reduces the memory footprint significantly, as only essential data is stored. For example, in training a Llama 2 70B model with a batch size of 8192 tokens, activation storage can be reduced from over 110 GB to just 5 GB. However, this approach introduces additional computational overhead, which YaFSDP allows to avoid by not using the activation checkpointing for some layers which is possible due to memory optimization. Communication Optimization YaFSDP improves GPU communication efficiency by ensuring that data is transferred only when necessary and by overlapping communication with computation. It utilizes CUDA streams to manage concurrent computations and communications effectively. The tool uses two streams: a computation stream and a communication stream. Events synchronize these streams, ensuring that operations are executed in the correct order without introducing deadlocks. The forward pass on the third layer doesn’t start until the all_gather operation is completed (condition 1). Likewise, the all_gather operation on the third layer won’t begin until the forward pass on the first layer that uses the same buffer is completed (condition 2). Since there are no cycles in this scheme, deadlock is impossible. Experimental Results and Performance Gains The implementation of YaFSDP has shown remarkable improvements in training efficiency. In a pre-training scenario with a model having 70 billion parameters, YaFSDP was able to save the resources of approximately 150 GPUs. This translates into significant monthly cost savings, ranging from $0.5 to $1.5 million, depending on the virtual GPU provider or platform. YaFSDP reduces training time by up to 26% compared to existing methods like FSDP and optimizes memory usage, making it possible to train larger models more efficiently. Yandex has made YaFSDP publicly available on GitHub. ML engineers can leverage this tool to enhance the efficiency of their LLM training processes. By open-sourcing YaFSDP, Yandex aims to foster innovation and collaboration in the AI community, enabling developers to train models faster and cost-effectively. YaFSDP represents a significant advancement in LLM training. Addressing the critical challenges of memory consumption and communication inefficiencies enables faster and more efficient training of large language models. Developing large language models requires substantial investments in time and GPU resources, translating directly into high costs. The larger the model, the more pronounced these challenges become. Recently, Yandex has introduced a new solution: YaFSDP , an open-source tool that promises to revolutionize LLM training by significantly reducing GPU resource consumption and training time. In a pre-training scenario involving a model with 70 billion parameters, using YaFSDP can save the resources of approximately 150 GPUs. This translates to potential monthly savings of roughly $0.5 to $1.5 million, depending on the virtual GPU provider or platform. YaFSDP Yandex has made YaFSDP publicly available on GitHub . GitHub GitHub The Challenge of Distributed LLM Training The Challenge of Distributed LLM Training Training LLMs across multiple GPUs involves complex operations that lead to inefficiencies and high memory consumption. One of the main issues is the need to send and receive massive amounts of data between GPUs. For instance, in a typical all_reduce operation, twice the amount of gradient data as there are network parameters must be communicated. In the case of a Llama 70B model, this means transferring 280 GB of data per iteration. Furthermore, weights, gradients, and optimizer states are duplicated across GPUs, leading to an enormous memory load. The Llama 70B model and the Adam optimizer require over 1 TB of memory, far exceeding the typical 80 GB memory capacity of most GPUs. This redundancy severely slows down the training process and often makes it impractical to fit even moderately sized models into GPU memory. Introducing YaFSDP Introducing YaFSDP Yandex’s YaFSDP offers a highly effective solution to these challenges. By focusing on optimizing memory consumption and eliminating communication bottlenecks, YaFSDP enhances the efficiency of LLM training. It works by sharding layers instead of individual parameters, maintaining efficient communications, and avoiding redundant operations. Additionally, YaFSDP pre-allocates buffers for all required data, ensuring that the Torch allocator does not introduce inefficiencies. YaFSDP operates by utilizing two buffers for intermediate weights and gradients, with odd layers using one buffer and even layers using the other. The weights from different layers are stored in the same memory. If the layers have the same structure, they will always be identical. It is crucial to ensure that when you need layer X, the buffer contains the weights for layer X. All parameters will be stored in the corresponding memory chunk within the buffer. Memory Consumption Memory Consumption During training, the primary memory consumers are weights, gradients, optimizer states, buffers, and activations. YaFSDP significantly reduces memory consumption by optimizing how these elements are stored and accessed. Weights, Gradients, and Optimizer States: These depend on the number of processes, and their memory consumption tends to approach zero as the number of processes increases. By sharding these components across GPUs, YaFSDP minimizes duplication and thus reduces memory usage. Weights, Gradients, and Optimizer States : These depend on the number of processes, and their memory consumption tends to approach zero as the number of processes increases. By sharding these components across GPUs, YaFSDP minimizes duplication and thus reduces memory usage. Weights, Gradients, and Optimizer States Buffers consume a constant amount of memory and store intermediate values during computations. Buffers consume a constant amount of memory and store intermediate values during computations. Buffers Activations depend on the model size and the number of tokens processed per GPU. Activations depend on the model size and the number of tokens processed per GPU. Activations Activation Checkpointing Activation Checkpointing Activation checkpointing is a technique that stores only necessary activations during the forward pass and recomputes them during the backward pass. This reduces the memory footprint significantly, as only essential data is stored. For example, in training a Llama 2 70B model with a batch size of 8192 tokens, activation storage can be reduced from over 110 GB to just 5 GB. However, this approach introduces additional computational overhead, which YaFSDP allows to avoid by not using the activation checkpointing for some layers which is possible due to memory optimization. Communication Optimization Communication Optimization YaFSDP improves GPU communication efficiency by ensuring that data is transferred only when necessary and by overlapping communication with computation. It utilizes CUDA streams to manage concurrent computations and communications effectively. The tool uses two streams: a computation stream and a communication stream. Events synchronize these streams, ensuring that operations are executed in the correct order without introducing deadlocks. The forward pass on the third layer doesn’t start until the all_gather operation is completed (condition 1). Likewise, the all_gather operation on the third layer won’t begin until the forward pass on the first layer that uses the same buffer is completed (condition 2). Since there are no cycles in this scheme, deadlock is impossible. Experimental Results and Performance Gains Experimental Results and Performance Gains The implementation of YaFSDP has shown remarkable improvements in training efficiency. In a pre-training scenario with a model having 70 billion parameters, YaFSDP was able to save the resources of approximately 150 GPUs. This translates into significant monthly cost savings, ranging from $0.5 to $1.5 million, depending on the virtual GPU provider or platform. YaFSDP reduces training time by up to 26% compared to existing methods like FSDP and optimizes memory usage, making it possible to train larger models more efficiently. Yandex has made YaFSDP publicly available on GitHub . ML engineers can leverage this tool to enhance the efficiency of their LLM training processes. By open-sourcing YaFSDP, Yandex aims to foster innovation and collaboration in the AI community, enabling developers to train models faster and cost-effectively. GitHub GitHub YaFSDP represents a significant advancement in LLM training. Addressing the critical challenges of memory consumption and communication inefficiencies enables faster and more efficient training of large language models.

YaFSDP - An LLM Training Tool That Cuts GPU Usage by 20% - Is Out Now 

YaFSDP - একটি LLM প্রশিক্ষণ টুল যা GPU ব্যবহার 20% কম করে - এখন শেষ 

YaFSDP – ein LLM-Trainingstool, das die GPU-Nutzung um 20 % senkt – ist jetzt erhältlich

YaFSDP, una herramienta de formación LLM que reduce el uso de GPU en un 20 %, ya está disponible 

YaFSDP - Un outil de formation LLM qui réduit l'utilisation du GPU de 20 % - est maintenant disponible 

YaFSDP - एक LLM प्रशिक्षण उपकरण जो GPU के उपयोग को 20% तक कम करता है - अब उपलब्ध है

YaFSDP - GPU 使用率を 20% 削減する LLM トレーニング ツール - がリリースされました

YaFSDP - GPU 사용량을 20%까지 줄이는 LLM 교육 도구 - 출시됨 

YaFSDP - uma ferramenta de treinamento LLM que reduz o uso de GPU em 20% - já foi lançada 

YaFSDP — инструмент обучения LLM, который сокращает использование графического процессора на 20% — уже доступен 

YaFSDP - GPU Kullanımını %20 Oranında Azaltan Yüksek Lisans Eğitim Aracı - Artık Çıktı 

YaFSDP - Công cụ đào tạo LLM giúp cắt giảm 20% mức sử dụng GPU - Hiện đã ra mắt 

YaFSDP - 一款可将 GPU 使用率降低 20% 的 LLM 培训工具 - 现已推出

This year Bitcoin has caused the biggest buzz in the world of Web3 since its launch in 2009. It has been a huge year for Bitcoin, the world’s largest cryptocurrency. It hit a new all-time high this year as ETFs were approved in the US, Australia, and Hong Kong and institutions jumped on the BTC bandwagon. New exciting ways to interact with Bitcoin also emerged such as Casey Rodarmor’s Runes protocol and there have also been several significant technological breakthroughs making advanced computation possible on Bitcoin for the first time such as BitVMX and SNARKnado. At the same time, a huge wave of new projects emerged claiming to have built new innovations on top of Bitcoin, some of them raising ridiculous amounts of money. Many have already fallen foul to criticism or called scams such as SatoshiVM and Merlin. Others have yet to prove they have real connections to Bitcoin beyond just marketing and hype. But what many people don’t know is that real layers on top of Bitcoin have existed for some time and a dedicated crew of cypherpunks, freedom maximalists, and daring developers have been building on top of Bitcoin for years, taking it from a store of value and transforming it into something that could power the global financial system of the future. In this article, I wanted to shine a light on the true Bitcoin L2s with real use cases and adoption that I believe will gain even greater popularity in 2024 and beyond. Enter the world of Bitcoin L2s & Sidechains So first, what is a Bitcoin Layer Two? I think that Sergio Lerner, long-time Bitcoin contributor and creator of Rootstock, explained it well in a recent interview with CoinTelegraph. “It must use BTC as its native asset and as a settlement mechanism to enforce transactions and demonstrate a functional dependence on Bitcoin.” Many people have different ideas than this but for me this definition is close. It’s a great litmus test to avoid scammers and shilling. Beyond the technological dependence of Bitcoin, my definition is that a true Bitcoin layer two should support the growth and adoption of layer one Bitcoin. A real Bitcoin Layer 2s should ultimately contribute to the success of Bitcoin but it should exist to provide functionality that the layer one was never designed for. This allows Bitcoin to remain secure, pure and home of the hardest form of money while allowing new features and functionality to be developed on top of it. So which Bitcoin L2s should we be exploring in 2024? Lightning Lightning is essentially the payment network for Bitcoin. It allows for near-instantaneous Bitcoin payments and has gained widespread adoption in retail environments. I’ve personally paid for many a beer and coffee with Lightning over the years. It allows for the unilateral transfer of Bitcoin between the main chain and the Lightning layer2 and its laser focus on Bitcoin payments has made it a critical piece of the Bitcoin puzzle. Lightning launched in 2019 with a smart “Pass The Torch” campaign where users would add 10,000 SATS (0.00002451 BTC) to a Lightning payment and pass it one. The campaign reached the likes of Block Inc founder, Jack Dorsey, Litecoin Creator Charlie Lee, Lightning Labs CEO Elizabeth Stark, and CZ of Binance fame. One really cool feature of Lightning Network is that an increasing amount of crypto exchanges have started adding Lightning support as way for users to withdraw and deposit Bitcoin from the exchange. This means faster transaction times and cheaper fees. The most recent exchange to do this was Coinbase in partnership with Lightspark. There are many different wallets available that support Lightning. Some custodial and non-custodial. Its always worth doing your research thoroughly before deciding on which one is for you. Custodial lightning wallets tend to be easier to use but you are reliant on a third party to keep your bitcoin safe. Rootstock Rootstock is the home of apps on Bitcoin. It combines all the programmability of Ethereum with the security and store of value of Bitcoin. Developers have launched lots of different dApps on top of Bitcoin using Rootstock including DEXes like Sovryn, Bitcoin collateralized stablecoins like Money on Chain, and even a naming service known as RIF Name Service which is similar to Ethereum’s ENS. Rootstock uses the Solidity programming language and has all the same developer tools as Ethereum Layer2s making it easy to get started on. It is also supported by MetaMask which makes it easy for users of other EVM chains to easily connect to the network. One neat dApp on Rootstock I discovered  recently is called Asami Club. It pays people in Bitcoin-backed stablecoins for sharing content on social media. The Rootstock ecosystem site has over 100 different apps, exchanges, and wallets listed. Users move their Bitcoin into Rootstock through a bridge known as the PowPeg, described as the most secure, permissionless, and uncensorable Bitcoin bridge. From my research, it seems it was also the first-ever bridge between Bitcoin and another network. Some of the contributors to the network are still working on ways to improve this connection using BitVMX. Rootstock was created by a group of Bitcoin OGs out of Argentina in 2015. It was launched to the public in 2018. The cool thing for me is anyone can build whatever they want on the blockchain in a truly permissionless manner, it’s fully open source and already has lots of individuals and organizations around the world working on it. Liquid For me, Liquid is exciting as it was created by Adam Back, a British cryptographer, and cypherpunk who spoke directly to Satoshi Nakamoto, the legendary creator of Bitcoin. Satoshi credits Adam’s work in the original Bitcoin whitepaper. Liquid has a different focus from Lightning and Rootstock as it is primarily focused on enabling the issuance of security tokens and other digital assets secured by Bitcoin. This is things like the much discussed El Salvador Bitcoin Bond which will be tokenized on Liquid as well as the recently announced tokenized version of MicroStrategy's Class A Common Stock known as CMSTR Note. Liquid is one to watch as we see more institutions exploring how they tokenize assets on Bitcoin whether it be fine art or real estate. In fact, while researching this article I found some very recent news that Bitfinex just launched a $6.25 million token bond on Liquid to fund the construction of a Hilton hotel at the San Salvador International Airport. Sounds like things are taking off! (pun intended) Beyond Bitcoin’s Big Three In this article, I’ve written about what I believe to be three of the critical networks for scaling Bitcoin and allowing it to reach worldwide adoption. I’ve intentionally avoided talking about the typical crypto casino world of meme coins. These can be a lot of fun to play around with but ultimately have no real value or utility to the world. The three examples given above all support real use cases and will prosper whatever current crypto cycle we are in which is a major reason I’m so bullish on them. There is a lot of innovation happening right now on Bitcoin. I’m excited to see what comes next for the world’s most decentralized cryptocurrency as well as the three most important layers built on top of it. I’m sure that the innovations will continue and there is much more to explore. Perhaps in 2035, I’ll write another article diving into just how far Bitcoin’s Big Three have come and what other interesting networks have emerged onto the scene. This year Bitcoin has caused the biggest buzz in the world of Web3 since its launch in 2009. This year Bitcoin has caused the biggest buzz in the world of Web3 since its launch in 2009. This year Bitcoin has caused the biggest buzz in the world of Web3 since its launch in 2009. It has been a huge year for Bitcoin , the world’s largest cryptocurrency. It hit a new all-time high this year as ETFs were approved in the US , Australia , and Hong Kong and institutions jumped on the BTC bandwagon. New exciting ways to interact with Bitcoin also emerged such as Casey Rodarmor’s Runes protocol and there have also been several significant technological breakthroughs making advanced computation possible on Bitcoin for the first time such as BitVMX and SNARKnado . Bitcoin US Australia Hong Kong BTC Runes BitVMX SNARKnado At the same time, a huge wave of new projects emerged claiming to have built new innovations on top of Bitcoin, some of them raising ridiculous amounts of money. Many have already fallen foul to criticism or called scams such as SatoshiVM and Merlin . Others have yet to prove they have real connections to Bitcoin beyond just marketing and hype. SatoshiVM Merlin But what many people don’t know is that real layers on top of Bitcoin have existed for some time and a dedicated crew of cypherpunks, freedom maximalists, and daring developers have been building on top of Bitcoin for years, taking it from a store of value and transforming it into something that could power the global financial system of the future. In this article, I wanted to shine a light on the true Bitcoin L2s with real use cases and adoption that I believe will gain even greater popularity in 2024 and beyond. Enter the world of Bitcoin L2s & Sidechains So first, what is a Bitcoin Layer Two? I think that Sergio Lerner, long-time Bitcoin contributor and creator of Rootstock, explained it well in a recent interview with CoinTelegraph . CoinTelegraph “It must use BTC as its native asset and as a settlement mechanism to enforce transactions and demonstrate a functional dependence on Bitcoin.” “It must use BTC as its native asset and as a settlement mechanism to enforce transactions and demonstrate a functional dependence on Bitcoin.” Many people have different ideas than this but for me this definition is close. It’s a great litmus test to avoid scammers and shilling. Beyond the technological dependence of Bitcoin, my definition is that a true Bitcoin layer two should support the growth and adoption of layer one Bitcoin. A real Bitcoin Layer 2s should ultimately contribute to the success of Bitcoin but it should exist to provide functionality that the layer one was never designed for. This allows Bitcoin to remain secure, pure and home of the hardest form of money while allowing new features and functionality to be developed on top of it. So which Bitcoin L2s should we be exploring in 2024? Lightning Lightning Lightning is essentially the payment network for Bitcoin. It allows for near-instantaneous Bitcoin payments and has gained widespread adoption in retail environments. I’ve personally paid for many a beer and coffee with Lightning over the years. It allows for the unilateral transfer of Bitcoin between the main chain and the Lightning layer2 and its laser focus on Bitcoin payments has made it a critical piece of the Bitcoin puzzle. Lightning launched in 2019 with a smart “Pass The Torch” campaign where users would add 10,000 SATS (0.00002451 BTC) to a Lightning payment and pass it one. The campaign reached the likes of Block Inc founder, Jack Dorsey, Litecoin Creator Charlie Lee, Lightning Labs CEO Elizabeth Stark, and CZ of Binance fame. One really cool feature of Lightning Network is that an increasing amount of crypto exchanges have started adding Lightning support as way for users to withdraw and deposit Bitcoin from the exchange. This means faster transaction times and cheaper fees. The most recent exchange to do this was Coinbase in partnership with Lightspark . Coinbase in partnership with Lightspark There are many different wallets available that support Lightning. Some custodial and non-custodial. Its always worth doing your research thoroughly before deciding on which one is for you. Custodial lightning wallets tend to be easier to use but you are reliant on a third party to keep your bitcoin safe. Rootstock Rootstock is the home of apps on Bitcoin. It combines all the programmability of Ethereum with the security and store of value of Bitcoin. Developers have launched lots of different dApps on top of Bitcoin using Rootstock including DEXes like Sovryn , Bitcoin collateralized stablecoins like Money on Chain , and even a naming service known as RIF Name Service which is similar to Ethereum’s ENS. Rootstock uses the Solidity programming language and has all the same developer tools as Ethereum Layer2s making it easy to get started on. It is also supported by MetaMask which makes it easy for users of other EVM chains to easily connect to the network. Sovryn Money on Chain RIF Name Service One neat dApp on Rootstock I discovered  recently is called Asami Club . It pays people in Bitcoin-backed stablecoins for sharing content on social media. The Rootstock ecosystem site has over 100 different apps, exchanges, and wallets listed. Asami Club Rootstock ecosystem Users move their Bitcoin into Rootstock through a bridge known as the PowPeg, described as the most secure, permissionless, and uncensorable Bitcoin bridge . From my research, it seems it was also the first-ever bridge between Bitcoin and another network. Some of the contributors to the network are still working on ways to improve this connection using BitVMX . most secure, permissionless, and uncensorable Bitcoin bridge improve this connection using BitVMX Rootstock was created by a group of Bitcoin OGs out of Argentina in 2015. It was launched to the public in 2018. The cool thing for me is anyone can build whatever they want on the blockchain in a truly permissionless manner, it’s fully open source and already has lots of individuals and organizations around the world working on it. Liquid For me, Liquid is exciting as it was created by Adam Back, a British cryptographer, and cypherpunk who spoke directly to Satoshi Nakamoto, the legendary creator of Bitcoin. Satoshi credits Adam’s work in the original Bitcoin whitepaper. original Bitcoin whitepaper. Liquid has a different focus from Lightning and Rootstock as it is primarily focused on enabling the issuance of security tokens and other digital assets secured by Bitcoin. This is things like the much discussed El Salvador Bitcoin Bond which will be tokenized on Liquid as well as the recently announced tokenized version of MicroStrategy's Class A Common Stock known as CMSTR Note . El Salvador Bitcoin Bond CMSTR Note Liquid is one to watch as we see more institutions exploring how they tokenize assets on Bitcoin whether it be fine art or real estate. In fact, while researching this article I found some very recent news that Bitfinex just launched a $6.25 million token bond on Liquid to fund the construction of a Hilton hotel at the San Salvador International Airport . Sounds like things are taking off! (pun intended) fund the construction of a Hilton hotel at the San Salvador International Airport (pun intended) Beyond Bitcoin’s Big Three In this article, I’ve written about what I believe to be three of the critical networks for scaling Bitcoin and allowing it to reach worldwide adoption. I’ve intentionally avoided talking about the typical crypto casino world of meme coins. These can be a lot of fun to play around with but ultimately have no real value or utility to the world. The three examples given above all support real use cases and will prosper whatever current crypto cycle we are in which is a major reason I’m so bullish on them. There is a lot of innovation happening right now on Bitcoin. I’m excited to see what comes next for the world’s most decentralized cryptocurrency as well as the three most important layers built on top of it. I’m sure that the innovations will continue and there is much more to explore. Perhaps in 2035, I’ll write another article diving into just how far Bitcoin’s Big Three have come and what other interesting networks have emerged onto the scene.

Bitcoin's Big Three: The Best Layer 2s in 2024

বিটকয়েনের বিগ থ্রি: 2024 সালে সেরা স্তর 2

Die großen Drei von Bitcoin: Die besten Layer 2s im Jahr 2024

Los tres grandes de Bitcoin: las mejores capas 2 en 2024

Les trois grands de Bitcoin : les meilleures couches 2 en 2024

बिटकॉइन के बिग थ्री: 2024 में सर्वश्रेष्ठ लेयर 2

ビットコインのビッグスリー：2024年のベストレイヤー2

비트코인의 빅 3: 2024년 최고의 레이어 2

As três grandes do Bitcoin: as melhores camadas 2 em 2024

Большая тройка Биткойна: лучшие уровни 2 в 2024 году

Bitcoin'in Üç Büyükleri: 2024'ün En İyi Katman 2'leri

Bộ ba lớn của Bitcoin: Lớp 2 tốt nhất năm 2024

比特币三大巨头：2024 年最佳第二层

Supercavitating 'Em Steered Starship: Riding in Plasma Bubble

সুপারক্যাভিটেটিং 'এম স্টিয়ারড স্টারশিপ: এটি প্রতিরোধ করার চেষ্টা করার পরিবর্তে প্লাজমা পুনর্নির্দেশ করা

Superkavitierendes gesteuertes Raumschiff: Plasma umleiten statt verhindern

Supercavitando 'Em Steered Starship: redirigiendo el plasma en lugar de intentar prevenirlo

Supercavitation d'un vaisseau spatial dirigé : rediriger le plasma au lieu d'essayer de l'empêcher

सुपरकैविटेटिंग 'एम ने स्टारशिप को चलाया: इसे रोकने की कोशिश करने के बजाय प्लाज्मा को पुनर्निर्देशित करना

スーパーキャビテーション 操縦された宇宙船: プラズマを防ぐのではなく方向転換する

슈퍼캐비테이션 'Em 조종 우주선: 플라즈마를 방지하는 대신 방향을 바꾸다'

Supercavitando a nave estelar dirigida: redirecionando o plasma em vez de tentar evitá-lo

Суперкавитация управляемого звездолета: перенаправление плазмы вместо попыток ее предотвратить

Süper Kavitasyona Sahip Olanlar Yıldız Gemisini Yönlendirdi: Plazmayı Engellemeye Çalışmak Yerine Yönlendirmek

Supercavitation 'Em Steered Starship: Chuyển hướng plasma thay vì cố gắng ngăn chặn nó

超空泡操纵星舰：改变等离子体方向，而不是试图阻止它

Vector Databases: Unlocking New Dimensions in Similarity Search 

With Stable Diffusion 3 freshly released, I thought it would be nice to make a simple Web-Interface for it. About a month ago, me and a couple of friends released a library for making websites with Python, and this just seemed like too good of an opportunity to pass up. The interface includes support for the classic SD1, SDXL, and the new SD3. Inference is done using the diffusers library from Huggingface, and the interface is built with Rio. Given how new SD3 is in diffusers, you'll spot the odd issue here and there, but I'm sure that'll improve now with thousands of people using it. This is a simple Web-Interface for Stable Diffusion, based on our new Rio framework. We've gotten a lot of requests for larger sample projects, so given the highly anticipated release of Stable Diffusion 3, I figured this would be a great example. With all of that out of the way, let's get started! Install at least one model from the list below to begin using Difoosion. Rio: https://github.com/rio-labs/rio This makes use of Stable Diffusion, which is a sizable AI model. While it can technically run on the CPU, an Nvidia GPU with at least 8GB of VRAM is highly recommended. You can find installation instructions in the project's README: https://gitlab.com/mad-moo/difoosion Get Started Clone this repository git clone git@gitlab.com:Vivern/difoosion.git Install the project and its dependencies (Use a venv if you like) cd difoosion
python -m pip install -e . This will download some large packages like diffusers and torch. Give it a few minutes to complete. Run the interface python -m rio run --release After a few seconds, this will display a URL in the console. Open this URL in your browser to start using the interface. Install Models When you first connect, you won't have any models installed yet. The site will automatically allow you to download one of the built-in ones. diffusers doesn't seem to offer any way to track the download progress of models. That's why you won't be able to see how far the download has progressed in the interface. Check the terminal to see updates. Get cracking! Type a prompt of what you'd like to see, click "Generate," and wait for your masterpiece to be created :) Tech Stack All models are run with the diffusers library from huggingface. There is a thin wrapper around the library to workaround some issues and make it easier to use. The web interface is built with Rio, a new web framework that is designed to be 100% Python, and as easy to use as possible. I'm one of the Rio developers, and this project was meant to both test Rio out in a bigger project, as well as be a showcase & reference to other Rio developers. Dependencies are managed with rye. Caveats Stable Diffusion 3 is still very new, and support in diffusers immature. This has led to a couple of known problems: Generating non-square images can lead to errors from pytorch. Stick to aspect ratios of 1:1 for now. Freeing up memory is very unreliable. On some of my devices, the model keeps gobbling up more and more RAM as time goes on, while on others, it seems to free up just fine. Keep a close eye on your memory usage. With Stable Diffusion 3 freshly released, I thought it would be nice to make a simple Web-Interface for it. About a month ago, me and a couple of friends released a library for making websites with Python, and this just seemed like too good of an opportunity to pass up. The interface includes support for the classic SD1, SDXL, and the new SD3. Inference is done using the diffusers library from Huggingface, and the interface is built with Rio . Given how new SD3 is in diffusers , you'll spot the odd issue here and there, but I'm sure that'll improve now with thousands of people using it. diffusers Rio diffusers This is a simple Web-Interface for Stable Diffusion, based on our new Rio framework. We've gotten a lot of requests for larger sample projects, so given the highly anticipated release of Stable Diffusion 3, I figured this would be a great example. With all of that out of the way, let's get started! Install at least one model from the list below to begin using Difoosion. Rio: https://github.com/rio-labs/rio https://github.com/rio-labs/rio This makes use of Stable Diffusion, which is a sizable AI model. While it can technically run on the CPU, an Nvidia GPU with at least 8GB of VRAM is highly recommended. You can find installation instructions in the project's README: https://gitlab.com/mad-moo/difoosion This makes use of Stable Diffusion, which is a sizable AI model. While it can technically run on the CPU, an Nvidia GPU with at least 8GB of VRAM is highly recommended. https://gitlab.com/mad-moo/difoosion Get Started Clone this repository Clone this repository git clone git@gitlab.com:Vivern/difoosion.git git clone git@gitlab.com:Vivern/difoosion.git Install the project and its dependencies (Use a venv if you like) Install the project and its dependencies (Use a venv if you like) cd difoosion
python -m pip install -e . cd difoosion
python -m pip install -e . This will download some large packages like diffusers and torch . Give it a few minutes to complete. diffusers torch Run the interface Run the interface python -m rio run --release python -m rio run --release After a few seconds, this will display a URL in the console. Open this URL in your browser to start using the interface. Install Models Install Models When you first connect, you won't have any models installed yet. The site will automatically allow you to download one of the built-in ones. diffusers doesn't seem to offer any way to track the download progress of models. That's why you won't be able to see how far the download has progressed in the interface. Check the terminal to see updates. diffusers Get cracking! Get cracking! Type a prompt of what you'd like to see, click "Generate," and wait for your masterpiece to be created :) Tech Stack All models are run with the diffusers library from huggingface. There is a thin wrapper around the library to workaround some issues and make it easier to use. All models are run with the diffusers library from huggingface . There is a thin wrapper around the library to workaround some issues and make it easier to use. diffusers library from huggingface The web interface is built with Rio, a new web framework that is designed to be 100% Python, and as easy to use as possible. I'm one of the Rio developers, and this project was meant to both test Rio out in a bigger project, as well as be a showcase & reference to other Rio developers. The web interface is built with Rio , a new web framework that is designed to be 100% Python, and as easy to use as possible. I'm one of the Rio developers, and this project was meant to both test Rio out in a bigger project, as well as be a showcase & reference to other Rio developers. Rio Dependencies are managed with rye. Dependencies are managed with rye . rye Caveats Stable Diffusion 3 is still very new, and support in diffusers immature. This has led to a couple of known problems: diffusers Generating non-square images can lead to errors from pytorch. Stick to aspect ratios of 1:1 for now. Generating non-square images can lead to errors from pytorch. Stick to aspect ratios of 1:1 for now. Freeing up memory is very unreliable. On some of my devices, the model keeps gobbling up more and more RAM as time goes on, while on others, it seems to free up just fine. Keep a close eye on your memory usage. Freeing up memory is very unreliable. On some of my devices, the model keeps gobbling up more and more RAM as time goes on, while on others, it seems to free up just fine. Keep a close eye on your memory usage.

Difoosion - A Simple Web-Interface for Stable Diffusion Models

Authors:
(1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution};
(2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution};
(3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Table of Links Abstract and Intro
Dataloaders
Experimental Setup
Numerical Results
Discussion
Related Work
Conclusion, Acknowledgments, and References
A. Numerical Results Cont. 2. DATALOADERS In the process of training a Deep Learning model, the dataset needs to be read from memory and pre-processed, before it can be passed as input to the model. This operation requires loading the data into memory all at once. In most cases and, especially, with large datasets, a memory shortage arises due to the limited amount of it available in the system, which also deteriorates the system’s response time. This bottleneck is often remedied in deep learning libraries by using a so-called dataloader. This structure provides a way to iterate over the dataset by leveraging parallel processing, pre-fetching, batching, and other techniques to reduce data loading time and memory overhead as much as possible (Paszke et al., 2019). The main goal of a dataloader is to perform the actions responsible for transferring data samples from a storage location to the memory co-located with the processing units for training in order to form a batch of samples to be fed into the model. These actions are restrained by the storage system’s bandwidth and specifically its I/O performance. Thus, depending on the system’s hardware specifications, its filesystem serving it, and the throughput of the link with the computing units, it can have an immense influence on the total amount of time needed to complete the training. The following specification of the dataloader component is mainly focused towards PyTorch (torch.DataLoader() (PyTorch Core Team)), while its TensorFlow counterpart (tf.Dataset() (Abadi et al., 2015)), albeit not the same, bears great similarities. When employing a dataloader, apart from providing the dataset for input, the user has the option to configure a number of hyperparameters, tailored to their needs and resources. A common one available in all dataloaders is the batch size, which, as mentioned before, defines the number of samples that will be used before updating the internal model parameters. This parameter is intrinsically linked with the concept of ”mini-batch” in stochastic gradient descent (Hinton et al., 2012) and therefore, it is one of the first parameters that usually undergoes fine-tuning when one needs to achieve better training results. Secondly, the user can define the sampling approach, which determines the strategy for drawing samples from the dataset and inserting them into a batch. This could include selecting samples based on specific criteria, as well as a probability distribution. In this step exists the option of shuffling, where the samples can be rearranged before every dataset iteration, with the goal usually being to improve the generalization of the training model. Another parameter is the collate/padding function, which essentially specifies the process of linking together all the individual samples inside a batch (think of stacking vectors into a tensor), in order to form a single element to be fed as input to the training model. Moreover, the dataloader can be configured to automatically store fetched data samples in pinned (page-locked) memory, thus enabling faster data transfer to CUDA-enabled devices. Dataloaders come with a component called workers, whose purpose is to optimize this data-transferring process. Workers are defined as sub-processes responsible to carry out the data loading in an asynchronous fashion. When creating an instance of a dataloader, the user has the option to specify the number of workers that will be spawned and will be in control of this operation. If the number of workers is equal to zero, no sub-processes will be created, which, in turn, means that data fetching happens synchronously in the same process and, thus, the computing units (GPU) have to wait for the data loading to be completed (PyTorch Core Team). Reversely, there will be generated sub-processes equal to the number of workers, which will prevent the blocking of computation code with data loading. This is accomplished by pre-fetching future batches in advance to be ready when needed. This paper is available on arxiv under CC 4.0 license. Authors: (1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Authors: Authors: (1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Table of Links Abstract and Intro Dataloaders Experimental Setup Numerical Results Discussion Related Work Conclusion, Acknowledgments, and References A. Numerical Results Cont. Abstract and Intro Abstract and Intro Dataloaders Dataloaders Experimental Setup Experimental Setup Numerical Results Numerical Results Discussion Discussion Related Work Related Work Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References A. Numerical Results Cont. A. Numerical Results Cont. 2. DATALOADERS In the process of training a Deep Learning model, the dataset needs to be read from memory and pre-processed, before it can be passed as input to the model. This operation requires loading the data into memory all at once. In most cases and, especially, with large datasets, a memory shortage arises due to the limited amount of it available in the system, which also deteriorates the system’s response time. This bottleneck is often remedied in deep learning libraries by using a so-called dataloader. This structure provides a way to iterate over the dataset by leveraging parallel processing, pre-fetching, batching, and other techniques to reduce data loading time and memory overhead as much as possible (Paszke et al., 2019). The main goal of a dataloader is to perform the actions responsible for transferring data samples from a storage location to the memory co-located with the processing units for training in order to form a batch of samples to be fed into the model. These actions are restrained by the storage system’s bandwidth and specifically its I/O performance. Thus, depending on the system’s hardware specifications, its filesystem serving it, and the throughput of the link with the computing units, it can have an immense influence on the total amount of time needed to complete the training. The following specification of the dataloader component is mainly focused towards PyTorch (torch.DataLoader() (PyTorch Core Team)), while its TensorFlow counterpart (tf.Dataset() (Abadi et al., 2015)), albeit not the same, bears great similarities. When employing a dataloader, apart from providing the dataset for input, the user has the option to configure a number of hyperparameters, tailored to their needs and resources. A common one available in all dataloaders is the batch size, which, as mentioned before, defines the number of samples that will be used before updating the internal model parameters. This parameter is intrinsically linked with the concept of ”mini-batch” in stochastic gradient descent (Hinton et al., 2012) and therefore, it is one of the first parameters that usually undergoes fine-tuning when one needs to achieve better training results. Secondly, the user can define the sampling approach, which determines the strategy for drawing samples from the dataset and inserting them into a batch. This could include selecting samples based on specific criteria, as well as a probability distribution. In this step exists the option of shuffling, where the samples can be rearranged before every dataset iteration, with the goal usually being to improve the generalization of the training model. Another parameter is the collate/padding function, which essentially specifies the process of linking together all the individual samples inside a batch (think of stacking vectors into a tensor), in order to form a single element to be fed as input to the training model. Moreover, the dataloader can be configured to automatically store fetched data samples in pinned (page-locked) memory, thus enabling faster data transfer to CUDA-enabled devices. Dataloaders come with a component called workers, whose purpose is to optimize this data-transferring process. Workers are defined as sub-processes responsible to carry out the data loading in an asynchronous fashion. When creating an instance of a dataloader, the user has the option to specify the number of workers that will be spawned and will be in control of this operation. If the number of workers is equal to zero, no sub-processes will be created, which, in turn, means that data fetching happens synchronously in the same process and, thus, the computing units (GPU) have to wait for the data loading to be completed (PyTorch Core Team). Reversely, there will be generated sub-processes equal to the number of workers, which will prevent the blocking of computation code with data loading. This is accomplished by pre-fetching future batches in advance to be ready when needed. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

An Overview of the Data-Loader Landscape: Dataloaders

ডেটা-লোডার ল্যান্ডস্কেপের একটি ওভারভিউ: ডেটালোডার

Ein Überblick über die Dataloader-Landschaft: Dataloader

Una descripción general del panorama de los cargadores de datos: cargadores de datos

Un aperçu du paysage des chargeurs de données : les chargeurs de données

डेटा-लोडर परिदृश्य का अवलोकन: डेटालोडर

データローダーの概要: データローダー

Data-Loader 환경 개요: Dataloader

Uma visão geral do cenário do Data-Loader: Dataloaders

Обзор ландшафта загрузчиков данных: загрузчики данных

Veri Yükleyici Ortamına Genel Bakış: Veri Yükleyiciler

Tổng quan về bối cảnh trình tải dữ liệu: Trình tải dữ liệu

数据加载器概况：数据加载器

Authors:
(1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution};
(2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution};
(3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Table of Links Abstract and Intro
Dataloaders
Experimental Setup
Numerical Results
Discussion
Related Work
Conclusion, Acknowledgments, and References
A. Numerical Results Cont. 4. NUMERICAL RESULTS For the first set of experiments, we evaluated the performance of all libraries when changing the number of workers (see 2) as well as the batch size. These experiments were run in a local server with the following specifications: Intel(R) Core(TM) i9-10900K, 2 x NVIDIA GeForce RTX 3090, and an HDD with 10TB of storage (6GB/s) [5]. We evaluated the three modes: default (single GPU), distributed (two GPUs), and filtering (single GPU) for all possible combinations of 0, 1, and 2 workers. For CIFAR10 and RANDOM, the batch size was either 16, 64, or 128. For CoCo, we used smaller values: 1, 2, and 4. All these experiments used a cutoff of 10 and the variant that runs the model (forward & backward pass). 4.1 Speed as a function of batch size and workers The first thing we notice while examining the experiments is that the variance between libraries depends on the problem and the dataset used. Figure 4 shows one such comparison for CIFAR10 on a single GPU, whereas 5 show the same result for CoCo, also on a single GPU. This was to be expected given that in the latter, the time taken to compute the forward and backward pass dominates the overall running time, which is not the case for image classification. You may also note the overall difference in speed: from 4000 samples per second to only 20. Secondly, we point out that increasing the batch size increases processing speed in almost all cases. However, this is not the case for the number of workers. We can observe in Figure 6 that FFCV performance degrades as the number of workers increases, while Deep Lake plateaus at 1 worker and not 2. One explanation is that the libraries have their own internal algorithms that decide how to span threads and processes as needed. The point above is relevant for users of these libraries, as experience with one of them might not translate well into another one. 4.2 Speed Gains when using DDP A desirable feature of a dataloader is its ability to scale linearly with the number of GPUs. This is not always possible and depends on the internal loading mechanism of each library. We explore how these libraries performed by comparing the speed increase when using one or two GPUs. Figure 7 shows the results for the RANDOM dataset. Each bar represents the maximum speed achieved across all batch sizes, number of workers, and repetitions. In a way, this reflects the maximum speed achievable by the library. We observe that libraries speed up about 40%, less than half of a linear increase on average. Two cases are particularly surprising. On the one hand, Torchdata performs worse with two GPUs than on a single one. On the other hand, FFCV achieved a speed increase of more than theoretically possible. There are several artifacts that can be at play here, but most likely, it is due to the limited number of repetitions we were able to run (due to limited resources). Also, this might indicate a misconfiguration in Torchdata: the documentation on running experiments in multi-GPU environments is limited for most libraries. 4.3 Comparison between with and without forward and backward pass As we discussed when presenting Algorithm 1, we had to decide whether we would incorporate the forward and backward passes into the speed calculation. There are arguments for both. On the one hand, including the forward and backward passes better reflect the algorithm’s actual training time. At the same time, some libraries might preemptively optimize steps normally done during the forward pass, so stopping there would seem as if they take longer than they do. On the other hand, if the time taken by the forward and backward pass is much larger than the time it takes for only loading the data, including that time in the calculation would inevitably hide the difference between libraries. To understand if the behavior change was noticeable, we used the RANDOM dataset to compare the difference in average speed when including the two model operations in the calculation and when not. The results are presented in Figure 8. We can observe that most libraries have a slightly increased speed when excluding the model (except for FFCV, whose performance drops in half), and most importantly, the relative performance among libraries remains almost the same. 4.4 Speed trade-offs when filtering data For our filtering experiments, we selected two classes to keep for CIFAR10 and RANDOM: dog and truck, and 0 and 13, respectively. For CoCO we selected three classes: pizza, couch, cat. We observed that most libraries do not have a good filtering mechanism that avoids iterating over the whole dataset. For example, our PyTorch filtering implementation requires building a custom sampler with the indices of the desired images. This is quite fast for a small dataset but becomes unfeasible for large datasets: filtering CoCo using PyTorch was prohibitively expensive. In general, performance was quite similar when filtering and when not[6]. Similarly to Figure 7, in Figure 9, we can see the slowdown as a result of filtering: even though it was considerable for Torchdata and Webdataset, we saw a reversal of the results when working with the CoCo Dataset. 4.5 Training while streaming over the network Ideally, we could decouple the dataset storage from the machine learning training process and simply connect the database storing our data to the ML framework of choice, regardless of where the two are located. That involves sending the training data over a network and losing considerable speed. With the high costs involved in renting GPU accelerated hardware on the cloud, it might seem that the convenience is not worth it. But is it not? Some of the libraries considered in this paper allow specifying a dataset accessible via the internet: Webdataset, Hub, and Deep Lake are particularly good at this[7]. The question then becomes: how big is the tradeoff between ease of usage and running time? We set up the following experiment to offer some insight into this question. We ran two full epochs of the RANDOM dataset for the three libraries: Hub, Deep Lake, and Webdataset, while changing the origin of the data. Three locations were considered: a local copy in the machine running the experiments’ hard drive, a copy in an S3 bucket (in the closest region to our machine), and a copy stored in MinIO (an open source equivalent of S3 that can be hosted locally) running in a similar machine within the same local network (both machines were connected via WiFi). The experiments’ computer had an Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz, 16 GB of RAM, NVIDIA GeForce RTX 2070 Rev, and a Samsung SSD 850 hard drive with 256 GB of storage. Regarding latency, the Round Trip Time from the workstation running the experiments to the MinIO server (located in the same room and in the same local WiFi) took 59.2 ± 58.5ms (min. 8.8ms), while to the S3-bucket in AWS servers took 17.3 ± 1.3ms (min. 14.8ms). Figure 10 depicts the total running times for the nine experiments, and the white percentages denote the slowdown (increase in running time) compared to the local case. We can observe that even though for Hub and Webdataset there is a significant increase when moving to AWS, Deep Lake managed to maintain almost the same speed with an increase of only 13%. Another helpful insight could be extracted from the result: the MinIO setting shows a slowdown almost twice as bad as the AWS setting, as seen in Figure 10. This output could be explained primarily by the difference in average Round Trip Times shown above, highlighting that local networks (e.g., internal company networks[8]) might not be the most efficient way to host datasets due to their complex configurations and restrictions. Furthermore, this result also indicates that the storage serving the datasets over the network plays a crucial role in enabling remote training and might trigger questions on best practices to serve datasets. Namely, data from S3 is served in parallel from different servers with load balancing while we had access to a single MinIO instance. The plots for all additional experiments can be found in Appendix A. This paper is available on arxiv under CC 4.0 license. [5] https://www.seagate.com/www-content/productcontent/barracuda-fam/barracuda-new/enus/docs/100835983b.pdf [6] in speed. Building the sampler for PyTorch is done upfront, considerably affecting the total running time. [7] Squirrel has this capability, but we did not manage to specify a MinIO address, so we excluded it from the comparison. We had a similar issue with Torchdata. [8] In this case a university network Authors: (1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Authors: Authors: (1) Iason Ofeidis, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (2) Diego Kiedanski, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven {Equal contribution}; (3) Leandros TassiulasLevon Ghukasyan, Activeloop, Mountain View, CA, USA, Department of Electrical Engineering, and Yale Institute for Network Science, Yale University, New Haven. Table of Links Abstract and Intro Dataloaders Experimental Setup Numerical Results Discussion Related Work Conclusion, Acknowledgments, and References A. Numerical Results Cont. Abstract and Intro Abstract and Intro Dataloaders Dataloaders Experimental Setup Experimental Setup Numerical Results Numerical Results Discussion Discussion Related Work Related Work Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References A. Numerical Results Cont. A. Numerical Results Cont. 4. NUMERICAL RESULTS For the first set of experiments, we evaluated the performance of all libraries when changing the number of workers (see 2) as well as the batch size. These experiments were run in a local server with the following specifications: Intel(R) Core(TM) i9-10900K, 2 x NVIDIA GeForce RTX 3090, and an HDD with 10TB of storage (6GB/s) [5]. We evaluated the three modes: default (single GPU), distributed (two GPUs), and filtering (single GPU) for all possible combinations of 0, 1, and 2 workers. For CIFAR10 and RANDOM, the batch size was either 16, 64, or 128. For CoCo, we used smaller values: 1, 2, and 4. All these experiments used a cutoff of 10 and the variant that runs the model (forward & backward pass). 4.1 Speed as a function of batch size and workers The first thing we notice while examining the experiments is that the variance between libraries depends on the problem and the dataset used. Figure 4 shows one such comparison for CIFAR10 on a single GPU, whereas 5 show the same result for CoCo, also on a single GPU. This was to be expected given that in the latter, the time taken to compute the forward and backward pass dominates the overall running time, which is not the case for image classification. You may also note the overall difference in speed: from 4000 samples per second to only 20. Secondly, we point out that increasing the batch size increases processing speed in almost all cases. However, this is not the case for the number of workers. We can observe in Figure 6 that FFCV performance degrades as the number of workers increases, while Deep Lake plateaus at 1 worker and not 2. One explanation is that the libraries have their own internal algorithms that decide how to span threads and processes as needed. The point above is relevant for users of these libraries, as experience with one of them might not translate well into another one. 4.2 Speed Gains when using DDP A desirable feature of a dataloader is its ability to scale linearly with the number of GPUs. This is not always possible and depends on the internal loading mechanism of each library. We explore how these libraries performed by comparing the speed increase when using one or two GPUs. Figure 7 shows the results for the RANDOM dataset. Each bar represents the maximum speed achieved across all batch sizes, number of workers, and repetitions. In a way, this reflects the maximum speed achievable by the library. We observe that libraries speed up about 40%, less than half of a linear increase on average. Two cases are particularly surprising. On the one hand, Torchdata performs worse with two GPUs than on a single one. On the other hand, FFCV achieved a speed increase of more than theoretically possible. There are several artifacts that can be at play here, but most likely, it is due to the limited number of repetitions we were able to run (due to limited resources). Also, this might indicate a misconfiguration in Torchdata: the documentation on running experiments in multi-GPU environments is limited for most libraries. 4.3 Comparison between with and without forward and backward pass As we discussed when presenting Algorithm 1, we had to decide whether we would incorporate the forward and backward passes into the speed calculation. There are arguments for both. On the one hand, including the forward and backward passes better reflect the algorithm’s actual training time. At the same time, some libraries might preemptively optimize steps normally done during the forward pass, so stopping there would seem as if they take longer than they do. On the other hand, if the time taken by the forward and backward pass is much larger than the time it takes for only loading the data, including that time in the calculation would inevitably hide the difference between libraries. To understand if the behavior change was noticeable, we used the RANDOM dataset to compare the difference in average speed when including the two model operations in the calculation and when not. The results are presented in Figure 8. We can observe that most libraries have a slightly increased speed when excluding the model (except for FFCV, whose performance drops in half), and most importantly, the relative performance among libraries remains almost the same. 4.4 Speed trade-offs when filtering data For our filtering experiments, we selected two classes to keep for CIFAR10 and RANDOM: dog and truck, and 0 and 13, respectively. For CoCO we selected three classes: pizza, couch, cat. We observed that most libraries do not have a good filtering mechanism that avoids iterating over the whole dataset. For example, our PyTorch filtering implementation requires building a custom sampler with the indices of the desired images. This is quite fast for a small dataset but becomes unfeasible for large datasets: filtering CoCo using PyTorch was prohibitively expensive. In general, performance was quite similar when filtering and when not[6]. Similarly to Figure 7, in Figure 9, we can see the slowdown as a result of filtering: even though it was considerable for Torchdata and Webdataset, we saw a reversal of the results when working with the CoCo Dataset. 4.5 Training while streaming over the network Ideally, we could decouple the dataset storage from the machine learning training process and simply connect the database storing our data to the ML framework of choice, regardless of where the two are located. That involves sending the training data over a network and losing considerable speed. With the high costs involved in renting GPU accelerated hardware on the cloud, it might seem that the convenience is not worth it. But is it not? Some of the libraries considered in this paper allow specifying a dataset accessible via the internet: Webdataset, Hub, and Deep Lake are particularly good at this[7]. The question then becomes: how big is the tradeoff between ease of usage and running time? We set up the following experiment to offer some insight into this question. We ran two full epochs of the RANDOM dataset for the three libraries: Hub, Deep Lake, and Webdataset, while changing the origin of the data. Three locations were considered: a local copy in the machine running the experiments’ hard drive, a copy in an S3 bucket (in the closest region to our machine), and a copy stored in MinIO (an open source equivalent of S3 that can be hosted locally) running in a similar machine within the same local network (both machines were connected via WiFi). The experiments’ computer had an Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz, 16 GB of RAM, NVIDIA GeForce RTX 2070 Rev, and a Samsung SSD 850 hard drive with 256 GB of storage. Regarding latency, the Round Trip Time from the workstation running the experiments to the MinIO server (located in the same room and in the same local WiFi) took 59.2 ± 58.5ms (min. 8.8ms), while to the S3-bucket in AWS servers took 17.3 ± 1.3ms (min. 14.8ms). Figure 10 depicts the total running times for the nine experiments, and the white percentages denote the slowdown (increase in running time) compared to the local case. We can observe that even though for Hub and Webdataset there is a significant increase when moving to AWS, Deep Lake managed to maintain almost the same speed with an increase of only 13%. Another helpful insight could be extracted from the result: the MinIO setting shows a slowdown almost twice as bad as the AWS setting, as seen in Figure 10. This output could be explained primarily by the difference in average Round Trip Times shown above, highlighting that local networks (e.g., internal company networks[8]) might not be the most efficient way to host datasets due to their complex configurations and restrictions. Furthermore, this result also indicates that the storage serving the datasets over the network plays a crucial role in enabling remote training and might trigger questions on best practices to serve datasets. Namely, data from S3 is served in parallel from different servers with load balancing while we had access to a single MinIO instance. The plots for all additional experiments can be found in Appendix A. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [5] https://www.seagate.com/www-content/productcontent/barracuda-fam/barracuda-new/enus/docs/100835983b.pdf [6] in speed. Building the sampler for PyTorch is done upfront, considerably affecting the total running time. [7] Squirrel has this capability, but we did not manage to specify a MinIO address, so we excluded it from the comparison. We had a similar issue with Torchdata. [8] In this case a university network

An Overview of the Data-Loader Landscape: Numerical Results

ডেটা-লোডার ল্যান্ডস্কেপের একটি ওভারভিউ: সংখ্যাসূচক ফলাফল

Ein Überblick über die Data-Loader-Landschaft: Numerische Ergebnisse

Una descripción general del panorama del cargador de datos: resultados numéricos

Un aperçu du paysage des chargeurs de données : résultats numériques

डेटा-लोडर परिदृश्य का अवलोकन: संख्यात्मक परिणाम

データローダーの概要: 数値結果

Data-Loader 환경 개요: 수치 결과

Uma visão geral do cenário do Data-Loader: resultados numéricos

Обзор ландшафта загрузчиков данных: численные результаты

Veri Yükleyici Ortamına Genel Bakış: Sayısal Sonuçlar

Tổng quan về bối cảnh trình tải dữ liệu: Kết quả bằng số

数据加载器概况：数值结果

Stories, News & Mentions about TORC

Torc’s Stories on HackerNoon