DataStema

datastema

Large Language Models (LLMs) are a subset of (AI) that are designed to understand and generate natural language. These models are trained on vast amounts of textual data and use complex algorithms to learn patterns in language. LLMs have been used to develop a range of applications, from chatbots to language translation services. In this article, we will discuss how to create an LLM on a single computer and provide source code examples. artificial intelligence Prerequisites Before we begin, it’s essential to ensure that your computer meets the following requirements: A powerful computer with at least 16GB of RAM and 1TB of free storage A modern GPU, such as an NVIDIA GeForce or AMD Radeon, with at least 8GB of video memory A Linux operating system, such as Ubuntu or CentOS Basic knowledge of the command line interface (CLI) and Python programming language Step 1: Install Anaconda Anaconda is a popular open-source platform for data science and machine learning. It includes various tools and libraries that we will use to create an LLM. To install Anaconda, follow these steps: Download the Anaconda installer from the official website: https://www.anaconda.com/products/individual Open the terminal and navigate to the directory where the installer was downloaded Run the installer by typing the following command: bash Anaconda3–2021.11-Linux-x86_ 64.sh Follow the instructions on the screen to complete the installation Step 2: Create a Python environment We will create a Python environment to install the necessary libraries and dependencies for the LLM. To create an environment, follow these steps: Open the terminal and type the following command: conda create — name lm python=3.8 Activate the environment by typing: conda activate lm Step 3: Install TensorFlow TensorFlow is an open-source platform for building and training machine learning models. We will use TensorFlow to create the LLM. To install TensorFlow, follow these steps: Type the following command in the terminal: pip install tensorflow Verify that TensorFlow is installed by importing it in Python: import tensorflow as tf Step 4: Download a pre-trained LLM Training an LLM from scratch requires an enormous amount of computational power and time. Luckily, pre-trained models are available that we can fine-tune for our specific use case. One of the most popular pre-trained LLMs is GPT-2 (Generative Pre-trained Transformer 2). To download the pre-trained GPT-2 model, follow these steps: Open the terminal and type the following command: git clone https://github.com/openai/gpt-2.git Navigate to the gpt-2 directory by typing: cd gpt-2 Download the pre-trained model by typing: python download_ 117M model.py Step 5: Fine-tune the pre-trained LLM Fine-tuning a pre-trained LLM involves training the model on a specific dataset for a specific task. In our example, we will fine-tune the pre-trained GPT-2 model on a text generation task. To fine-tune the model, follow these steps: Create a new directory for the fine-tuned model by typing: mkdir my_model Navigate to the my_model directory by typing: cd my_model Create a new Python file by typing: touch train.py Open the file in a text editor and paste the following code: train.py import tensorflow as tf
import numpy as np
import os
import json
import random
import time
import argparse

# Define the command-line arguments
parser = argparse.ArgumentParser()
parser.add_argument("--dataset_path", type=str, required=True,
                    help="Path to the dataset")
parser.add_argument("--model_path", type=str, required=True,
                    help="Path to the pre-trained model")
parser.add_argument("--output_path", type=str, required=True,
                    help="Path to save the fine-tuned model")
parser.add_argument("--batch_size", type=int, default=16,
                    help="Batch size for training")
parser.add_argument("--epochs", type=int, default=1,
                    help="Number of epochs to train for")
args = parser.parse_args()
# Load the pre-trained GPT-2 model
with open(os.path.join(args.model_path, "hparams.json"), "r") as f:
    hparams = json.load(f)
model = tf.compat.v1.estimator.Estimator(
    model_fn=model_fn,
    model_dir=args.output_path,
    params=hparams,
    config=tf.compat.v1.estimator.RunConfig(
        save_checkpoints_steps=5000,
        keep_checkpoint_max=10,
        save_summary_steps=5000
    )
)
# Define the input function for the dataset
def input_fn(mode):
    dataset = tf.data.TextLineDataset(args.dataset_path)
    dataset = dataset.repeat()
    dataset = dataset.shuffle(buffer_size=10000)
    dataset = dataset.batch(args.batch_size)
    dataset = dataset.map(lambda x: tf.strings.substr(x, 0, hparams["n_ctx"]))
    iterator = dataset.make_one_shot_iterator()
    return iterator.get_next()
# Define the training function
def train():
    for epoch in range(args.epochs):
        model.train(input_fn=lambda: input_fn(tf.estimator.ModeKeys.TRAIN))
        print(f"Epoch {epoch+1} completed.")
# Start the training
train() In the code above, we define the command-line arguments for the dataset path, pre-trained model path, output path, batch size, and number of epochs to train for. We then load the pre-trained GPT-2 model and define the input function for the dataset. Finally, we define the training function and start the training. Step 6: Generate text using the fine-tuned LLM Once the LLM is fine-tuned, we can use it to generate text. To generate text, follow these steps: Open the terminal and navigate to the my_model directory by typing: cd my_model Create a new Python file by typing: touch generate.py Open the file in a text editor and paste the following code: generate.py import tensorflow as tf
import numpy as np
import os
import json
import random
import time
import argparse

# Define the command-line arguments
parser = argparse.ArgumentParser()
parser.add_argument("--model_path", type=str, required=True,
                    help="Path to the fine-tuned model")
parser.add_argument("--length", type=int, default=100,
                    help="Length of the generated text")
parser.add_argument("--temperature", type=float, default=0.7,
                    help="Temperature for text generation")
args = parser.parse_args()
# Load the fine-tuned model
with open(os.path.join(args.model_path, "hparams.json"), "r") as f:
    hparams = json.load(f)
model_fn = model_fn(hparams, tf.estimator.ModeKeys.PREDICT)
model = tf.compat.v1.estimator.Estimator(
    model_fn=model_fn,
    model_dir=args.model_path,
    params=hparams
)
# Define the generation function
def generate_text(length, temperature):
    start_token = "<|startoftext|>"
    tokens = tokenizer.convert_tokens_to_ids([start_token])
    token_length = len(tokens)
    while token_length < length:
        prediction_input = np.array(tokens[-hparams["n_ctx"]:])
        output = list(model.predict(input_fn=lambda: [[prediction_input]]))[0]["logits"]
        logits = output[-1] / temperature
        logits = logits - np.max(logits)
        probs = np.exp(logits) / np.sum(np.exp(logits))
        token = np.random.choice(range(hparams["n_vocab"]), p=probs)
        tokens.append(token)
        token_length += 1
    output_text = tokenizer.convert_ids_to_tokens(tokens)
    output_text = "".join(output_text).replace("▁", " ")
    output_text = output_text.replace(start_token, "")
    return output_text
# Generate text
text = generate_text(args.length, args.temperature)
print(text) In the code above, we define the command-line arguments for the fine-tuned model path, length of the generated text, and temperature for text generation. We then load the fine-tuned model and define the generation function. Finally, we generate text using the generate_text function and print the output, and we are done! Conclusion In this article, we have learned how to create a Large Language Model (LLM) on a single computer using TensorFlow and the GPT-2 architecture. We started by installing TensorFlow and downloading the GPT-2 code from the OpenAI GitHub repository. We then trained a GPT-2 model on a dataset and fine-tuned the pre-trained GPT-2 model using the same dataset. Finally, we generated text using the fine-tuned LLM. By following the steps in this article, you can create your own LLM and generate text for various tasks such as language translation, chatbots, and content generation. References Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. OpenAI. (2019). GPT-2: Language models are unsupervised multitask learners. Github. Retrieved from . https://github.com/openai/gpt-2 TensorFlow. (2021). TensorFlow: An open source machine learning framework for everyone. Retrieved from . https://www.tensorflow.org/ Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … & Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. Also published here.

How to Host a Large Language Model Locally

স্থানীয়ভাবে একটি বড় ভাষার মডেল কীভাবে হোস্ট করবেন

So hosten Sie ein großes Sprachmodell lokal

Cómo alojar localmente un modelo de lenguaje grande

Comment héberger localement un grand modèle de langage

बड़े भाषा मॉडल को स्थानीय रूप से कैसे होस्ट करें

大規模な言語モデルをローカルでホストする方法

대규모 언어 모델을 로컬로 호스팅하는 방법

Como hospedar um modelo de idioma grande localmente

Büyük Bir Dil Modelini Yerel Olarak Nasıl Barındırırsınız?

Cách lưu trữ một mô hình ngôn ngữ lớn cục bộ

如何在本地托管大型语言模型

In this article, I want to tell you how to connect and further generate Allure reports for a test project if your test project is made on the Citrus Framework. I hope you know about Allure framework, which is designed to generate reports after passing tests. The report opens on a browser page, where you can see which tests passed successfully and which did not, you can also see at which test step the expected result did not match. Since the Citrus Framework can use the TestNG or Junit engines, you need to include the Allure library according to the engine you are using. I will show the example of TestNG and the project builder we use Maven. 1. First of all, the Allure distribution must be installed locally and the path to the distribution must be written in PATH, similar to installing Maven. The distribution can be taken from here by choosing the latest version: https: //repo.maven.apache.org/maven2/io/qameta /allure/allure-commandline You can check if you have Allure locally with this command in cmd: allure --version If everything is ok, you will see the current version installed locally: 2. In the pom.xml file, you need to add the dependency “allure-testng” and two plugins “maven-surefire-plugin” and “allure-maven”, probably “maven-surefire-plugin” will already be in your list, then add the configuration, as shown below: </properties> <properties> < aspectj.version > 1.9.4 </ aspectj.version > < allure.version > 2.22.0 </ allure.version > < dependencies > < dependency > < groupId > io.qameta.allure </ groupId > < artifactId > allure-testng </ artifactId > < version > ${allure.version} </ version > </ dependency > </ dependencies > < build > < plugins > < plugin > < groupId > org.apache.maven.plugins </ groupId > < artifactId > maven-surefire-plugin </ artifactId > < version > 2.22.2 </ version > < configuration > < failIfNoTests > false </ failIfNoTests > < workingDirectory > ${project.build.directory} </ workingDirectory > < argLine > -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </ argLine > < systemPropertyVariables > < allure.results.directory > ${project.build.directory}/allure-results </ allure.results.directory > </ systemPropertyVariables > </ configuration > < dependencies > < dependency > < groupId > org.aspectj </ groupId > < artifactId > aspectjweaver </ artifactId > < version > ${aspectj.version} </ version > </ dependency > </ dependencies > </ plugin > < plugin > < groupId > io.qameta.allure </ groupId > < artifactId > allure-maven </ artifactId > < version > 2.10.0 </ version > < configuration > < reportVersion > ${allure.version} </ reportVersion > < allureDownloadUrl > https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/${allure.version}/allure-commandline-${allure.version}.zip </ allureDownloadUrl > </ configuration > </ plugin > </ plugins > </ build > After Reload Maven, the connected Allure plugin will appear in the list of plugins: 3. Now we can add Allure annotations to tests as shown in the screenshot: private TestContext context; @CitrusTest sendRequest(); getResponse();
    } $(http() .send() } .receive()
                        .response(HttpStatus.OK)
                        .message()
                        .type(MessageType.JSON) );
    } user.setData(data); user.setSupport(support); }
} @Story( "GET REQUEST" ) public class FirstTestGetUser extends TestNGCitrusSupport { @Test(description = "Getting information about user" ) public void getTestActions ( ) { this .context = citrus.getCitrusContext().createTestContext(); //        "Send GET request" //        "Get response" @Step( "Send GET request" ) public void sendRequest ( ) { .client( "restClientReqres" ) .get( "users/" + context.getVariable( "userId" ))); @Step( "Get response" ) public void getResponse ( ) { $(http().client( "restClientReqres" ) .body( new ObjectMappingPayloadBuilder(getJsonData(), "objectMapper" )) public User getJsonData ( ) { User user = new User(); Data data = new Data(); data.setId(Integer.valueOf(context.getVariable( "userId" ))); data.setEmail( "janet.weaver@reqres.in" ); data.setFirstName( "Janete" ); data.setLastName( "Weaver" ); data.setAvatar( "https://reqres.in/img/faces/2-image.jpg" ); Support support = new Support(); support.setUrl( "https://reqres.in/#support-heading" ); support.setText( "To keep ReqRes free, contributions towards server costs are appreciated!" ); return user; We added Allure annotations @Story for class and @Step for methods in the test﻿ above. 4. After passing the test, we can generate a report by calling the "allure:serve" command: 5. Browser will automatically launch with the report page: Thus, we connected Allure reports to our test project. If you are using the Junit engine in your test project, then you will need to follow the same steps, only you will need the “allure-junit4” or “allure-junit5” dependency. This is all. I wish you happy testing).

Let's Connect Allure Reports to a Test Project Made on Citrus Framework

In this tutorial, we fine-tune a model for topic classification using the Hugging Face Transformers and Datasets libraries. By the end of this tutorial, you will have a powerful fine-tuned model for classifying topics and published it to Hugging Face 🤗 for people to use. RoBERTa Prerequisites This article assumes you have a 🤗 account and working Python, NLP, and Deep learning knowledge. Hugging Face : A basic understanding of machine learning concepts, such as supervised learning, training, and evaluation. Understanding of machine learning concepts will help follow the code presented in the tutorial. Python programming experience : Some experience with deep learning and natural language processing (NLP) concepts, such as neural networks, word embeddings, and tokenization, will be beneficial for grasping the ideas presented in the tutorial. Familiarity with deep learning and natural language processing . Access to a Google Colab or Jupyter Notebook environment : To publish your fine-tuned model to the Hugging Face Hub, you will need a Hugging Face account. If you do not already have an account, you can sign up for one at . A Hugging Face account https://huggingface.co/join By meeting these prerequisites, you will be well-prepared to follow the tutorial and get the most out of it. Let's get our hands dirty 😁 We start by installing the dependencies. !pip install transformers datasets huggingface_hub tensorboard==2.11
!sudo apt-get install git-lfs --yes We then import the needed modules. import torch
from datasets import load_dataset
from transformers import (
    RobertaTokenizerFast,
    RobertaForSequenceClassification,
    TrainingArguments,
    Trainer,
    AutoConfig,
)
from huggingface_hub import HfFolder, notebook_login We need to log in to Hugging Face by using a token. notebook_login() Let's set some variables for easier configuration. model_id = "roberta-base"
dataset_id = "ag_news"
# relace the value with your model: ex <hugging-face-user>/<model-name>
repository_id = "achimoraites/roberta-base_ag_news" Preprossessing Next, we load our dataset and do some preprocessing. # Load dataset
dataset = load_dataset(dataset_id)

# Training and testing datasets
train_dataset = dataset['train']
test_dataset = dataset["test"].shard(num_shards=2, index=0)

# Validation dataset
val_dataset = dataset['test'].shard(num_shards=2, index=1)

# Preprocessing
tokenizer = RobertaTokenizerFast.from_pretrained(model_id)

# This function tokenizes the input text using the RoBERTa tokenizer. 
# It applies padding and truncation to ensure that all sequences have the same length (256 tokens).
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True, max_length=256)

train_dataset = train_dataset.map(tokenize, batched=True, batch_size=len(train_dataset))
val_dataset = val_dataset.map(tokenize, batched=True, batch_size=len(val_dataset))
test_dataset = test_dataset.map(tokenize, batched=True, batch_size=len(test_dataset)) Set The Dataset Format The function is used to specify the dataset format, making it compatible with PyTorch. set_format() The argument lists the columns that should be included in the formatted dataset. columns In this case, the columns are "input_ids", "attention_mask", and "label". By setting the format and specifying the relevant columns, we prepare the datasets for use with the Hugging Face Trainer class, which requires PyTorch tensors as input. # Set dataset format
train_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"])
val_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"])
test_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"]) To make our model easier to use, we will create an mapping. This will map the class ids to their labels. id2label This makes it easier to interpret the model’s output during inference. # We will need this to directly output the class names when using the pipeline without mapping the labels later.
# Extract the number of classes and their names
num_labels = dataset['train'].features['label'].num_classes
class_names = dataset["train"].features["label"].names
print(f"number of labels: {num_labels}")
print(f"the labels: {class_names}")

# Create an id2label mapping
id2label = {i: label for i, label in enumerate(class_names)}

# Update the model's configuration with the id2label mapping
config = AutoConfig.from_pretrained(model_id)
config.update({"id2label": id2label}) Training and Evaluation Now, we will set up our training parameters, Hugging Face 🤗 repository, and Tensorboard. # TrainingArguments
training_args = TrainingArguments(
    output_dir=repository_id,
    num_train_epochs=5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    evaluation_strategy="epoch",
    logging_dir=f"{repository_id}/logs",
    logging_strategy="steps",
    logging_steps=10,
    learning_rate=5e-5,
    weight_decay=0.01,
    warmup_steps=500,
    save_strategy="epoch",
    load_best_model_at_end=True,
    save_total_limit=2,
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repository_id,
    hub_token=HfFolder.get_token(),
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
) We can start the training process by running: # Fine-tune the model
trainer.train() Evaluate the model: # Evaluate the model
trainer.evaluate() Publishing We are ready to publish our model to Hugging Face 🤗 # Save our tokenizer and create a model card
tokenizer.save_pretrained(repository_id)
trainer.create_model_card()
# Push the results to the hub
trainer.push_to_hub() Test The Model At this point, our model should have been published and will be available for use. Let's test it! # TEST MODEL

from transformers import pipeline

classifier = pipeline('text-classification',repository_id)

text = "Kederis proclaims innocence Olympic champion Kostas Kederis today left hospital ahead of his date with IOC inquisitors claiming his innocence and vowing: quot;After the crucifixion comes the resurrection. quot; .."
result = classifier(text)

predicted_label = result[0]["label"]
print(f"Predicted label: {predicted_label}") Congratulations You have fine-tuned and published a RoBERTa model for text classification using Hugging Face 🤗 transformers and datasets libraries! **For reference, here is my fine-tuned model on Hugging Face 🤗
\ https://huggingface.co/achimoraites/roberta-base_ag_news **You can find the code here
\ https://github.com/achimoraites/machine-learning-playground/blob/main/NLP/Text classification/RoBERTa_Finetuning.ipynb Happy 🤖 learning 😀! Also published here Feature image: Photo by Vlada Karpovich from Pexels: https://www.pexels.com/photo/crop-young-businesswoman-using-laptop-while-drinking-tea-at-home-4050347/

Fine-Tuning RoBERTa for Topic Classification

After the success of and the rise of deep learning, tackling video was an obvious next step. Just like the classification of images, the task of video classification is the most straightforward start on the path to general video understanding models. As for the specific labels that are being classified, the computer vision research community has gravitated toward classifying human actions in videos. image classification dataset challenges One of the earliest human action recognition video datasets, even before deep learning took off, was from 2004. Action recognition datasets have come a long way since then, some focusing on clips from , while others . the KTH dataset Hollywood movies focusing on sports In 2017, DeepMind released one of the largest and most impactful human action recognition datasets yet, . As of the writing of this post, four versions of the Kinetics dataset have been released: , , , and . The version number indicates the number of action classes. Additionally, each version adds new videos to replace those that have been deleted from YouTube over time. Kinetics 400 600 700 700–2020 This integration includes a sophisticated way to download the dataset, as well as examples of how to evaluate and improve models trained on the dataset. This post walks through the integration of Kinetics into the open-source dataset curation and model analysis tool, . FiftyOne Downloading Kinetics is now as easy as: import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("kinetics-600") Setup To run the examples in this post, you need to : install FiftyOne pip install fiftyone You will also need to install which is used by FiftyOne to download videos from YouTube: Pytube pip install pytube Downloading Kinetics Until recently, the only way to access the Kinetics dataset was to download each video directly from their sources on YouTube. This resulted in including videos having been deleted, YouTube throttling downloads, and inefficiencies in clipping videos. numerous issues The Common Visual Data Foundation (CVDF) has collaborated with the Kinetics dataset maintainers to for the general public to download. It should be noted that the CVDF-hosted version does not include all samples present in the original dataset, only those that were available on YouTube at the time that the CVDF version was created. host all versions of the dataset on AWS The CVDF has made it much easier to gain access to the full dataset. However, you still need to handle the challenges of visualizing, wrangling, and subsetting the dataset to meet your needs. In some cases, you don’t want to have to download the entire dataset, to begin with. This is where the integration of comes in. With just one line of Python code, you can now specify the version, the split, and the classes that you want and then visualize it in the with just another line of code. Kinetics into the FiftyOne Dataset Zoo FiftyOne App import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "kinetics-700-2020",
    split="validation",
    classes=["grooming cat", "grooming dog"],
    max_samples=10,
)

session = fo.launch_app(dataset) Training and Evaluating a Model After having downloaded Kinetics, you can now start using it to train action recognition models. Since the dataset is already in FiftyOne, it is easy to use libraries like or to train a model directly on the dataset. PyTorch PyTorch Lightning Flash pip install lightning-flash lightning-flash[video] torchvision pytorchvideo import torch

from flash import Trainer
from flash.video import VideoClassificationData, VideoClassifier

import fiftyone as fo
import fiftyone.zoo as foz

classes = [
    "swimming backstroke",
    "swimming breast stroke",
    "swimming butterfly stroke",
    "swimming front crawl",
]

# Load Kinetics
dataset = foz.load_zoo_dataset(
    "kinetics-700-2020",
    splits=["train", "validation"],
    classes=classes,
    max_samples=50,
    shuffle=True,
)

# Replace spaces in class names with underscore
labels = dataset.distinct("ground_truth.label")
labels_map = {l: l.replace(" ", "_") for l in labels}
dataset = dataset.map_labels("ground_truth", labels_map).clone()

# Create views for dataset splits
train_view = dataset.match_tags("train")
val_view = dataset.match_tags("validation")

# Create the Flash Datamodule
datamodule = VideoClassificationData.from_fiftyone(
    train_dataset=train_view,
    val_dataset=val_view,
    predict_dataset=val_view,
    label_field="ground_truth",
    batch_size=1,
    clip_sampler="uniform",
    clip_duration=1,
    decode_audio=False,
)

# Build the model
model = VideoClassifier(
    backbone="x3d_xs",
    labels=datamodule.labels,
    pretrained=True,
)

trainer = Trainer(
    max_epochs=10,
    limit_train_batches=5,
    gpus=torch.cuda.device_count(),
)

# Finetune the model
trainer.finetune(model, datamodule=datamodule, strategy="freeze") After your model is trained, you can then generate predictions on the validation and test splits and use FiftyOne to of the model. evaluate the performance from itertools import chain

from flash.core.classification import FiftyOneLabelsOutput

def get_fo_label_preds(samples, datamodule, trainer):
    # Return a list of predictions in fo.Detection format
    predictions = trainer.predict(
        model,
        datamodule=datamodule,
        output=FiftyOneLabelsOutput(return_filepath=False, labels=datamodule.labels),
    )
    predictions = list(chain.from_iterable(predictions))  # flatten batches
    return predictions

predictions = get_fo_label_preds(val_view, datamodule, trainer)

# Add predictions to FiftyOne dataset
val_view.set_values(
    "predictions", predictions
)

session = fo.launch_app(val_view)

results = val_view.evaluate_classifications(
    "ground_truth",
    "predictions",
    eval_key="eval",
) The results of the evaluation can be used for things like plotting and . confusion matrices precision-recall curves pip install ipywidgets underscore_classes = [c.replace(" ", "_") for c in classes]

plot = results.plot_confusion_matrix(classes=underscore_classes)
plot.show() As you can see, since we only finetuned the model on a few dozen samples, it is overfitting to the backstroke and butterfly stroke classes. This implies that we should download additional samples of the other two classes and continue training. Analyzing the model to find the best and worst-performing samples can shed light on the best ways to improve your model’s performance. from fiftyone import ViewField as F

eval_view = val_view.filter_labels(
    "predictions", (F("confidence") > 0.6) & (F("eval") == False)
)

session.view = eval_view The following shows one of the top examples in this evaluation view of highly confident but incorrectly predicted samples. There are multiple issues that we can see with this sample. First, the footage is first-person which is rare in this dataset. If we want to predict on first-person videos, then more should be added to the training set. Second, there are examples of both breaststroke and backstroke in the video so it would be difficult to assign a label. Third, the ground truth label is front crawl which does not appear at all in the dataset. Using FiftyOne to get hands-on and analyze specific samples can lead to results like these highlighting ways that you can improve the dataset itself. Since Kinetics is a very large dataset, we could easily download additional videos to supplement problematic samples that we may want to exclude from training. Improvements to your dataset can lead to easier gains in model performance than working on improving the model architecture itself. Summary The makes it easier than ever to be able to download exactly the subset of Kinetics that you want or even the dataset in its entirety. Additionally, FiftyOne allows for in-depth evaluation and analysis of video models leading to better datasets and higher performing models. integration of the Kinetics dataset into FiftyOne -- (Originally posted by ) here

Kinetics Dataset - Training and Evaluating Models for Video Classification 

Photo by on Mick De Paola Unsplash The “Maybe just a quick one” series title is inspired by my most common reply to “Fancy a drink?”, which, may or may not end up in a long night. Likewise, these posts are intended to be short but I get carried away sometimes, so, apologies in advance. About 🤗 Transformers 🤗 Transformers ( ) is a collection of state-of-the-art NLU (Natural Language Understanding) and NLG (Natural Language Generation ) models.  They offer a wide variety of architectures to choose from (BERT, GPT-2, RoBERTa etc) as well as a of pre-trained models uploaded by users and organisations. Hugging Face transformers hub Fine-tuning a model One of the things that makes this library such a powerful tool is that we can use the models as a basis for tasks. In other words, they can be a starting point to apply some fine-tuning using our own data. The library is designed to easily work with both Tensorflow or PyTorch. transfer learning 🤗 Datasets is a  wrapper library that provides some tools to load and process data in many commonly used formats (CSV, JSON etc). It also makes sharing datasets and metrics for Natural Language Processing extremely easy. Hugging Face Datasets 🤗 Datasets originated from a fork of the awesome TensorFlow Datasets and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. Well, let’s write some code In this example, we will start with  a pre-trained model and fine-tune it on the dataset. We will then test it  on classifying tweets as hate speech, offensive language, or neither. All coding is done in . BERT (uncased) Hate Speech and Offensive Language Google Colab Please note:  this dataset contains text that can be considered racist, sexist, homophobic, or generally offensive. So let’s start by installing some necessary packages, import them and load the dataset. The dataset is stored in Google Drive, and the path to load it from is . So if you code along , please make sure you change the path to point to your own dataset file. /content/drive/MyDrive/Data/labeled_data.csv We are using the function to load it and then split it into train, validation, and test sets. The 3 sets are then gathered together to form a .This is a dictionary class, that offers us many methods to process the data. We will then remove some of the columns we don’t need for our classification task. load_dataset DatasetDict !pip install transformers
!pip install datasets datasets load_dataset,DatasetDict transformers AutoTokenizer,TFAutoModelForSequenceClassification tensorflow tf numpy np matplotlib.pyplot plt

The dataset located our Google Drive data folder.

DATA_PATH = dataset = load_dataset( , data_files=DATA_PATH,split= )
train_testvalid = dataset.train_test_split()
test_valid = train_testvalid[ ].train_test_split()
train_test_valid_dataset = DatasetDict({ : train_testvalid[ ], : test_valid[ ], : test_valid[ ]})
dataset = train_test_valid_dataset.remove_columns([ , , , , ]) #Install the necessary packages from import from import import as import as import as is in "/content/drive/MyDrive/Data/labeled_data.csv" 'csv' 'train' 'test' 'train' 'train' 'test' 'test' 'valid' 'train' 'hate_speech' 'offensive_language' 'neither' 'Unnamed: 0' 'count' So now we need to preprocess the data. The tool responsible for this is a What do tokenizers do? Very simply put, they split the data in tokens (these can be characters, words, part of words, depending on the model), and convert them into tensors of numeric ids, which is the form that the model can read. For this task, we are using the tokenizer from the pre-trained model we selected ( ). But let’s see how we achieve this: Tokenizer. bert-base-cased tokenizer = AutoTokenizer.from_pretrained( ) tokenizer(examples[ ], padding= , truncation= )

tokenized_datasets = dataset.map(tokenize_function, batched= )
train_dataset = tokenized_datasets[ ]
eval_dataset = tokenized_datasets[ ]
test_dataset = tokenized_datasets[ ]
tf_train_dataset = train_dataset.remove_columns([ ]).with_format( )
tf_eval_dataset = eval_dataset.remove_columns([ ]).with_format( )
tf_test_dataset = test_dataset.remove_columns([ ]).with_format( )

train_features = {x: tf_train_dataset[x].to_tensor() x tokenizer.model_input_names}
train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset[ ]))
train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch( )

eval_features = {x: tf_eval_dataset[x].to_tensor() x tokenizer.model_input_names}
eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset[ ]))
eval_tf_dataset = eval_tf_dataset.batch( )

test_features = {x: tf_test_dataset[x].to_tensor() x tokenizer.model_input_names}
test_tf_dataset = tf.data.Dataset.from_tensor_slices((test_features, tf_test_dataset[ ]))
test_tf_dataset =test_tf_dataset.batch( ) "bert-base-cased" : def tokenize_function (examples) return "tweet" "max_length" True True "train" "valid" 'test' "tweet" "tensorflow" "tweet" "tensorflow" "tweet" "tensorflow" for in "class" 8 for in "class" 8 for in "class" 8 Notice how we used the dataset function, to apply our user-defined to all the elements of the dataset. map tokenize_function After applying the tokenization, we created 3 to feed the model with. We are now ready to train the model. Again, we are using the selected pre-trained model, transferring the “knowledge” it already has, but replacing its head with one that is suited to our task. We are using the class, which represents a generic Tensorflow (hence the TF prefix) model, with a sequence classification head. Also, notice the num_labels parameter which is set to 3, as this is a multi-class task with 3 distinct labels. After the training is finished, we plot the Sparse Categorical Accuracy and the Loss of both the train and the validation dataset. Tensorflow Datasets TFAutoModelForSequenceClassification model = TFAutoModelForSequenceClassification.from_pretrained( , num_labels= )
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate= ),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits= ),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)

history = model.fit(train_tf_dataset, validation_data=eval_tf_dataset, epochs= )

plt.plot(history.history[ ])
plt.plot(history.history[ ])
plt.title( )
plt.ylabel( )
plt.xlabel( )
plt.legend([ , ], loc= )
plt.show()
plt.plot(history.history[ ])
plt.plot(history.history[ ])
plt.title( )
plt.ylabel( )
plt.xlabel( )
plt.legend([ , ], loc= )
plt.show() "bert-base-cased" 3 5e-5 True 2 'sparse_categorical_accuracy' 'val_sparse_categorical_accuracy' 'model sparse categorical accuracy' 'accuracy' 'epoch' 'train' 'val' 'upper left' 'loss' 'val_loss' 'model loss' 'loss' 'epoch' 'train' 'val' 'upper left' We can now evaluate the model on the test dataset we created earlier: test_loss, test_acc = model.evaluate(test_tf_dataset,verbose= )
print( , test_acc) / - s - loss: - sparse_categorical_accuracy: Test accuracy: model.save_pretrained( ) 2 '\nTest accuracy:' 194 194 62 0.2596 0.9135 0.9134925603866577 "/content/drive/MyDrive/Data/hate-speech-bert" Notice that we save the model with the function offered by Transformers. This action generates a directory with two files by default: a .json file that contains the model configuration and a .h5 file with the model weights. We can also push the model to the Hugging Face Models Hub should we want to, in order to make it available to the public. save_pretrained Does it work though? Let’s see how our model does in classifying some unseen text. I will use some stereotypical racist/offensive/sexist texts posted on social media. : Warning Due to the nature of this task, the language used here can be racist, sexist, and offensive. However, this is the only way to evaluate the model’s ability. pred2label = { : , : , : }
preds = model(tokenizer([ , , ],return_tensors= ,padding=True,truncation=True))[ ]
print(preds)
class_preds = np.argmax(preds, axis= ) pred class_preds:
  print(pred2label[pred]) 0 'Hate Speech' 1 'Offensive Language' 2 'Neither' "Jews are useless , I don't see why they even exist" "Gay people suck" "Women are dressed up like whores these days" "tf" 'logits' 1 for in tf.Tensor(
[[ ]
 [ ]
 [ ]], shape=( , ), dtype=float32)
Hate Speech
Hate Speech
Offensive Language 0.37532297 0.14053927 -0.8647832 0.04699412 -0.17951615 -0.3738104 0.2524849 2.586107 -2.8212454 3 3 Looks like our model managed to correctly (well that’s subjective, but generally speaking these look correct) classify the texts. Further reading: If the above seems interesting to you, there is a lot more that this library can do. I would start by checking their documentation which is quite extensive as well as some quick video courses they provide: 🤗 Transformers Model Hub 🤗 Datasets (with Youtube Videos, we always like them ) Crash course Fine-tuning with custom datasets Happy coding.

How to Fine Tune a 🤗 (Hugging Face) Transformer Model

Yes, building AI is hard! Every step from data annotation, training and deployment comes with its own set of challenges. This blog post will try to deal with the first two of those: Data Annotation Training Over the past few years at , we have been developing highly efficient AI systems deployed in millions of consumer devices as of today! And in this process, we developed a lot of tools to simplify our workflow. We have finally decided to bridge the AI skills gap by making some of our tools public. One of the tools, Brain Builder deals with the first of the aforementioned issues: Data Annotation. Neurala is an AI-assisted annotation tool that fits seamlessly into commonly used frameworks like and Caffe. This post will walk you through the steps you’d need to incorporate Brain Builder into your AI workflow. We will build a semantic segmentation network using the data tagged by Brain Builder and model written using TensorFlow and Keras. Brain Builder TensorFlow Semantic segmentation is one of the commonly encountered applications of computer vision. It is the task of labeling every pixel of an image with a . It is used extensively for image analysis and enables applications like portrait mode on smartphones. The following tutorial will describe how you can create a semantic segmentation model to segment people using your own data and train it over multiple GPUs, using Brain Builder and TensorFlow. class Data Annotation using Brain Builder Data annotation is the most crucial step in building an AI system since this is what your model learns from. And a deep learning model is only as good as the data it’s fed! Brain Builder helps you in curating a quality dataset and has AI-assisted tools to speed up the data curation step. One of the most important features of Brain Builder is automated video tagging. AI-assisted video tagging lets you collect a lot of data in a small amount of time by extracting and tagging frames from videos recorded at very high frame rates. The following video shows the process of tagging people in videos using brain builder: The folder contains everything you will need in order for TensorFlow to be able to read and process your dataset. All the ground truth images are , where every pixel in the image corresponds to a class index. The files provide you details about the mapping of class index to their subsequent R, G, B values that you can later use for visualizing your results. class index-labeled pngs class_dictionary.csv, classes.txt and palette.txt We will now write a popular architecture for semantic segmentation called using TensorFlow and Keras. Using Estimators and the TensorFlow Dataset API, we’d enable training over multiple GPUs to reduce the training times massively! UNet (Please note that we’ve modified the UNet slightly in our implementation to make this a quicker experiment!) Multi-GPU training using TensorFlow Estimators and Dataset API End-to-end integration of Keras with TensorFlow has made it easy to enable multi-GPU training of Keras models using TensorFlow Estimators and Dataset API. TensorFlow Estimators The class represents a model, as well as how this model should be trained and evaluated. Some of the key advantages of using Estimators are: Estimator Estimator based models can be run across multiple GPUs without changing the model code; Estimators simplify sharing the implementation of models across developers; Estimators build the graph for you and eliminate the need of an explicit (and rather painful) session. The following piece of code creates a Keras model, compiles it and converts it to an TensorFlow Estimator: dataset = TFDatasetLoader(data_dirpath, input_size, n_classes, batch_size, num_epochs)model = UNet(input_size, n_classes)model.compile(optimizer=tf.train.AdamOptimizer(learning_rate=0.01), loss='binary_crossentropy', metrics=['accuracy']) strategy = tf.contrib.distribute.MirroredStrategy(num_gpus=num_gpus)config = tf.estimator.RunConfig(train_distribute=strategy)estimator = tf.keras.estimator.model_to_estimator(model, config=config) If you’re familiar with Keras, the only new things you’d notice are the last three lines of code. The strategy we use for multi-GPU training is called . In this strategy, each GPU has a copy of the graph and gets a subset of the data on which it computes the local gradients. Once the local gradients are computed, each GPU then waits for other GPUs to finish in a synchronous manner. When all the gradients have arrived each GPU averages them and updates its parameter and the next step begins. You can learn more about the Distributed TensorFlow training on link. MirroredStrategy this Now that the strategy has been defined, we create a object using that strategy and use it to call the function which converts the Keras model into an Estimator object. RunConfig model_to_estimator With the estimator ready, we can simply train and evaluate it using these two lines of code: estimator.train(lambda:dataset.imgs_input_fn(mode="train"), hooks=[time_hist])estimator.evaluate(lambda:dataset.imgs_input_fn(mode="val")) The important thing to note in these two calls is the function. This function feeds the model the data using the TensorFlow Dataset API. dataset.imgs_input_fn TensorFlow Dataset API Dataset is a highly efficient API for data input that ties nicely with the Estimators and tf.Keras. The core of the new input pipeline is the (and maybe the ). A is a collection of , each with the same structure, where one element can be one or more tensors. The different tensors inside an element are called . Each component has a certain data type and shape, but different components inside one element can have different data types and shapes. Dataset Iterator Dataset elements components The following code snippet shows how to create a dataset object that can be fed to the Estimators: dataset = tf.data.Dataset.from_tensor_slices((image_paths, gt_paths))dataset = dataset.map(lambda image, label: tuple(tf.py_func(self._one_hot_gt, [image, label], [tf.string, tf.uint8])))dataset = dataset.map(self._data_to_tensor)dataset = dataset.repeat(self.epochs)dataset = dataset.batch(self.batch_size)dataset = dataset.prefetch(self.batch_size) The important function to take a note of is the function that lets you read the data and process it any way you want (resize, one-hot code, etc). lets you specify how many times you want to iterate over this dataset. automatically batches the data based on the batch size provided by the user and prefetches the next batch while the current batch is being processed. map() repeat() batch() prefetch() Training Results and Speed Comparison We train our segmentation model for 100 epochs over two NVIDIA GTX1080 GPUs. The model attains about 91% pixel accuracy with just about 4 hours of training. Here are some comparisons from our tests: As you can see, using two GPUs instead of one gives us a speedup of almost 2x to attain the same level of accuracy. We hope this tutorial helped you understand how you can integrate Brain Builder into your AI workflow to build highly efficient data preparation and training pipelines. We welcome the feedback from the community in order to be able to improve Brain Builder further! . Leave a comment below with feedback, feature requests or any further questions you might have related to the tutorial and/or Brain Builder. You can sign up for Brain Builder Beta here You can find the code for this tutorial on . this link

Multi-GPU training with Brain Builder and TensorFlow

That’s a short warning to all Tensorflow users working with visual content. Short notice: don’t use any functions! tf.image.resize I was rewriting codebase of our — Let’s Enhance to make it ready for bigger and faster models and API we are working on. As we work with image generation (superresolution, deblurring, etc) we do rely on a typical image processing libraries like OpenCV and PIL. I always had suspicions that it makes sense to use Tensorflow image processing capabilities — in theory, they should be faster. So, I decided to stick to native Tensorflow image preprosessing and dataset building tools using dataset.map() to keep all operations in tensors all around my code. neural network image upscaling service The problem was quite awful — my new and shiny code for superresolution wasn’t able to reproduce not only any state-of-art networks but even my own code that I wrote 4 months ago. And the ugliest part was that results of superresolution itself were pretty good sometimes, the network was working, although not reaching target PSNR and having strange visual artifacts sometimes, like doubling of small lines. Let the debug begin What initially was looking as minor bug became 60 days of struggle and sleepless nights. My faulty logic was simple — there’s something wrong with network definition or training process. Data preprocessing is definitely fine, as I am having meaningful results and visual control over image processing in Tensorboard. I tweaked everything I was able to find, defined network using Keras, Slim, raw TF — nothing, looked for changes in TF 1.3->1.4->1.5 and different CUDA versions, paddings behaviors. I am ashamed even to tell you about my latest suspicions, which involved defects in GPU RAM and statics. I was tweaking perceptual losses and style losses looking for a reason. And each iteration took days to retrain until some meaningful result… Yesterday I found when looking into Tensorboard. It was almost subliminal feeling that something is wrong with the image. I disregarded network output and just overlayed target image and input image(that is a downscaled target image) in Photoshop. Here’s what I got. The Bug, Looks strange, somekind of displacement happening here. Totally againts any logic, this just can’t be true! My code is dead simple. Read image, crop image, resize image. All in Tensorflow. Anyway, RTFM. tf.image.resize_bicubic has a parameter — “align corners”. How in the world would you like to downscale image and not have corners aligned? You can! So there is a very weird behavior of this function known for a long time — . They can’t fix it as this can break lots of old code and pre-trained networks. read this thread Our function isn't even reflection equivariant. It would be lovely to fix this, but I'd be worried about breaking old models. tf.image.resize_area This code does actually displace your image by one pixel to the left and top. Thread suggests that even the interpolation is broken in TensorFlow. It’s 2018, people. Here are the actual downscaling results with TF. Stick to Scipy/OpenCV/numpy/PIL whatever you prefer for image processing. The second I changed it my network worked like a charm (next day actually, when I saw training results).

How Tensorflow’s tf.image.resize stole 60 days of my life

A look at two databases that have made claims to the Kubernetes native label: TiDB and DataStax Astra DB. The cloud computing revolution has inspired and benefitted from multiple interrelated trends. The availability of self-service, public cloud infrastructure has helped to drive the adoption of microservice architectures and DevOps practices, including automation and observability. The drive toward containerization and container orchestration has led to the widespread adoption of Kubernetes as an environment for managing cloud-native applications. But one of the lagging areas in this revolution has been data and data infrastructure. For too long, data has been something that has lived outside of Kubernetes, leading to a lot of extra effort and complexity for developers in deploying cloud-native applications. One oft-repeated axiom in the early years of Kubernetes was that it was not yet ready for stateful workloads. Thankfully, a major shift has been quietly underway and has reached a point of maturity. The transformation happened slowly initially, beginning with efforts to containerize existing databases. This worked relatively well in small databases that ran on a single compute node, or databases that had been designed in a cloud-native world, like Apache Cassandra and DynamoDB, but challenges remained. Over the past two to three years, a new generation of databases has emerged. These “Kubernetes native” databases have been designed from the ground up to run on this open-source orchestration system. Here, we’ll define the qualities that make a database Kubernetes native and the benefits of adopting a Kubernetes native database. To do that, we’ll look at two databases claiming the Kubernetes native label: TiDB and DataStax Astra DB. Kubernetes Native MySQL with TiDB First, let’s examine a database with a relational emphasis: (short for Titanium Database). TiDB is an open-source system built by PingCAP that provides a and a columnar database to support hybrid transactional and analytic processing (known as HTAP, for short). TiDB MySQL-compatible database As shown in Figure 1 below, TiDB has a microservice design. The TiDB query layer, TiKV MySQL databases, TiFlash columnar databases, Spark nodes, and metadata management are each deployed as scalable microservices in their clusters. This design separates compute-intensive work from storage-intensive work, as the query and database layers are independently scalable. One critical commitment the TiDB creators made was that the database only runs on Kubernetes. Is that enough to make it Kubernetes native? Let’s dig a bit deeper. First, TiDB is deployed and managed by a Kubernetes operator using (CRDs). The TiDB CRDs include the TiDBCluster, which enables you to specify the scaling and configuration of each microservice and how the database layer components use storage through Kubernetes Persistent Volumes. Additional CRDs are used to deploy monitoring tools and manage operational tasks like backup and restore. custom resources that interfaces with the default K8s scheduler to make more application-aware scheduling decisions. This emphasis on using existing Kubernetes capabilities where available is the mark of a Kubernetes native database. TiDB also has an optional scheduler extension Kubernetes Native Cassandra with DataStax Astra DB Now, look at another Kubernetes native database and note some similarities and differences. that was one of the first to claim to be cloud native, but what does it look like to deploy Cassandra in Kubernetes? is a highly scalable NoSQL database Cassandra that has been factored into microservices, as shown in Figure 2. is a version of Cassandra DataStax Astra DB Like TiDB, the database includes microservices concerned with query processing and data storage, as well as services for identity and access control, data repair, and backup/restore. The data services are particularly interesting in their use of , with Kubernetes Persistent Volumes used only for caching and object storage used for longer-term persistence. Separating compaction into its service enables this compute-intensive processing to happen in the background without affecting the performance of data services serving read and write traffic. storage Astra DB is offered as a managed service available in multiple cloud regions. Each region contains a data plane consisting of the services mentioned above, managed by a Kubernetes operator, as well as infrastructure services, including the stack for observability and for metadata management. Kube-Promethus etcd The data planes are managed by a control plane that can run in one or more clouds to manage customer accounts and databases and provision Kubernetes clusters in new regions. in which multiple user databases can share the same microservices and supporting infrastructure, lowering unit economics for smaller-scale users. One novel aspect of Astra DB is its multi-tenant architecture As users grow their applications, they can move to dedicated resources to achieve optimal performance at scale, all on a “ ” basis. pay-as-you-go Kubernetes Native Database Principles Based on our observations of TiDB and Astra DB, we can derive some ideas of what makes a database Kubernetes native. Many of these correspond to a list of principles for cloud-native data, which I described in an : earlier article First, a database broken into constituent microservices enables each service to be scaled independently. Some types of compute-intensive processing may even be scaled to zero for a true serverless solution, especially when combined with a multitenant design. Composable microservice architecture: Microservices composed of a Kubernetes native database should make maximum usage of Kubernetes APIs for managing the fundamental resources of cloud-native applications: compute resources such as , Kubernetes ingress and services for exposing network access to data and more. This includes leveraging capabilities already present in Kubernetes, such as etcd for metadata management, instead of bringing along components with duplicative functionality. Treat compute, network, and storage as commodities: StatefulSets and deployments for managing workloads, the Persistent Volume subsystem for storage Following common patterns for Kubernetes applications will yield multiple operational benefits, for example, . By default, Kubernetes itself sets a great example that databases should follow for how to be secure: using Kubernetes Secrets to distribute security credentials, only exposing ports as needed, and so on. Leverage Kubernetes best practices: exposing liveness and readiness checks on each microservice to help availability and exposing metrics via the Prometheus PromQL API for observability A Kubernetes native database should embody the Kubernetes principles of declarative management via operators and custom resources, rather than relying on legacy database management UIs and CLIs. When necessary, . The goal is a clean separation of data plane functionality (managing data) from control plane functionality (managing the database). Declarative management via operators: Kubernetes extension points, such as scheduler extensions, can be used to add application-specific behavior Databases and other data infrastructures that faithfully adopt these principles will yield benefits, including a , lower operational complexity resulting in faster time to market, and standards-compliant solutions meeting today’s high availability and security demands. high performance for optimal cost at all scales The Future of Kubernetes Native Data Infrastructure Much progress is still to be made, and it’s not limited to databases alone. Kubernetes native principles can be applied to other types of data infrastructure, including streaming, analytics, and machine learning. Kubernetes native solutions will continue to make strides in multicluster and multi-cloud deployments to scale globally and will adopt multitenancy and serverless principles for better cost optimization. Kubernetes itself has room for improvement in adding more flexibility to StatefulSets and support for multicluster federation. The key to continued progress is open collaboration. The is a highly active group of data geeks bringing together builders of data-intensive applications and the infrastructure that supports them. Data on Kubernetes Community Join us to talk about ideas like developing reusable operators that can manage multiple databases or defining a common set of CRDs for concepts like backup/restore and data loading. Together we’ll continue to push the horizon of cloud computing for the benefit of all. Learn more about Kassandra native databases and more at the digital summit on March 14, 2023. Cassandra Forward This article is based on Chapter 7, “The Kubernetes Native Database,” from the O’Reilly book “ Managing Cloud Native Data on Kubernetes ” by Jeff Carpenter and Patrick McFadin. [ By Jeff Carpenter, DataStax Jeff Carpenter has worked as a software engineer and architect in multiple industries and as a developer advocate at DataStax, helping engineers succeed with Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems, including Stargate and K8ssandra. He is co-author of the O’Reilly books "Cassandra: The Definitive Guide" and "Managing Cloud Native Data on Kubernetes."

Kubernetes-Native Database: TiDB Vs. DataStax Astra DB

Base de datos nativa de Kubernetes: TiDB vs. Base de datos DataStax Astra

Base de données native Kubernetes : TiDB Vs. DataStax Astra DB

कुबेरनेट्स-नेटिव डेटाबेस: TiDB बनाम। डेटास्टैक्स एस्ट्रा डीबी

Kubernetes ネイティブ データベース: TiDB Vs. DataStax アストラ DB

Banco de dados nativo do Kubernetes: TiDB vs. Data Stax Astra DB

Cơ sở dữ liệu gốc Kubernetes: TiDB Vs. DataStax Astra DB

Kubernetes 原生数据库：TiDB 对比。 DataStax Astra 数据库

is a big data and AI start-up that uses geospatial data to monitor environmental parameters. Our goal is to become the . Blue Sky Analytics Bloomberg of Environmental data for environmental monitoring, ESG (environment, social, and governance) due-diligence and climate risk assessment In other words, we aim to be the go-to source for all environmental data to drive sustainable decision making. We believe that safeguarding the global market against climate risk and other climate-change-induced threats requires temporally and spatially continuous high-resolution datasets that inform us of pollution levels, water quality, emissions, fires, changes in soil composition, etc. We asked ourselves, “what if there was a platform that presented environmental data the way we have access to financial data?” Sounds like an impossible dream, right? That’s why we need a technology stack that’s built to power our mission and a team too young to not know what’s impossible (our average age is 24.8). …and, our air quality and farm fire crisis solutions have won us (Healthy Cities Solutions) and the , so there’s power in believing in and pursuing what others may deem impossible. MIT Solve ‘Space Oscar’ from European Space Agency In this post, we’ll outline our technology — which handles historical as well as continuous data — and talk a bit about we’ve architected our platform to give spatial-temporal context to pre-existing data, what we’ve learned along the way, and what’s next. In short, you’ll get a glimpse of how we are making sense of terabytes of raw data and turning them into environmental insights. Our goal? You’re inspired to follow your dreams and build your own “impossible” projects. how What we do (and why we do it) We started this journey by creating a high-resolution database for monitoring air pollution in India, called , and we aim to cover surface water monitoring in 2020, land composition and urban informatics in 2021, and other environmental parameters in the coming years. BreeZo Currently, despite the alarming air pollution levels in India, monitoring is inadequate. There are , which are unequally distributed, and, as a result, large parts of the country go unmonitored. around 200 government monitors across India Our products aim to bridge this data gap by using satellite data and public monitors to provide a more comprehensive picture of the air pollution crisis.We make our data available to customers via accessible APIs and platforms; our approach centers on ensuring public access (democratising high-resolution environmental information) and facilitating policy-making and enterprise decisions (allowing corporations and government entities to understand current environmental situations as they craft initiatives). But, what we can’t monitor, we can’t solve. Climate change is a global challenge that we must come together to fix; it poses a huge risk and will have grave effects on public health, ecology and the financial system. Thus, it’s vital for governments, industries, and scientists to collaborate and come up with innovative solutions — leveraging AI, big data, and space technology — to prevent and mitigate negative outcomes. That’s where we come in: we create a platform that brings the much-needed high-quality data individuals need to make data-driven sustainable decision making, such as ways to transition from carbon-intensive economies to low-carbon approaches. The Tech Stack: How we do it and why our time-series database matters to us In order to create a large-scale environmental monitoring platform from geospatial data, we needed a database (specifically a time-series database (TSDB)) that could handle enormous quantities of spatial time-series data. We looked at various options, and landed on . Timescale DB The most common way to solve our “problem” is to use NoSQL databases, which can be treated as streams. However, that would mean that we would need to write our logic to implement the spatial and temporal context into our data. As we monitor environmental parameters using satellite data, the ability to handle and fill null values is especially critical. We had a lot of hacked together ways to query data with null filling and deal with data irregularities, using multiple cron jobs for each source and code at the application layer to do spatial queries. Essentially, we were spending a lot of time managing our time-series data and not enough time analysing it. For example: we wrote one script for null-filling, which was very slow to begin with — and we had to write it in bash, separate from our main app, to ensure it was performant enough and didn’t stop the main thread. So, NoSQL databases weren’t enough. But, what could we use? We looked at options like Amazon Timestream, but found that, while it works for IoT data, it didn’t work for our scenario: we handle not just IoT, but satellite data as well. For us, TimescaleDB proved to be particularly unique among TSDBs, allowing us to power spatial queries on time-series data and having its roots in Postgres. Some important features for our adoption were continuous aggregation, gap-filling null values, time bucketing, and, most importantly SQL support. Timescale has brought a change in our development paradigm, as it serializes everything in a timely order, and we can store, query, and analyze our temporal and spatial data using SQL (directly from the database). Example We have a database table where we store all the pixels at 10 kilometer grid for all of India, and we run spatial queries for individual states (taking into account cloud cover, which act as NULL values, and averaging them over that time). India is 3.3. million square kilometers, which comes out to be 33,000 rows of daily emission (with a 10 kilometer grid). Now, we can run a spatial join and average it over time. 3.3*1000000/10*10 = 33,000 Measurements (Raw satellite data), from multiple satellite sources downsampled to common denominator -- Measurements table
CREATE TABLE measurements (
	u_wind NUMERIC( , ),
	v_wind NUMERIC( , ),
	albedo NUMERIC( , ),
	aod469 NUMERIC( , ),
	aod550 NUMERIC( , ),
	aod670 NUMERIC( , ),
	aod865 NUMERIC( , ),
	aod1240 NUMERIC( , ),
	aod_s5p NUMERIC( , ),
	blh NUMERIC( , ),
	temperature NUMERIC( , ),
	recorded_at DATE,
	grid GEOMETRY
); 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 Maps of districts, states & region -- Enum shape types 
CREATE TYPE shapes_type AS ENUM ( , , , ); 
-- Create shape table 
CREATE TABLE shapes ( id UUID NO-T NULL, name VARCHAR( ), TYPE shapes_type, shape GEOMETRY ); for 'Country' 'State' 'District' 'Region' 255 Sample problem or analysis So, how would someone actually apply or use this information? Let’s say we’re a District Magistrate who wants to set weekly KPIs for emissions in an area where farmers burn crops on a daily basis. To get a baseline to inform our decision, we’d want to know the weekly emissions average emissions in each district over the last 3 months, focusing on those in our state. This is a pretty complex (and powerful) geospatial query, but it’s just one example of the type of analysis we do on a daily basis. Crunching 33,000, 3 30 data points SELECT
	name,
	-- Aggregates all the points one district
	json_agg(json_build_object( , datetime, , u_wind, , v_wind, , albedo, , aod469, , aod550, ,
 aod670, , aod865, , aod1240, , aod_s5p, ,
 temperature, , blh)) AS pollutants
FROM (
	-- Selects all the districts Punjab
	SELECT
		name,
		shape
	FROM WHERE
		TYPE = -- Selects all the state
		AND ST_WITHIN(shape, (
				SELECT
					shape FROM WHERE
					name = ))) AS Districts
	LEFT JOIN (
		-- Select all the points last week daily average and within Punjab
		SELECT
			time_bucket_gapfill ( , recorded_at, NOW() - interval ,
 NOW()) AS datetime, grid, avg(u_wind) AS u_wind, avg(v_wind) AS v_wind,
 avg(albedo) AS albedo, avg(aod469) AS aod469, avg(aod550) AS aod550,
 avg(aod670) AS aod670, avg(aod865) AS aod865, avg(aod1240) AS aod1240, 
avg(aod_s5p) AS aod_s5p, avg(temperature) AS temperature, avg(blh) AS blh
		FROM WHERE
			-- Get the points within week
			recorded_at < NOW()
			AND recorded_at > NOW() - interval -- Get the point only Punjab
			AND ST_WITHIN(grid, (
					SELECT
						shape FROM WHERE
						name = ))
		GROUP BY
			grid, datetime) AS Records
		-- Join the points based on geometry
		ON ST_Within(grid, Districts.shape)
		-- group them together
	GROUP BY
		name in 'datetime' 'u_wind' 'v_wind' 'albedo' 'aod469' 'aod550' 'aod670' 'aod865' 'aod1240' 'aod_s5p' 'temperature' 'blh' in "shapes" 'District' `Districts` in `Punjab` "shapes" 'Punjab' in with '1 day' '1 week' "measurements" 1 '1 week' in "shapes" 'Punjab' finally As mentioned previously, we collect an enormous amount of data from various sources, namely 1000+ ground monitors across India and satellite missions, like . To power our platform, we use TimescaleDB built-in functions to build chunks that allow us to store historical data in an easily accessible, scalable way. For example, we might have 12 values for March 1, 2014, but we only need for that day for historical analysis. Sentinel 5P one value When we started: We had 100s of rasters for all the satellite images we were ingesting. As a result, we wrote a custom Python script for every analysis, which was time-consuming and tedious. Post the TimescaleDB transition: We store all the ingested data in the database and simply extract raster values. We use geospatial JOINs, utilising TimescaleDB and PostGIS to combine geospatial data and time-series data and perform time-slicing. This allows us to gain insights and store them, make complex spatial-temporal queries, and carry out deeper analysis. Simply put, no more reading 100s of raster files or writing custom scripts for each analysis; we just dump into our database, and use SQL to analyze and find solutions. Compression of data and storage complexities are areas we no longer need to stress about. Moreover, we don’t have to worry about storing data in a time-series fashion, or optimization on that time-series data. And, as mentioned earlier, we’re able to efficiently handle null filling, data aggregation, and time slicing, making our development process significantly less complicated. All of this is to say that, if you’re building a data-intensive solution like we are or combining various types of data, TimescaleDB is a great place to start working with time-series data. It’s helped us expedite our mission of commanding the environmental data space, and the possibilities are endless: you can pace it according to your needs, as well as infinitely scale as you grow. Previously published at https://medium.com/blue-sky-thinking/global-environmental-data-stack-for-climate-action-a5c3ed494091

Building a Global Environmental Datastack for Climate Action

3 Quick Ways to Optimize RecyclerView

RecyclerView অপ্টিমাইজ করার 3টি দ্রুত উপায়

3 schnelle Möglichkeiten zur Optimierung von RecyclerView

Tres formas rápidas de optimizar RecyclerView

3 façons rapides d'optimiser RecyclerView

रीसाइक्लरव्यू को अनुकूलित करने के 3 त्वरित तरीके

RecyclerView を最適化する 3 つの簡単な方法

RecyclerView를 최적화하는 3가지 빠른 방법

3 maneiras rápidas de otimizar o RecyclerView

3 быстрых способа оптимизировать RecyclerView

RecyclerView'ı Optimize Etmenin 3 Hızlı Yolu

3 cách nhanh chóng để tối ưu hóa RecyclerView

优化 RecyclerView 的 3 种快速方法

How to Create a Steel Base Plate Design Web App Using Python and Viktor

পাইথন এবং ভিক্টর ব্যবহার করে কীভাবে একটি স্টিল বেস প্লেট ডিজাইন ওয়েব অ্যাপ তৈরি করবেন

So erstellen Sie mit Python und Viktor eine Web-App zum Entwerfen von Stahlgrundplatten

Cómo crear una aplicación web de diseño de placa base de acero usando Python y Viktor

Comment créer une application Web de conception de plaques de base en acier à l'aide de Python et Viktor

पायथन और विक्टर का उपयोग करके स्टील बेस प्लेट डिज़ाइन वेब ऐप कैसे बनाएं

Python と Viktor を使用してスチールベースプレート設計 Web アプリを作成する方法

Python과 Viktor를 사용하여 강철 베이스 플레이트 디자인 웹 앱을 만드는 방법

Como criar um aplicativo Web de design de placa de base de aço usando Python e Viktor

Как создать веб-приложение для проектирования стальной опорной пластины с помощью Python и Viktor

Python ve Viktor'u Kullanarak Çelik Taban Plaka Tasarımı Web Uygulaması Nasıl Oluşturulur

Cách tạo ứng dụng web thiết kế tấm đế thép bằng Python và Viktor

如何使用 Python 和 Viktor 创建钢底板设计 Web 应用程序

Step 1: Install the necessary libraries To connect to a database and query data, you need to install the following two libraries: pandas: A Python library for data analysis, including functions for reading and writing data.
sqlalchemy: A Python library for interacting with relational databases, allowing you to use Python with various databases. You can install these libraries using the following command in the command prompt or terminal: pip install pandas
pip install sqlalchemy Step 2: Connect to the database Connecting to a database requires the following information: Database Type: The type of database you want to connect to, such as MySQL, PostgreSQL, etc.
Host Name: The host name or IP address where the database is located.
Port Number: The port number of the database, usually the default port number.
Username: The username required to connect to the database.
Password: The password required to connect to the database.
Database Name: The name of the database you want to connect to. You can use the following Python code to connect to the database: from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') Step 3: Query data using the read_sql function Use the read_sql function from pandas to query data from the database. The read_sql function requires two parameters: SQL Query: The SQL query you want to execute.
Database Connection: The database connection you created earlier. Here is an example of querying data: import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X(@Luca_DataTeam). Happy exploring!👋 Step 1: Install the necessary libraries To connect to a database and query data, you need to install the following two libraries: pandas: A Python library for data analysis, including functions for reading and writing data. sqlalchemy: A Python library for interacting with relational databases, allowing you to use Python with various databases. pandas : A Python library for data analysis, including functions for reading and writing data. pandas sqlalchemy : A Python library for interacting with relational databases, allowing you to use Python with various databases. sqlalchemy You can install these libraries using the following command in the command prompt or terminal: pip install pandas
pip install sqlalchemy pip install pandas
pip install sqlalchemy Step 2: Connect to the database Connecting to a database requires the following information: Database Type: The type of database you want to connect to, such as MySQL, PostgreSQL, etc. Host Name: The host name or IP address where the database is located. Port Number: The port number of the database, usually the default port number. Username: The username required to connect to the database. Password: The password required to connect to the database. Database Name: The name of the database you want to connect to. Database Type : The type of database you want to connect to, such as MySQL, PostgreSQL, etc. Database Type Host Name : The host name or IP address where the database is located. Host Name Port Number : The port number of the database, usually the default port number. Port Number Username : The username required to connect to the database. Username Password : The password required to connect to the database. Password Database Name : The name of the database you want to connect to. Database Name You can use the following Python code to connect to the database: from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') Step 3: Query data using the read_sql function Use the read_sql function from pandas to query data from the database. The read_sql function requires two parameters: SQL Query: The SQL query you want to execute. Database Connection: The database connection you created earlier. SQL Query : The SQL query you want to execute. SQL Query Database Connection : The database connection you created earlier. Database Connection Here is an example of querying data: import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X (@Luca_DataTeam). Happy exploring!👋 LinkedIn X

How to Connect to Oracle, MySql and PostgreSQL Databases Using Python

Virtual Meetings have become an essential part of corporate collaboration, serving as the de facto way that ideas are exchanged, decisions are made, and strategies are devised. However, the true value of meetings is often diluted by their length and the complexity of the discussions. This is why note-taking and effective summaries and action points are so important. Converting long and often pointless discussions into short (actionable) summaries, meeting attendees can quickly catch up on meetings they missed, review decisions that were made, and understand discussions without having to sift through hours and hours of recordings. We thought it would be cool to create our summarizer. If you don't have one already, keep reading. This technical tutorial will guide you through creating a simple web application that summarizes daily meeting Transcripts using ApyHub’s AI Summarize Document API using NodeJS. Pre-requisites: Access to Google Meet and permission to access meeting transcripts.
A preferred programming language environment (e.g., Python).
ApyHub Account.
GitHub CLI is installed on your computer. Step 1: Clone the GitHub repository Open your terminal and run this command: git clone https://github.com/iamspathan/AI-Transcript-Summarizer.git This command will clone the project in your local environment. Once the project is cloned, you can open the repository in the code editor. Step 2. Install Required Packages: Here, in Node Package Manager (npm), we will install all the dependencies that our application needs to summarize the meeting transcribe. npm install Step 3: Add your API Key to the index.js file. Here, we will add our Secret API key from ApyHub. To generate an API key, you can visit the documentation page of AI Summarize Document API. Your secret looks like this ➡️
apy-token: APY****************************************** Copy this token and paste it in as shown in the code comment below. const express = require('express');
const fileUpload = require('express-fileupload');
const cors = require('cors');
const axios = require('axios');
const FormData = require('form-data');
const open = require('open');
const app = express();
const port = 3000;

app.use(cors());
app.use(fileUpload());

// Serve static files from the 'public' directory
app.use(express.static('public'));

app.post('/summarize', (req, res) => {
    console.log("summarixe called", req.files.file);
    if (!req.files || Object.keys(req.files).length === 0) {
        return res.status(400).send('No files were uploaded.');
    }

    // Assuming the file is attached to the request under the key 'file'
    const file = req.files.file;

    // Prepare the file to be sent using form-data
    const formData = new FormData();
    formData.append('file', file.data, file.name);

    // Set up the request headers, including the dynamic headers from form-data and the secret-apy token.
    const requestHeaders = {
        ...formData.getHeaders(),
        'apy-token': 'ADD YOUR OWN APY-TOKEN HERE'
    };

    // Call the AI Summarize API
    axios.post('https://api.apyhub.com/ai/summarize-documents/file', formData, { headers: requestHeaders })
        .then(apiResponse => {
            // Assuming the API returns JSON with a summary field
            console.log(apiResponse.data.data.summary);
            res.send({ summary: apiResponse.data.data.summary });
        })
        .catch(error => {
            console.error('Error calling the AI Summarize API:', error);
            res.status(500).send('Failed to summarize the document.');
        });
});

app.listen(port, () => {
    console.log(`Server running at http://localhost:${port}`);
    // Open the index.html file in the default browser
    open(`http://localhost:${port}`);
}); Once you’ve added the API key. Save the changes using the Ctrl+S Key. Step 4: Running Your Application Once you’ve added the API key. Save the changes using Ctrl+S in your terminal, and run this command.: node index.js That’s it :) As you can see, Now you can generate a meeting transcript anytime you want :) Virtual Meetings have become an essential part of corporate collaboration, serving as the de facto way that ideas are exchanged, decisions are made, and strategies are devised. However, the true value of meetings is often diluted by their length and the complexity of the discussions. This is why note-taking and effective summaries and action points are so important. Converting long and often pointless discussions into short (actionable) summaries, meeting attendees can quickly catch up on meetings they missed, review decisions that were made, and understand discussions without having to sift through hours and hours of recordings. We thought it would be cool to create our summarizer. If you don't have one already, keep reading. This technical tutorial will guide you through creating a simple web application that summarizes daily meeting Transcripts using ApyHub’s AI Summarize Document API using NodeJS. ApyHub’s AI Summarize Document API Pre-requisites: Access to Google Meet and permission to access meeting transcripts. A preferred programming language environment (e.g., Python). ApyHub Account. GitHub CLI is installed on your computer. Access to Google Meet and permission to access meeting transcripts. Google Meet A preferred programming language environment (e.g., Python). ApyHub Account . ApyHub Account GitHub CLI is installed on your computer. Step 1: Clone the GitHub repository Open your terminal and run this command: git clone https://github.com/iamspathan/AI-Transcript-Summarizer.git git clone https://github.com/iamspathan/AI-Transcript-Summarizer.git This command will clone the project in your local environment. Once the project is cloned, you can open the repository in the code editor. Step 2. Install Required Packages: Here, in Node Package Manager (npm), we will install all the dependencies that our application needs to summarize the meeting transcribe. npm install npm install Step 3: Add your API Key to the index.js file. Here, we will add our Secret API key from ApyHub. To generate an API key, you can visit the documentation page of AI Summarize Document API . Your secret looks like this ➡️ apy-token: APY****************************************** AI Summarize Document API apy-token: APY****************************************** Copy this token and paste it in as shown in the code comment below. const express = require('express');
const fileUpload = require('express-fileupload');
const cors = require('cors');
const axios = require('axios');
const FormData = require('form-data');
const open = require('open');
const app = express();
const port = 3000;

app.use(cors());
app.use(fileUpload());

// Serve static files from the 'public' directory
app.use(express.static('public'));

app.post('/summarize', (req, res) => {
    console.log("summarixe called", req.files.file);
    if (!req.files || Object.keys(req.files).length === 0) {
        return res.status(400).send('No files were uploaded.');
    }

    // Assuming the file is attached to the request under the key 'file'
    const file = req.files.file;

    // Prepare the file to be sent using form-data
    const formData = new FormData();
    formData.append('file', file.data, file.name);

    // Set up the request headers, including the dynamic headers from form-data and the secret-apy token.
    const requestHeaders = {
        ...formData.getHeaders(),
        'apy-token': 'ADD YOUR OWN APY-TOKEN HERE'
    };

    // Call the AI Summarize API
    axios.post('https://api.apyhub.com/ai/summarize-documents/file', formData, { headers: requestHeaders })
        .then(apiResponse => {
            // Assuming the API returns JSON with a summary field
            console.log(apiResponse.data.data.summary);
            res.send({ summary: apiResponse.data.data.summary });
        })
        .catch(error => {
            console.error('Error calling the AI Summarize API:', error);
            res.status(500).send('Failed to summarize the document.');
        });
});

app.listen(port, () => {
    console.log(`Server running at http://localhost:${port}`);
    // Open the index.html file in the default browser
    open(`http://localhost:${port}`);
}); const express = require('express');
const fileUpload = require('express-fileupload');
const cors = require('cors');
const axios = require('axios');
const FormData = require('form-data');
const open = require('open');
const app = express();
const port = 3000;

app.use(cors());
app.use(fileUpload());

// Serve static files from the 'public' directory
app.use(express.static('public'));

app.post('/summarize', (req, res) => {
    console.log("summarixe called", req.files.file);
    if (!req.files || Object.keys(req.files).length === 0) {
        return res.status(400).send('No files were uploaded.');
    }

    // Assuming the file is attached to the request under the key 'file'
    const file = req.files.file;

    // Prepare the file to be sent using form-data
    const formData = new FormData();
    formData.append('file', file.data, file.name);

    // Set up the request headers, including the dynamic headers from form-data and the secret-apy token.
    const requestHeaders = {
        ...formData.getHeaders(),
        'apy-token': 'ADD YOUR OWN APY-TOKEN HERE'
    };

    // Call the AI Summarize API
    axios.post('https://api.apyhub.com/ai/summarize-documents/file', formData, { headers: requestHeaders })
        .then(apiResponse => {
            // Assuming the API returns JSON with a summary field
            console.log(apiResponse.data.data.summary);
            res.send({ summary: apiResponse.data.data.summary });
        })
        .catch(error => {
            console.error('Error calling the AI Summarize API:', error);
            res.status(500).send('Failed to summarize the document.');
        });
});

app.listen(port, () => {
    console.log(`Server running at http://localhost:${port}`);
    // Open the index.html file in the default browser
    open(`http://localhost:${port}`);
}); Once you’ve added the API key. Save the changes using the Ctrl+S Key. Ctrl+S Step 4: Running Your Application Once you’ve added the API key. Save the changes using Ctrl+S in your terminal, and run this command.: node index.js node index.js That’s it :) As you can see, Now you can generate a meeting transcript anytime you want :)

Building an AI-Powered Summarizer using NodeJS and ApyHub API To Make Meetings Easy

How to Use Knowledge Graphs for Retrieval-Augmented Generation—Without a Graph DB

পুনরুদ্ধার-অগমেন্টেড জেনারেশনের জন্য জ্ঞান গ্রাফগুলি কীভাবে ব্যবহার করবেন—একটি গ্রাফ ডিবি ছাড়াই

So nutzen Sie Wissensgraphen zur abruferweiterten Generierung – ohne Graph-DB

Cómo utilizar gráficos de conocimiento para la generación de recuperación aumentada, sin una base de datos de gráficos

Comment utiliser les Knowledge Graphs pour la génération augmentée par récupération – sans base de données graphique

रिट्रीवल-ऑगमेंटेड जेनरेशन के लिए नॉलेज ग्राफ़ का उपयोग कैसे करें - बिना ग्राफ़ DB के

グラフ DB を使用せずにナレッジ グラフを検索拡張生成に使用する方法

검색 증강 생성을 위해 지식 그래프를 사용하는 방법 - 그래프 DB 없이

Como usar gráficos de conhecimento para geração aumentada de recuperação - sem um banco de dados gráfico

Как использовать графы знаний для генерации с расширенным поиском — без базы данных графов

Almayla Artırılmış Oluşturma için Bilgi Grafikleri Nasıl Kullanılır - Grafik Veritabanı Olmadan

Cách sử dụng Sơ đồ tri thức cho thế hệ tăng cường truy xuất—Không cần cơ sở dữ liệu đồ thị

如何使用知识图谱进行检索增强生成（无需图形数据库）

What is AD and AAD? Active Directory (AD): Active Directory is a Microsoft technology used to manage computers and other devices on a network. It is a primary feature of Windows Server, an operating system that runs both local and Internet-based servers. AD allows network administrators to create and manage domains, users, and objects within a network, providing a way to organize a large number of users into logical groups and subgroups while providing access control at each level. Azure Active Directory (AAD): Azure Active Directory, now known as Microsoft Entra ID, is a cloud-based identity and access management solution from Microsoft that helps organizations secure and manage identities for hybrid and multi-cloud environments. Differences Between the Two While AD and AAD share some basic identity management capabilities, they serve very different purposes. AD is meant for internal networks, managing traditional on-premises infrastructure and applications. In contrast, AAD is designed to support web-based services that use RESTful interfaces for Office 365, Google Apps, etc., and uses different protocols for working with these services (SAML, OAuth 2.0.) Typical Use Cases in Companies AD is typically used for: Organizing users and computers within an on-premises network.


Authenticating and authorizing access to network resources.


Managing policies and permissions through Group Policy. AAD is commonly used for: Managing user identities and access to cloud applications.
Implementing single sign-on (SSO) across multiple services.
Enhancing security with multi-factor authentication (MFA) and conditional access policies. Accessing AD and AAD with Python Both AD and AAD can be accessed using Python. For AAD, Microsoft provides the Microsoft Authentication Library (MSAL) for Python, which allows you to sign in users or apps with Microsoft identities and obtain tokens to call Microsoft APIs such as Microsoft Graph or your own APIs registered with the Microsoft identity platform For AD, while there isn't a direct SDK, you can interact with it using various Python libraries that handle LDAP protocols, which AD uses for directory services. Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X(@Luca_DataTeam). Happy exploring!👋 What is AD and AAD? Active Directory (AD): Active Directory is a Microsoft technology used to manage computers and other devices on a network. It is a primary feature of Windows Server, an operating system that runs both local and Internet-based servers. AD allows network administrators to create and manage domains, users, and objects within a network, providing a way to organize a large number of users into logical groups and subgroups while providing access control at each level. Azure Active Directory (AAD): Azure Active Directory, now known as Microsoft Entra ID, is a cloud-based identity and access management solution from Microsoft that helps organizations secure and manage identities for hybrid and multi-cloud environments. Differences Between the Two While AD and AAD share some basic identity management capabilities, they serve very different purposes. AD is meant for internal networks, managing traditional on-premises infrastructure and applications. In contrast, AAD is designed to support web-based services that use RESTful interfaces for Office 365, Google Apps, etc., and uses different protocols for working with these services (SAML, OAuth 2.0.) Typical Use Cases in Companies AD is typically used for: Organizing users and computers within an on-premises network. Authenticating and authorizing access to network resources. Managing policies and permissions through Group Policy. Organizing users and computers within an on-premises network. Organizing users and computers within an on-premises network. Authenticating and authorizing access to network resources. Authenticating and authorizing access to network resources. Managing policies and permissions through Group Policy. Managing policies and permissions through Group Policy. AAD is commonly used for: Managing user identities and access to cloud applications. Implementing single sign-on (SSO) across multiple services. Enhancing security with multi-factor authentication (MFA) and conditional access policies. Managing user identities and access to cloud applications. Implementing single sign-on (SSO) across multiple services. Enhancing security with multi-factor authentication (MFA) and conditional access policies. Accessing AD and AAD with Python Both AD and AAD can be accessed using Python. Both AD and AAD can be accessed using Python. For AAD, Microsoft provides the Microsoft Authentication Library (MSAL) for Python, which allows you to sign in users or apps with Microsoft identities and obtain tokens to call Microsoft APIs such as Microsoft Graph or your own APIs registered with the Microsoft identity platform For AD, while there isn't a direct SDK, you can interact with it using various Python libraries that handle LDAP protocols, which AD uses for directory services. Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X (@Luca_DataTeam). Happy exploring!👋 LinkedIn X

A Quick Overview of Active Directory (AD) and Azure Active Directory (AAD)

Welcome to another edition of Hack marketing diaries by team HackerNoon! This time, we wanna share our learnings and observations on surveillance capitalism and why we didn't make it our business model. There’s a reason we hate those invasive Ads on Facebook. Who would like their personal and critical information to be used to see those creepily relevant Ads? According to - Meta was also involved in a major about its access to the medical data of patients. The company’s pixel tracking tool was collecting their sensitive information from hospitals including the details about their health, appointments, and medical prescriptions. Pure evil. What's even more annoying? Abrupt Ads stoping you from consuming the content. Classic example - Ads on YouTube. the Markup “Major tax filing services such as H&R Block, TaxAct, and TaxSlayer have been quietly transmitting sensitive financial information to Facebook when Americans file their taxes online.” controversy last year Here’s how HackerNoon saw these issues when we used to become not only a publishing company but also a software company: equity crowdfunding “We don't want to advertise the shoes you've considered buying around the web, we do want to surface sponsors that are relevant to the subject matters being read, such as blockchain sponsors on blockchain content. We believe sponsorship is not evil, if done right. What's evil is surveillance. Our sponsors will be by subject matter. Our promotions are like billboards - relevant to the content and not the individual reader. Our sponsorship placements are minimalistic & non-intrusive. No sidebars, no popups, no video ads. We care about reading quality and will never take that experience away from the reader.” Presenting you HackerNoon’s Targeted Ads by content relevancy - where you target the audience interested in content relevant to your company, without invading their privacy or shoving annoying pop-ups down their throats. Our targeted marketing options allow you to avoid the noise and reach the right audience. It’s a win-win for all. Talking about wins - here's an easter egg for your company - use the code 'HackerFriend' during the meeting and avail 10% off on Targeted Ads! Book a call with us to see the demo Here’s how our targeted Ads by content relevancy work: HackerNoon has curated to date 50,000+ technology tags These tags are grouped into the relevant like , , , , , , and more! parent categories AI Web3 Programming Startups Cybersecurity Finance Every story organically gets at least eight relevant tech tags and a parent category Sponsors buy with all the tags and stories. multimodal placements on relevant categories These Ad placements include Banners, Logos, Newsletter ADs, and Audio ADs - what we call truly AIO (Activities, Interests, and Opinions) Optimized for: Brand Recall and Clickability (Get 3x clicks for the same impressions compared to elsewhere) Get quality leads at unbeatable prices, with CPM ~ $7 and CPC ~ $5. Find pricing and details in the attached deck. to learn how AWS, Bosch, Brave, DataStax, Bright Data, Twilio, SonarSource, Algolia, Algorand, Postman, IONOS, and thousands of other tech companies partner with HackerNoon to target their audiences. Book a meeting with us Until next time!

How to Advertise to Your Target Audience without Creeping Them Out

কীভাবে আপনার টার্গেট শ্রোতাদের কাছে বিজ্ঞাপন দেওয়া যায় তাদের আউট না করে

So werben Sie bei Ihrer Zielgruppe, ohne sie zu verschrecken

Cómo anunciarse ante su público objetivo sin asustarlos

Comment faire de la publicité auprès de votre public cible sans l'effrayer

अपने लक्षित दर्शकों को बिना परेशान किए उनके लिए विज्ञापन कैसे करें

ターゲット層を怖がらせずに広告を出す方法

타겟 고객을 소외시키지 않고 광고하는 방법

Como anunciar para seu público-alvo sem assustá-lo

Как рекламировать свою целевую аудиторию, не отпугивая ее

Hedef Kitlenizi Korkutmadan Nasıl Reklam Verebilirsiniz?

Cách quảng cáo tới đối tượng mục tiêu của bạn mà không khiến họ sợ hãi

如何向目标受众投放广告而不让他们感到害怕

Introduction Jupyter Notebook is a popular tool among data scientists and programmers for interactive data analysis, visualization, and machine learning. In this guide, we will walk you through the process of setting up Jupyter Notebook on a Windows server to enable remote access. Step 1: Install Jupyter Notebook First, you need to install Jupyter Notebook on your Windows server. You can do this using conda or pip. Here are the commands to install Jupyter Notebook: Using Conda: conda install -c conda-forge jupyterlab Using pip: pip install jupyterlab Step 2: Modify the Configuration File Generate the Jupyter Notebook configuration file by running the following command in the command prompt: jupyter notebook --generate-config Open the generated file and modify the following configurations: jupyter_notebook_config.py c.NotebookApp.ip=''  # Allow any IP to access
c.NotebookApp.password='your_generated_password'  # Set a password for authentication
c.NotebookApp.open_browser=False  # Do not open a browser when starting Jupyter Notebook
c.NotebookApp.port=your_selected_port  # Choose a port number for Jupyter Notebook Step 3: Set a Customized Password Launch IPython by running in the command prompt. and In the IPython environment, run the following commands: ipython from notebook.auth import passwd
passwd() Enter a password and confirm it. Copy the generated sha1 value. Replace the in the file with the generated sha1 value. 'your_generated_password' jupyter_notebook_config.py Step 4: Remote Access Start the Jupyter Notebook server by running in the command prompt. jupyter notebook To access Jupyter Notebook remotely, use your server's IP address along with the selected port number in a web browser. Enter the password when prompted to access the Jupyter Notebook interface. Conclusion By following these steps, you can set up and access Jupyter Notebook on a Windows server for interactive data analysis and development tasks. Enjoy using Jupyter Notebook with the flexibility of remote access! Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on or (@Luca_DataTeam). LinkedIn X Also published . here

How To Set Up Jupyter Notebook on a Windows Server

Surveillance capitalism… How about we not make that our business model? Here’s how HackerNoon saw it when we used to become but also a : equity crowdfunding not only a publishing company software company “We don't want to advertise the shoes you've considered buying around the web, we do want to surface sponsors that are relevant to the subject matters being read, such as blockchain sponsors on blockchain content. We believe sponsorship is not evil, if done right. What's evil is surveillance. Our sponsors will be either site-wide or by subject matter. We think there is a major difference between a "sponsor" and an "advertiser": Our promotions are like billboards - relevant to the content and not the individual reader. Without sponsors, we can't make the site free for readers forever. Our sponsorship placements are minimalistic & non-intrusive. No sidebars, no popups, no video ads. We care about reading quality and will never take that experience away from the reader. One of the main reasons Jonah Peretti left Huffington Post and started Buzzfeed was because the ad system was not baked into the tech. By increasing the ad duration and having no sales middlemen, Hacker Noon has been able to offer customers better rates and better relevancy, and we'll have healthy margins with longer-term customers. We want to own the relationship with the brands and have quality brands that validate the content. We have been extremely selective in choosing which brands to be HackerNoon sponsors and will continue to do so going forward.” So, how does ad placement by content relevancy work at HackerNoon now? HackerNoon has curated 50,000+ technology tags to date These tags are grouped into the like , , , , , , and more! relevant parent categories AI Web3 Programming Startups Cybersecurity Finance Every story organically gets eight relevant and a parent category tech tags Sponsors buy multimodal placements on relevant categories with all the tags and stories. These Ad placements include Banners, Logos, Newsletter ADs, and Audio ADs  - truly AIO (Activities, Interests, and Opinions) Optimized For: ( ) Brand Recall and Clickability Get 3x clicks for the same impressions compared to elsewhere to learn how AWS, Brave, DataStax, Bright Data, Twilio, SonarSource, Algolia, Algorand, Postman, IONOS, and thousands of other tech companies partner with HackerNoon to target their audiences. Book a meeting with us

Targeted Ads: Ad Placements by Content Relevancy

লক্ষ্যযুক্ত বিজ্ঞাপন: বিষয়বস্তুর প্রাসঙ্গিকতা অনুসারে বিজ্ঞাপনের স্থান

Gezielte Anzeigen: Anzeigenplatzierungen nach Inhaltsrelevanz

Anuncios dirigidos: ubicaciones de anuncios por relevancia del contenido

Annonces ciblées : emplacements d'annonces par pertinence du contenu

लक्षित विज्ञापन: सामग्री प्रासंगिकता के आधार पर विज्ञापन प्लेसमेंट

ターゲットを絞った広告: コンテンツの関連性による広告の配置

타겟 광고: 콘텐츠 관련성에 따른 광고 배치

Anúncios direcionados: canais de anúncios por relevância de conteúdo

Таргетированная реклама: места размещения объявлений по релевантности контента

Hedefli Reklamlar: İçerik Alaka Düzeyine Göre Reklam Yerleşimleri

Quảng cáo được nhắm mục tiêu: Vị trí đặt quảng cáo theo mức độ liên quan của nội dung

定向广告：按内容相关性划分的广告展示位置

Stories, News & Mentions about DATASTEMA

Datastema’s Stories on HackerNoon