DataStema

datastema

In this article, I want to tell you how to connect and further generate Allure reports for a test project if your test project is made on the Citrus Framework. I hope you know about Allure framework, which is designed to generate reports after passing tests. The report opens on a browser page, where you can see which tests passed successfully and which did not, you can also see at which test step the expected result did not match. Since the Citrus Framework can use the TestNG or Junit engines, you need to include the Allure library according to the engine you are using. I will show the example of TestNG and the project builder we use Maven. 1. First of all, the Allure distribution must be installed locally and the path to the distribution must be written in PATH, similar to installing Maven. The distribution can be taken from here by choosing the latest version: https: //repo.maven.apache.org/maven2/io/qameta /allure/allure-commandline You can check if you have Allure locally with this command in cmd: allure --version If everything is ok, you will see the current version installed locally: 2. In the pom.xml file, you need to add the dependency “allure-testng” and two plugins “maven-surefire-plugin” and “allure-maven”, probably “maven-surefire-plugin” will already be in your list, then add the configuration, as shown below: </properties> <properties> < aspectj.version > 1.9.4 </ aspectj.version > < allure.version > 2.22.0 </ allure.version > < dependencies > < dependency > < groupId > io.qameta.allure </ groupId > < artifactId > allure-testng </ artifactId > < version > ${allure.version} </ version > </ dependency > </ dependencies > < build > < plugins > < plugin > < groupId > org.apache.maven.plugins </ groupId > < artifactId > maven-surefire-plugin </ artifactId > < version > 2.22.2 </ version > < configuration > < failIfNoTests > false </ failIfNoTests > < workingDirectory > ${project.build.directory} </ workingDirectory > < argLine > -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </ argLine > < systemPropertyVariables > < allure.results.directory > ${project.build.directory}/allure-results </ allure.results.directory > </ systemPropertyVariables > </ configuration > < dependencies > < dependency > < groupId > org.aspectj </ groupId > < artifactId > aspectjweaver </ artifactId > < version > ${aspectj.version} </ version > </ dependency > </ dependencies > </ plugin > < plugin > < groupId > io.qameta.allure </ groupId > < artifactId > allure-maven </ artifactId > < version > 2.10.0 </ version > < configuration > < reportVersion > ${allure.version} </ reportVersion > < allureDownloadUrl > https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/${allure.version}/allure-commandline-${allure.version}.zip </ allureDownloadUrl > </ configuration > </ plugin > </ plugins > </ build > After Reload Maven, the connected Allure plugin will appear in the list of plugins: 3. Now we can add Allure annotations to tests as shown in the screenshot: private TestContext context; @CitrusTest sendRequest(); getResponse();
    } $(http() .send() } .receive()
                        .response(HttpStatus.OK)
                        .message()
                        .type(MessageType.JSON) );
    } user.setData(data); user.setSupport(support); }
} @Story( "GET REQUEST" ) public class FirstTestGetUser extends TestNGCitrusSupport { @Test(description = "Getting information about user" ) public void getTestActions ( ) { this .context = citrus.getCitrusContext().createTestContext(); //        "Send GET request" //        "Get response" @Step( "Send GET request" ) public void sendRequest ( ) { .client( "restClientReqres" ) .get( "users/" + context.getVariable( "userId" ))); @Step( "Get response" ) public void getResponse ( ) { $(http().client( "restClientReqres" ) .body( new ObjectMappingPayloadBuilder(getJsonData(), "objectMapper" )) public User getJsonData ( ) { User user = new User(); Data data = new Data(); data.setId(Integer.valueOf(context.getVariable( "userId" ))); data.setEmail( "janet.weaver@reqres.in" ); data.setFirstName( "Janete" ); data.setLastName( "Weaver" ); data.setAvatar( "https://reqres.in/img/faces/2-image.jpg" ); Support support = new Support(); support.setUrl( "https://reqres.in/#support-heading" ); support.setText( "To keep ReqRes free, contributions towards server costs are appreciated!" ); return user; We added Allure annotations @Story for class and @Step for methods in the test﻿ above. 4. After passing the test, we can generate a report by calling the "allure:serve" command: 5. Browser will automatically launch with the report page: Thus, we connected Allure reports to our test project. If you are using the Junit engine in your test project, then you will need to follow the same steps, only you will need the “allure-junit4” or “allure-junit5” dependency. This is all. I wish you happy testing).

Let's Connect Allure Reports to a Test Project Made on Citrus Framework

In this tutorial, we fine-tune a model for topic classification using the Hugging Face Transformers and Datasets libraries. By the end of this tutorial, you will have a powerful fine-tuned model for classifying topics and published it to Hugging Face 🤗 for people to use. RoBERTa Prerequisites This article assumes you have a 🤗 account and working Python, NLP, and Deep learning knowledge. Hugging Face : A basic understanding of machine learning concepts, such as supervised learning, training, and evaluation. Understanding of machine learning concepts will help follow the code presented in the tutorial. Python programming experience : Some experience with deep learning and natural language processing (NLP) concepts, such as neural networks, word embeddings, and tokenization, will be beneficial for grasping the ideas presented in the tutorial. Familiarity with deep learning and natural language processing . Access to a Google Colab or Jupyter Notebook environment : To publish your fine-tuned model to the Hugging Face Hub, you will need a Hugging Face account. If you do not already have an account, you can sign up for one at . A Hugging Face account https://huggingface.co/join By meeting these prerequisites, you will be well-prepared to follow the tutorial and get the most out of it. Let's get our hands dirty 😁 We start by installing the dependencies. !pip install transformers datasets huggingface_hub tensorboard==2.11
!sudo apt-get install git-lfs --yes We then import the needed modules. import torch
from datasets import load_dataset
from transformers import (
    RobertaTokenizerFast,
    RobertaForSequenceClassification,
    TrainingArguments,
    Trainer,
    AutoConfig,
)
from huggingface_hub import HfFolder, notebook_login We need to log in to Hugging Face by using a token. notebook_login() Let's set some variables for easier configuration. model_id = "roberta-base"
dataset_id = "ag_news"
# relace the value with your model: ex <hugging-face-user>/<model-name>
repository_id = "achimoraites/roberta-base_ag_news" Preprossessing Next, we load our dataset and do some preprocessing. # Load dataset
dataset = load_dataset(dataset_id)

# Training and testing datasets
train_dataset = dataset['train']
test_dataset = dataset["test"].shard(num_shards=2, index=0)

# Validation dataset
val_dataset = dataset['test'].shard(num_shards=2, index=1)

# Preprocessing
tokenizer = RobertaTokenizerFast.from_pretrained(model_id)

# This function tokenizes the input text using the RoBERTa tokenizer. 
# It applies padding and truncation to ensure that all sequences have the same length (256 tokens).
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True, max_length=256)

train_dataset = train_dataset.map(tokenize, batched=True, batch_size=len(train_dataset))
val_dataset = val_dataset.map(tokenize, batched=True, batch_size=len(val_dataset))
test_dataset = test_dataset.map(tokenize, batched=True, batch_size=len(test_dataset)) Set The Dataset Format The function is used to specify the dataset format, making it compatible with PyTorch. set_format() The argument lists the columns that should be included in the formatted dataset. columns In this case, the columns are "input_ids", "attention_mask", and "label". By setting the format and specifying the relevant columns, we prepare the datasets for use with the Hugging Face Trainer class, which requires PyTorch tensors as input. # Set dataset format
train_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"])
val_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"])
test_dataset.set_format("torch", columns=["input_ids", "attention_mask", "label"]) To make our model easier to use, we will create an mapping. This will map the class ids to their labels. id2label This makes it easier to interpret the model’s output during inference. # We will need this to directly output the class names when using the pipeline without mapping the labels later.
# Extract the number of classes and their names
num_labels = dataset['train'].features['label'].num_classes
class_names = dataset["train"].features["label"].names
print(f"number of labels: {num_labels}")
print(f"the labels: {class_names}")

# Create an id2label mapping
id2label = {i: label for i, label in enumerate(class_names)}

# Update the model's configuration with the id2label mapping
config = AutoConfig.from_pretrained(model_id)
config.update({"id2label": id2label}) Training and Evaluation Now, we will set up our training parameters, Hugging Face 🤗 repository, and Tensorboard. # TrainingArguments
training_args = TrainingArguments(
    output_dir=repository_id,
    num_train_epochs=5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    evaluation_strategy="epoch",
    logging_dir=f"{repository_id}/logs",
    logging_strategy="steps",
    logging_steps=10,
    learning_rate=5e-5,
    weight_decay=0.01,
    warmup_steps=500,
    save_strategy="epoch",
    load_best_model_at_end=True,
    save_total_limit=2,
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repository_id,
    hub_token=HfFolder.get_token(),
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
) We can start the training process by running: # Fine-tune the model
trainer.train() Evaluate the model: # Evaluate the model
trainer.evaluate() Publishing We are ready to publish our model to Hugging Face 🤗 # Save our tokenizer and create a model card
tokenizer.save_pretrained(repository_id)
trainer.create_model_card()
# Push the results to the hub
trainer.push_to_hub() Test The Model At this point, our model should have been published and will be available for use. Let's test it! # TEST MODEL

from transformers import pipeline

classifier = pipeline('text-classification',repository_id)

text = "Kederis proclaims innocence Olympic champion Kostas Kederis today left hospital ahead of his date with IOC inquisitors claiming his innocence and vowing: quot;After the crucifixion comes the resurrection. quot; .."
result = classifier(text)

predicted_label = result[0]["label"]
print(f"Predicted label: {predicted_label}") Congratulations You have fine-tuned and published a RoBERTa model for text classification using Hugging Face 🤗 transformers and datasets libraries! **For reference, here is my fine-tuned model on Hugging Face 🤗
\ https://huggingface.co/achimoraites/roberta-base_ag_news **You can find the code here
\ https://github.com/achimoraites/machine-learning-playground/blob/main/NLP/Text classification/RoBERTa_Finetuning.ipynb Happy 🤖 learning 😀! Also published here Feature image: Photo by Vlada Karpovich from Pexels: https://www.pexels.com/photo/crop-young-businesswoman-using-laptop-while-drinking-tea-at-home-4050347/

Fine-Tuning RoBERTa for Topic Classification

After the success of and the rise of deep learning, tackling video was an obvious next step. Just like the classification of images, the task of video classification is the most straightforward start on the path to general video understanding models. As for the specific labels that are being classified, the computer vision research community has gravitated toward classifying human actions in videos. image classification dataset challenges One of the earliest human action recognition video datasets, even before deep learning took off, was from 2004. Action recognition datasets have come a long way since then, some focusing on clips from , while others . the KTH dataset Hollywood movies focusing on sports In 2017, DeepMind released one of the largest and most impactful human action recognition datasets yet, . As of the writing of this post, four versions of the Kinetics dataset have been released: , , , and . The version number indicates the number of action classes. Additionally, each version adds new videos to replace those that have been deleted from YouTube over time. Kinetics 400 600 700 700–2020 This integration includes a sophisticated way to download the dataset, as well as examples of how to evaluate and improve models trained on the dataset. This post walks through the integration of Kinetics into the open-source dataset curation and model analysis tool, . FiftyOne Downloading Kinetics is now as easy as: import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("kinetics-600") Setup To run the examples in this post, you need to : install FiftyOne pip install fiftyone You will also need to install which is used by FiftyOne to download videos from YouTube: Pytube pip install pytube Downloading Kinetics Until recently, the only way to access the Kinetics dataset was to download each video directly from their sources on YouTube. This resulted in including videos having been deleted, YouTube throttling downloads, and inefficiencies in clipping videos. numerous issues The Common Visual Data Foundation (CVDF) has collaborated with the Kinetics dataset maintainers to for the general public to download. It should be noted that the CVDF-hosted version does not include all samples present in the original dataset, only those that were available on YouTube at the time that the CVDF version was created. host all versions of the dataset on AWS The CVDF has made it much easier to gain access to the full dataset. However, you still need to handle the challenges of visualizing, wrangling, and subsetting the dataset to meet your needs. In some cases, you don’t want to have to download the entire dataset, to begin with. This is where the integration of comes in. With just one line of Python code, you can now specify the version, the split, and the classes that you want and then visualize it in the with just another line of code. Kinetics into the FiftyOne Dataset Zoo FiftyOne App import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "kinetics-700-2020",
    split="validation",
    classes=["grooming cat", "grooming dog"],
    max_samples=10,
)

session = fo.launch_app(dataset) Training and Evaluating a Model After having downloaded Kinetics, you can now start using it to train action recognition models. Since the dataset is already in FiftyOne, it is easy to use libraries like or to train a model directly on the dataset. PyTorch PyTorch Lightning Flash pip install lightning-flash lightning-flash[video] torchvision pytorchvideo import torch

from flash import Trainer
from flash.video import VideoClassificationData, VideoClassifier

import fiftyone as fo
import fiftyone.zoo as foz

classes = [
    "swimming backstroke",
    "swimming breast stroke",
    "swimming butterfly stroke",
    "swimming front crawl",
]

# Load Kinetics
dataset = foz.load_zoo_dataset(
    "kinetics-700-2020",
    splits=["train", "validation"],
    classes=classes,
    max_samples=50,
    shuffle=True,
)

# Replace spaces in class names with underscore
labels = dataset.distinct("ground_truth.label")
labels_map = {l: l.replace(" ", "_") for l in labels}
dataset = dataset.map_labels("ground_truth", labels_map).clone()

# Create views for dataset splits
train_view = dataset.match_tags("train")
val_view = dataset.match_tags("validation")

# Create the Flash Datamodule
datamodule = VideoClassificationData.from_fiftyone(
    train_dataset=train_view,
    val_dataset=val_view,
    predict_dataset=val_view,
    label_field="ground_truth",
    batch_size=1,
    clip_sampler="uniform",
    clip_duration=1,
    decode_audio=False,
)

# Build the model
model = VideoClassifier(
    backbone="x3d_xs",
    labels=datamodule.labels,
    pretrained=True,
)

trainer = Trainer(
    max_epochs=10,
    limit_train_batches=5,
    gpus=torch.cuda.device_count(),
)

# Finetune the model
trainer.finetune(model, datamodule=datamodule, strategy="freeze") After your model is trained, you can then generate predictions on the validation and test splits and use FiftyOne to of the model. evaluate the performance from itertools import chain

from flash.core.classification import FiftyOneLabelsOutput

def get_fo_label_preds(samples, datamodule, trainer):
    # Return a list of predictions in fo.Detection format
    predictions = trainer.predict(
        model,
        datamodule=datamodule,
        output=FiftyOneLabelsOutput(return_filepath=False, labels=datamodule.labels),
    )
    predictions = list(chain.from_iterable(predictions))  # flatten batches
    return predictions

predictions = get_fo_label_preds(val_view, datamodule, trainer)

# Add predictions to FiftyOne dataset
val_view.set_values(
    "predictions", predictions
)

session = fo.launch_app(val_view)

results = val_view.evaluate_classifications(
    "ground_truth",
    "predictions",
    eval_key="eval",
) The results of the evaluation can be used for things like plotting and . confusion matrices precision-recall curves pip install ipywidgets underscore_classes = [c.replace(" ", "_") for c in classes]

plot = results.plot_confusion_matrix(classes=underscore_classes)
plot.show() As you can see, since we only finetuned the model on a few dozen samples, it is overfitting to the backstroke and butterfly stroke classes. This implies that we should download additional samples of the other two classes and continue training. Analyzing the model to find the best and worst-performing samples can shed light on the best ways to improve your model’s performance. from fiftyone import ViewField as F

eval_view = val_view.filter_labels(
    "predictions", (F("confidence") > 0.6) & (F("eval") == False)
)

session.view = eval_view The following shows one of the top examples in this evaluation view of highly confident but incorrectly predicted samples. There are multiple issues that we can see with this sample. First, the footage is first-person which is rare in this dataset. If we want to predict on first-person videos, then more should be added to the training set. Second, there are examples of both breaststroke and backstroke in the video so it would be difficult to assign a label. Third, the ground truth label is front crawl which does not appear at all in the dataset. Using FiftyOne to get hands-on and analyze specific samples can lead to results like these highlighting ways that you can improve the dataset itself. Since Kinetics is a very large dataset, we could easily download additional videos to supplement problematic samples that we may want to exclude from training. Improvements to your dataset can lead to easier gains in model performance than working on improving the model architecture itself. Summary The makes it easier than ever to be able to download exactly the subset of Kinetics that you want or even the dataset in its entirety. Additionally, FiftyOne allows for in-depth evaluation and analysis of video models leading to better datasets and higher performing models. integration of the Kinetics dataset into FiftyOne -- (Originally posted by ) here

Kinetics Dataset - Training and Evaluating Models for Video Classification 

Photo by on Mick De Paola Unsplash The “Maybe just a quick one” series title is inspired by my most common reply to “Fancy a drink?”, which, may or may not end up in a long night. Likewise, these posts are intended to be short but I get carried away sometimes, so, apologies in advance. About 🤗 Transformers 🤗 Transformers ( ) is a collection of state-of-the-art NLU (Natural Language Understanding) and NLG (Natural Language Generation ) models.  They offer a wide variety of architectures to choose from (BERT, GPT-2, RoBERTa etc) as well as a of pre-trained models uploaded by users and organisations. Hugging Face transformers hub Fine-tuning a model One of the things that makes this library such a powerful tool is that we can use the models as a basis for tasks. In other words, they can be a starting point to apply some fine-tuning using our own data. The library is designed to easily work with both Tensorflow or PyTorch. transfer learning 🤗 Datasets is a  wrapper library that provides some tools to load and process data in many commonly used formats (CSV, JSON etc). It also makes sharing datasets and metrics for Natural Language Processing extremely easy. Hugging Face Datasets 🤗 Datasets originated from a fork of the awesome TensorFlow Datasets and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. Well, let’s write some code In this example, we will start with  a pre-trained model and fine-tune it on the dataset. We will then test it  on classifying tweets as hate speech, offensive language, or neither. All coding is done in . BERT (uncased) Hate Speech and Offensive Language Google Colab Please note:  this dataset contains text that can be considered racist, sexist, homophobic, or generally offensive. So let’s start by installing some necessary packages, import them and load the dataset. The dataset is stored in Google Drive, and the path to load it from is . So if you code along , please make sure you change the path to point to your own dataset file. /content/drive/MyDrive/Data/labeled_data.csv We are using the function to load it and then split it into train, validation, and test sets. The 3 sets are then gathered together to form a .This is a dictionary class, that offers us many methods to process the data. We will then remove some of the columns we don’t need for our classification task. load_dataset DatasetDict !pip install transformers
!pip install datasets datasets load_dataset,DatasetDict transformers AutoTokenizer,TFAutoModelForSequenceClassification tensorflow tf numpy np matplotlib.pyplot plt

The dataset located our Google Drive data folder.

DATA_PATH = dataset = load_dataset( , data_files=DATA_PATH,split= )
train_testvalid = dataset.train_test_split()
test_valid = train_testvalid[ ].train_test_split()
train_test_valid_dataset = DatasetDict({ : train_testvalid[ ], : test_valid[ ], : test_valid[ ]})
dataset = train_test_valid_dataset.remove_columns([ , , , , ]) #Install the necessary packages from import from import import as import as import as is in "/content/drive/MyDrive/Data/labeled_data.csv" 'csv' 'train' 'test' 'train' 'train' 'test' 'test' 'valid' 'train' 'hate_speech' 'offensive_language' 'neither' 'Unnamed: 0' 'count' So now we need to preprocess the data. The tool responsible for this is a What do tokenizers do? Very simply put, they split the data in tokens (these can be characters, words, part of words, depending on the model), and convert them into tensors of numeric ids, which is the form that the model can read. For this task, we are using the tokenizer from the pre-trained model we selected ( ). But let’s see how we achieve this: Tokenizer. bert-base-cased tokenizer = AutoTokenizer.from_pretrained( ) tokenizer(examples[ ], padding= , truncation= )

tokenized_datasets = dataset.map(tokenize_function, batched= )
train_dataset = tokenized_datasets[ ]
eval_dataset = tokenized_datasets[ ]
test_dataset = tokenized_datasets[ ]
tf_train_dataset = train_dataset.remove_columns([ ]).with_format( )
tf_eval_dataset = eval_dataset.remove_columns([ ]).with_format( )
tf_test_dataset = test_dataset.remove_columns([ ]).with_format( )

train_features = {x: tf_train_dataset[x].to_tensor() x tokenizer.model_input_names}
train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset[ ]))
train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch( )

eval_features = {x: tf_eval_dataset[x].to_tensor() x tokenizer.model_input_names}
eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset[ ]))
eval_tf_dataset = eval_tf_dataset.batch( )

test_features = {x: tf_test_dataset[x].to_tensor() x tokenizer.model_input_names}
test_tf_dataset = tf.data.Dataset.from_tensor_slices((test_features, tf_test_dataset[ ]))
test_tf_dataset =test_tf_dataset.batch( ) "bert-base-cased" : def tokenize_function (examples) return "tweet" "max_length" True True "train" "valid" 'test' "tweet" "tensorflow" "tweet" "tensorflow" "tweet" "tensorflow" for in "class" 8 for in "class" 8 for in "class" 8 Notice how we used the dataset function, to apply our user-defined to all the elements of the dataset. map tokenize_function After applying the tokenization, we created 3 to feed the model with. We are now ready to train the model. Again, we are using the selected pre-trained model, transferring the “knowledge” it already has, but replacing its head with one that is suited to our task. We are using the class, which represents a generic Tensorflow (hence the TF prefix) model, with a sequence classification head. Also, notice the num_labels parameter which is set to 3, as this is a multi-class task with 3 distinct labels. After the training is finished, we plot the Sparse Categorical Accuracy and the Loss of both the train and the validation dataset. Tensorflow Datasets TFAutoModelForSequenceClassification model = TFAutoModelForSequenceClassification.from_pretrained( , num_labels= )
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate= ),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits= ),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)

history = model.fit(train_tf_dataset, validation_data=eval_tf_dataset, epochs= )

plt.plot(history.history[ ])
plt.plot(history.history[ ])
plt.title( )
plt.ylabel( )
plt.xlabel( )
plt.legend([ , ], loc= )
plt.show()
plt.plot(history.history[ ])
plt.plot(history.history[ ])
plt.title( )
plt.ylabel( )
plt.xlabel( )
plt.legend([ , ], loc= )
plt.show() "bert-base-cased" 3 5e-5 True 2 'sparse_categorical_accuracy' 'val_sparse_categorical_accuracy' 'model sparse categorical accuracy' 'accuracy' 'epoch' 'train' 'val' 'upper left' 'loss' 'val_loss' 'model loss' 'loss' 'epoch' 'train' 'val' 'upper left' We can now evaluate the model on the test dataset we created earlier: test_loss, test_acc = model.evaluate(test_tf_dataset,verbose= )
print( , test_acc) / - s - loss: - sparse_categorical_accuracy: Test accuracy: model.save_pretrained( ) 2 '\nTest accuracy:' 194 194 62 0.2596 0.9135 0.9134925603866577 "/content/drive/MyDrive/Data/hate-speech-bert" Notice that we save the model with the function offered by Transformers. This action generates a directory with two files by default: a .json file that contains the model configuration and a .h5 file with the model weights. We can also push the model to the Hugging Face Models Hub should we want to, in order to make it available to the public. save_pretrained Does it work though? Let’s see how our model does in classifying some unseen text. I will use some stereotypical racist/offensive/sexist texts posted on social media. : Warning Due to the nature of this task, the language used here can be racist, sexist, and offensive. However, this is the only way to evaluate the model’s ability. pred2label = { : , : , : }
preds = model(tokenizer([ , , ],return_tensors= ,padding=True,truncation=True))[ ]
print(preds)
class_preds = np.argmax(preds, axis= ) pred class_preds:
  print(pred2label[pred]) 0 'Hate Speech' 1 'Offensive Language' 2 'Neither' "Jews are useless , I don't see why they even exist" "Gay people suck" "Women are dressed up like whores these days" "tf" 'logits' 1 for in tf.Tensor(
[[ ]
 [ ]
 [ ]], shape=( , ), dtype=float32)
Hate Speech
Hate Speech
Offensive Language 0.37532297 0.14053927 -0.8647832 0.04699412 -0.17951615 -0.3738104 0.2524849 2.586107 -2.8212454 3 3 Looks like our model managed to correctly (well that’s subjective, but generally speaking these look correct) classify the texts. Further reading: If the above seems interesting to you, there is a lot more that this library can do. I would start by checking their documentation which is quite extensive as well as some quick video courses they provide: 🤗 Transformers Model Hub 🤗 Datasets (with Youtube Videos, we always like them ) Crash course Fine-tuning with custom datasets Happy coding.

How to Fine Tune a 🤗 (Hugging Face) Transformer Model

Yes, building AI is hard! Every step from data annotation, training and deployment comes with its own set of challenges. This blog post will try to deal with the first two of those: Data Annotation Training Over the past few years at , we have been developing highly efficient AI systems deployed in millions of consumer devices as of today! And in this process, we developed a lot of tools to simplify our workflow. We have finally decided to bridge the AI skills gap by making some of our tools public. One of the tools, Brain Builder deals with the first of the aforementioned issues: Data Annotation. Neurala is an AI-assisted annotation tool that fits seamlessly into commonly used frameworks like and Caffe. This post will walk you through the steps you’d need to incorporate Brain Builder into your AI workflow. We will build a semantic segmentation network using the data tagged by Brain Builder and model written using TensorFlow and Keras. Brain Builder TensorFlow Semantic segmentation is one of the commonly encountered applications of computer vision. It is the task of labeling every pixel of an image with a . It is used extensively for image analysis and enables applications like portrait mode on smartphones. The following tutorial will describe how you can create a semantic segmentation model to segment people using your own data and train it over multiple GPUs, using Brain Builder and TensorFlow. class Data Annotation using Brain Builder Data annotation is the most crucial step in building an AI system since this is what your model learns from. And a deep learning model is only as good as the data it’s fed! Brain Builder helps you in curating a quality dataset and has AI-assisted tools to speed up the data curation step. One of the most important features of Brain Builder is automated video tagging. AI-assisted video tagging lets you collect a lot of data in a small amount of time by extracting and tagging frames from videos recorded at very high frame rates. The following video shows the process of tagging people in videos using brain builder: The folder contains everything you will need in order for TensorFlow to be able to read and process your dataset. All the ground truth images are , where every pixel in the image corresponds to a class index. The files provide you details about the mapping of class index to their subsequent R, G, B values that you can later use for visualizing your results. class index-labeled pngs class_dictionary.csv, classes.txt and palette.txt We will now write a popular architecture for semantic segmentation called using TensorFlow and Keras. Using Estimators and the TensorFlow Dataset API, we’d enable training over multiple GPUs to reduce the training times massively! UNet (Please note that we’ve modified the UNet slightly in our implementation to make this a quicker experiment!) Multi-GPU training using TensorFlow Estimators and Dataset API End-to-end integration of Keras with TensorFlow has made it easy to enable multi-GPU training of Keras models using TensorFlow Estimators and Dataset API. TensorFlow Estimators The class represents a model, as well as how this model should be trained and evaluated. Some of the key advantages of using Estimators are: Estimator Estimator based models can be run across multiple GPUs without changing the model code; Estimators simplify sharing the implementation of models across developers; Estimators build the graph for you and eliminate the need of an explicit (and rather painful) session. The following piece of code creates a Keras model, compiles it and converts it to an TensorFlow Estimator: dataset = TFDatasetLoader(data_dirpath, input_size, n_classes, batch_size, num_epochs)model = UNet(input_size, n_classes)model.compile(optimizer=tf.train.AdamOptimizer(learning_rate=0.01), loss='binary_crossentropy', metrics=['accuracy']) strategy = tf.contrib.distribute.MirroredStrategy(num_gpus=num_gpus)config = tf.estimator.RunConfig(train_distribute=strategy)estimator = tf.keras.estimator.model_to_estimator(model, config=config) If you’re familiar with Keras, the only new things you’d notice are the last three lines of code. The strategy we use for multi-GPU training is called . In this strategy, each GPU has a copy of the graph and gets a subset of the data on which it computes the local gradients. Once the local gradients are computed, each GPU then waits for other GPUs to finish in a synchronous manner. When all the gradients have arrived each GPU averages them and updates its parameter and the next step begins. You can learn more about the Distributed TensorFlow training on link. MirroredStrategy this Now that the strategy has been defined, we create a object using that strategy and use it to call the function which converts the Keras model into an Estimator object. RunConfig model_to_estimator With the estimator ready, we can simply train and evaluate it using these two lines of code: estimator.train(lambda:dataset.imgs_input_fn(mode="train"), hooks=[time_hist])estimator.evaluate(lambda:dataset.imgs_input_fn(mode="val")) The important thing to note in these two calls is the function. This function feeds the model the data using the TensorFlow Dataset API. dataset.imgs_input_fn TensorFlow Dataset API Dataset is a highly efficient API for data input that ties nicely with the Estimators and tf.Keras. The core of the new input pipeline is the (and maybe the ). A is a collection of , each with the same structure, where one element can be one or more tensors. The different tensors inside an element are called . Each component has a certain data type and shape, but different components inside one element can have different data types and shapes. Dataset Iterator Dataset elements components The following code snippet shows how to create a dataset object that can be fed to the Estimators: dataset = tf.data.Dataset.from_tensor_slices((image_paths, gt_paths))dataset = dataset.map(lambda image, label: tuple(tf.py_func(self._one_hot_gt, [image, label], [tf.string, tf.uint8])))dataset = dataset.map(self._data_to_tensor)dataset = dataset.repeat(self.epochs)dataset = dataset.batch(self.batch_size)dataset = dataset.prefetch(self.batch_size) The important function to take a note of is the function that lets you read the data and process it any way you want (resize, one-hot code, etc). lets you specify how many times you want to iterate over this dataset. automatically batches the data based on the batch size provided by the user and prefetches the next batch while the current batch is being processed. map() repeat() batch() prefetch() Training Results and Speed Comparison We train our segmentation model for 100 epochs over two NVIDIA GTX1080 GPUs. The model attains about 91% pixel accuracy with just about 4 hours of training. Here are some comparisons from our tests: As you can see, using two GPUs instead of one gives us a speedup of almost 2x to attain the same level of accuracy. We hope this tutorial helped you understand how you can integrate Brain Builder into your AI workflow to build highly efficient data preparation and training pipelines. We welcome the feedback from the community in order to be able to improve Brain Builder further! . Leave a comment below with feedback, feature requests or any further questions you might have related to the tutorial and/or Brain Builder. You can sign up for Brain Builder Beta here You can find the code for this tutorial on . this link

Multi-GPU training with Brain Builder and TensorFlow

That’s a short warning to all Tensorflow users working with visual content. Short notice: don’t use any functions! tf.image.resize I was rewriting codebase of our — Let’s Enhance to make it ready for bigger and faster models and API we are working on. As we work with image generation (superresolution, deblurring, etc) we do rely on a typical image processing libraries like OpenCV and PIL. I always had suspicions that it makes sense to use Tensorflow image processing capabilities — in theory, they should be faster. So, I decided to stick to native Tensorflow image preprosessing and dataset building tools using dataset.map() to keep all operations in tensors all around my code. neural network image upscaling service The problem was quite awful — my new and shiny code for superresolution wasn’t able to reproduce not only any state-of-art networks but even my own code that I wrote 4 months ago. And the ugliest part was that results of superresolution itself were pretty good sometimes, the network was working, although not reaching target PSNR and having strange visual artifacts sometimes, like doubling of small lines. Let the debug begin What initially was looking as minor bug became 60 days of struggle and sleepless nights. My faulty logic was simple — there’s something wrong with network definition or training process. Data preprocessing is definitely fine, as I am having meaningful results and visual control over image processing in Tensorboard. I tweaked everything I was able to find, defined network using Keras, Slim, raw TF — nothing, looked for changes in TF 1.3->1.4->1.5 and different CUDA versions, paddings behaviors. I am ashamed even to tell you about my latest suspicions, which involved defects in GPU RAM and statics. I was tweaking perceptual losses and style losses looking for a reason. And each iteration took days to retrain until some meaningful result… Yesterday I found when looking into Tensorboard. It was almost subliminal feeling that something is wrong with the image. I disregarded network output and just overlayed target image and input image(that is a downscaled target image) in Photoshop. Here’s what I got. The Bug, Looks strange, somekind of displacement happening here. Totally againts any logic, this just can’t be true! My code is dead simple. Read image, crop image, resize image. All in Tensorflow. Anyway, RTFM. tf.image.resize_bicubic has a parameter — “align corners”. How in the world would you like to downscale image and not have corners aligned? You can! So there is a very weird behavior of this function known for a long time — . They can’t fix it as this can break lots of old code and pre-trained networks. read this thread Our function isn't even reflection equivariant. It would be lovely to fix this, but I'd be worried about breaking old models. tf.image.resize_area This code does actually displace your image by one pixel to the left and top. Thread suggests that even the interpolation is broken in TensorFlow. It’s 2018, people. Here are the actual downscaling results with TF. Stick to Scipy/OpenCV/numpy/PIL whatever you prefer for image processing. The second I changed it my network worked like a charm (next day actually, when I saw training results).

How Tensorflow’s tf.image.resize stole 60 days of my life

A look at two databases that have made claims to the Kubernetes native label: TiDB and DataStax Astra DB. The cloud computing revolution has inspired and benefitted from multiple interrelated trends. The availability of self-service, public cloud infrastructure has helped to drive the adoption of microservice architectures and DevOps practices, including automation and observability. The drive toward containerization and container orchestration has led to the widespread adoption of Kubernetes as an environment for managing cloud-native applications. But one of the lagging areas in this revolution has been data and data infrastructure. For too long, data has been something that has lived outside of Kubernetes, leading to a lot of extra effort and complexity for developers in deploying cloud-native applications. One oft-repeated axiom in the early years of Kubernetes was that it was not yet ready for stateful workloads. Thankfully, a major shift has been quietly underway and has reached a point of maturity. The transformation happened slowly initially, beginning with efforts to containerize existing databases. This worked relatively well in small databases that ran on a single compute node, or databases that had been designed in a cloud-native world, like Apache Cassandra and DynamoDB, but challenges remained. Over the past two to three years, a new generation of databases has emerged. These “Kubernetes native” databases have been designed from the ground up to run on this open-source orchestration system. Here, we’ll define the qualities that make a database Kubernetes native and the benefits of adopting a Kubernetes native database. To do that, we’ll look at two databases claiming the Kubernetes native label: TiDB and DataStax Astra DB. Kubernetes Native MySQL with TiDB First, let’s examine a database with a relational emphasis: (short for Titanium Database). TiDB is an open-source system built by PingCAP that provides a and a columnar database to support hybrid transactional and analytic processing (known as HTAP, for short). TiDB MySQL-compatible database As shown in Figure 1 below, TiDB has a microservice design. The TiDB query layer, TiKV MySQL databases, TiFlash columnar databases, Spark nodes, and metadata management are each deployed as scalable microservices in their clusters. This design separates compute-intensive work from storage-intensive work, as the query and database layers are independently scalable. One critical commitment the TiDB creators made was that the database only runs on Kubernetes. Is that enough to make it Kubernetes native? Let’s dig a bit deeper. First, TiDB is deployed and managed by a Kubernetes operator using (CRDs). The TiDB CRDs include the TiDBCluster, which enables you to specify the scaling and configuration of each microservice and how the database layer components use storage through Kubernetes Persistent Volumes. Additional CRDs are used to deploy monitoring tools and manage operational tasks like backup and restore. custom resources that interfaces with the default K8s scheduler to make more application-aware scheduling decisions. This emphasis on using existing Kubernetes capabilities where available is the mark of a Kubernetes native database. TiDB also has an optional scheduler extension Kubernetes Native Cassandra with DataStax Astra DB Now, look at another Kubernetes native database and note some similarities and differences. that was one of the first to claim to be cloud native, but what does it look like to deploy Cassandra in Kubernetes? is a highly scalable NoSQL database Cassandra that has been factored into microservices, as shown in Figure 2. is a version of Cassandra DataStax Astra DB Like TiDB, the database includes microservices concerned with query processing and data storage, as well as services for identity and access control, data repair, and backup/restore. The data services are particularly interesting in their use of , with Kubernetes Persistent Volumes used only for caching and object storage used for longer-term persistence. Separating compaction into its service enables this compute-intensive processing to happen in the background without affecting the performance of data services serving read and write traffic. storage Astra DB is offered as a managed service available in multiple cloud regions. Each region contains a data plane consisting of the services mentioned above, managed by a Kubernetes operator, as well as infrastructure services, including the stack for observability and for metadata management. Kube-Promethus etcd The data planes are managed by a control plane that can run in one or more clouds to manage customer accounts and databases and provision Kubernetes clusters in new regions. in which multiple user databases can share the same microservices and supporting infrastructure, lowering unit economics for smaller-scale users. One novel aspect of Astra DB is its multi-tenant architecture As users grow their applications, they can move to dedicated resources to achieve optimal performance at scale, all on a “ ” basis. pay-as-you-go Kubernetes Native Database Principles Based on our observations of TiDB and Astra DB, we can derive some ideas of what makes a database Kubernetes native. Many of these correspond to a list of principles for cloud-native data, which I described in an : earlier article First, a database broken into constituent microservices enables each service to be scaled independently. Some types of compute-intensive processing may even be scaled to zero for a true serverless solution, especially when combined with a multitenant design. Composable microservice architecture: Microservices composed of a Kubernetes native database should make maximum usage of Kubernetes APIs for managing the fundamental resources of cloud-native applications: compute resources such as , Kubernetes ingress and services for exposing network access to data and more. This includes leveraging capabilities already present in Kubernetes, such as etcd for metadata management, instead of bringing along components with duplicative functionality. Treat compute, network, and storage as commodities: StatefulSets and deployments for managing workloads, the Persistent Volume subsystem for storage Following common patterns for Kubernetes applications will yield multiple operational benefits, for example, . By default, Kubernetes itself sets a great example that databases should follow for how to be secure: using Kubernetes Secrets to distribute security credentials, only exposing ports as needed, and so on. Leverage Kubernetes best practices: exposing liveness and readiness checks on each microservice to help availability and exposing metrics via the Prometheus PromQL API for observability A Kubernetes native database should embody the Kubernetes principles of declarative management via operators and custom resources, rather than relying on legacy database management UIs and CLIs. When necessary, . The goal is a clean separation of data plane functionality (managing data) from control plane functionality (managing the database). Declarative management via operators: Kubernetes extension points, such as scheduler extensions, can be used to add application-specific behavior Databases and other data infrastructures that faithfully adopt these principles will yield benefits, including a , lower operational complexity resulting in faster time to market, and standards-compliant solutions meeting today’s high availability and security demands. high performance for optimal cost at all scales The Future of Kubernetes Native Data Infrastructure Much progress is still to be made, and it’s not limited to databases alone. Kubernetes native principles can be applied to other types of data infrastructure, including streaming, analytics, and machine learning. Kubernetes native solutions will continue to make strides in multicluster and multi-cloud deployments to scale globally and will adopt multitenancy and serverless principles for better cost optimization. Kubernetes itself has room for improvement in adding more flexibility to StatefulSets and support for multicluster federation. The key to continued progress is open collaboration. The is a highly active group of data geeks bringing together builders of data-intensive applications and the infrastructure that supports them. Data on Kubernetes Community Join us to talk about ideas like developing reusable operators that can manage multiple databases or defining a common set of CRDs for concepts like backup/restore and data loading. Together we’ll continue to push the horizon of cloud computing for the benefit of all. Learn more about Kassandra native databases and more at the digital summit on March 14, 2023. Cassandra Forward This article is based on Chapter 7, “The Kubernetes Native Database,” from the O’Reilly book “ Managing Cloud Native Data on Kubernetes ” by Jeff Carpenter and Patrick McFadin. [ By Jeff Carpenter, DataStax Jeff Carpenter has worked as a software engineer and architect in multiple industries and as a developer advocate at DataStax, helping engineers succeed with Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems, including Stargate and K8ssandra. He is co-author of the O’Reilly books "Cassandra: The Definitive Guide" and "Managing Cloud Native Data on Kubernetes."

Kubernetes-Native Database: TiDB Vs. DataStax Astra DB

Base de datos nativa de Kubernetes: TiDB vs. Base de datos DataStax Astra

Base de données native Kubernetes : TiDB Vs. DataStax Astra DB

कुबेरनेट्स-नेटिव डेटाबेस: TiDB बनाम। डेटास्टैक्स एस्ट्रा डीबी

Kubernetes ネイティブ データベース: TiDB Vs. DataStax アストラ DB

Banco de dados nativo do Kubernetes: TiDB vs. Data Stax Astra DB

Cơ sở dữ liệu gốc Kubernetes: TiDB Vs. DataStax Astra DB

Kubernetes 原生数据库：TiDB 对比。 DataStax Astra 数据库

is a big data and AI start-up that uses geospatial data to monitor environmental parameters. Our goal is to become the . Blue Sky Analytics Bloomberg of Environmental data for environmental monitoring, ESG (environment, social, and governance) due-diligence and climate risk assessment In other words, we aim to be the go-to source for all environmental data to drive sustainable decision making. We believe that safeguarding the global market against climate risk and other climate-change-induced threats requires temporally and spatially continuous high-resolution datasets that inform us of pollution levels, water quality, emissions, fires, changes in soil composition, etc. We asked ourselves, “what if there was a platform that presented environmental data the way we have access to financial data?” Sounds like an impossible dream, right? That’s why we need a technology stack that’s built to power our mission and a team too young to not know what’s impossible (our average age is 24.8). …and, our air quality and farm fire crisis solutions have won us (Healthy Cities Solutions) and the , so there’s power in believing in and pursuing what others may deem impossible. MIT Solve ‘Space Oscar’ from European Space Agency In this post, we’ll outline our technology — which handles historical as well as continuous data — and talk a bit about we’ve architected our platform to give spatial-temporal context to pre-existing data, what we’ve learned along the way, and what’s next. In short, you’ll get a glimpse of how we are making sense of terabytes of raw data and turning them into environmental insights. Our goal? You’re inspired to follow your dreams and build your own “impossible” projects. how What we do (and why we do it) We started this journey by creating a high-resolution database for monitoring air pollution in India, called , and we aim to cover surface water monitoring in 2020, land composition and urban informatics in 2021, and other environmental parameters in the coming years. BreeZo Currently, despite the alarming air pollution levels in India, monitoring is inadequate. There are , which are unequally distributed, and, as a result, large parts of the country go unmonitored. around 200 government monitors across India Our products aim to bridge this data gap by using satellite data and public monitors to provide a more comprehensive picture of the air pollution crisis.We make our data available to customers via accessible APIs and platforms; our approach centers on ensuring public access (democratising high-resolution environmental information) and facilitating policy-making and enterprise decisions (allowing corporations and government entities to understand current environmental situations as they craft initiatives). But, what we can’t monitor, we can’t solve. Climate change is a global challenge that we must come together to fix; it poses a huge risk and will have grave effects on public health, ecology and the financial system. Thus, it’s vital for governments, industries, and scientists to collaborate and come up with innovative solutions — leveraging AI, big data, and space technology — to prevent and mitigate negative outcomes. That’s where we come in: we create a platform that brings the much-needed high-quality data individuals need to make data-driven sustainable decision making, such as ways to transition from carbon-intensive economies to low-carbon approaches. The Tech Stack: How we do it and why our time-series database matters to us In order to create a large-scale environmental monitoring platform from geospatial data, we needed a database (specifically a time-series database (TSDB)) that could handle enormous quantities of spatial time-series data. We looked at various options, and landed on . Timescale DB The most common way to solve our “problem” is to use NoSQL databases, which can be treated as streams. However, that would mean that we would need to write our logic to implement the spatial and temporal context into our data. As we monitor environmental parameters using satellite data, the ability to handle and fill null values is especially critical. We had a lot of hacked together ways to query data with null filling and deal with data irregularities, using multiple cron jobs for each source and code at the application layer to do spatial queries. Essentially, we were spending a lot of time managing our time-series data and not enough time analysing it. For example: we wrote one script for null-filling, which was very slow to begin with — and we had to write it in bash, separate from our main app, to ensure it was performant enough and didn’t stop the main thread. So, NoSQL databases weren’t enough. But, what could we use? We looked at options like Amazon Timestream, but found that, while it works for IoT data, it didn’t work for our scenario: we handle not just IoT, but satellite data as well. For us, TimescaleDB proved to be particularly unique among TSDBs, allowing us to power spatial queries on time-series data and having its roots in Postgres. Some important features for our adoption were continuous aggregation, gap-filling null values, time bucketing, and, most importantly SQL support. Timescale has brought a change in our development paradigm, as it serializes everything in a timely order, and we can store, query, and analyze our temporal and spatial data using SQL (directly from the database). Example We have a database table where we store all the pixels at 10 kilometer grid for all of India, and we run spatial queries for individual states (taking into account cloud cover, which act as NULL values, and averaging them over that time). India is 3.3. million square kilometers, which comes out to be 33,000 rows of daily emission (with a 10 kilometer grid). Now, we can run a spatial join and average it over time. 3.3*1000000/10*10 = 33,000 Measurements (Raw satellite data), from multiple satellite sources downsampled to common denominator -- Measurements table
CREATE TABLE measurements (
	u_wind NUMERIC( , ),
	v_wind NUMERIC( , ),
	albedo NUMERIC( , ),
	aod469 NUMERIC( , ),
	aod550 NUMERIC( , ),
	aod670 NUMERIC( , ),
	aod865 NUMERIC( , ),
	aod1240 NUMERIC( , ),
	aod_s5p NUMERIC( , ),
	blh NUMERIC( , ),
	temperature NUMERIC( , ),
	recorded_at DATE,
	grid GEOMETRY
); 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 10 4 Maps of districts, states & region -- Enum shape types 
CREATE TYPE shapes_type AS ENUM ( , , , ); 
-- Create shape table 
CREATE TABLE shapes ( id UUID NO-T NULL, name VARCHAR( ), TYPE shapes_type, shape GEOMETRY ); for 'Country' 'State' 'District' 'Region' 255 Sample problem or analysis So, how would someone actually apply or use this information? Let’s say we’re a District Magistrate who wants to set weekly KPIs for emissions in an area where farmers burn crops on a daily basis. To get a baseline to inform our decision, we’d want to know the weekly emissions average emissions in each district over the last 3 months, focusing on those in our state. This is a pretty complex (and powerful) geospatial query, but it’s just one example of the type of analysis we do on a daily basis. Crunching 33,000, 3 30 data points SELECT
	name,
	-- Aggregates all the points one district
	json_agg(json_build_object( , datetime, , u_wind, , v_wind, , albedo, , aod469, , aod550, ,
 aod670, , aod865, , aod1240, , aod_s5p, ,
 temperature, , blh)) AS pollutants
FROM (
	-- Selects all the districts Punjab
	SELECT
		name,
		shape
	FROM WHERE
		TYPE = -- Selects all the state
		AND ST_WITHIN(shape, (
				SELECT
					shape FROM WHERE
					name = ))) AS Districts
	LEFT JOIN (
		-- Select all the points last week daily average and within Punjab
		SELECT
			time_bucket_gapfill ( , recorded_at, NOW() - interval ,
 NOW()) AS datetime, grid, avg(u_wind) AS u_wind, avg(v_wind) AS v_wind,
 avg(albedo) AS albedo, avg(aod469) AS aod469, avg(aod550) AS aod550,
 avg(aod670) AS aod670, avg(aod865) AS aod865, avg(aod1240) AS aod1240, 
avg(aod_s5p) AS aod_s5p, avg(temperature) AS temperature, avg(blh) AS blh
		FROM WHERE
			-- Get the points within week
			recorded_at < NOW()
			AND recorded_at > NOW() - interval -- Get the point only Punjab
			AND ST_WITHIN(grid, (
					SELECT
						shape FROM WHERE
						name = ))
		GROUP BY
			grid, datetime) AS Records
		-- Join the points based on geometry
		ON ST_Within(grid, Districts.shape)
		-- group them together
	GROUP BY
		name in 'datetime' 'u_wind' 'v_wind' 'albedo' 'aod469' 'aod550' 'aod670' 'aod865' 'aod1240' 'aod_s5p' 'temperature' 'blh' in "shapes" 'District' `Districts` in `Punjab` "shapes" 'Punjab' in with '1 day' '1 week' "measurements" 1 '1 week' in "shapes" 'Punjab' finally As mentioned previously, we collect an enormous amount of data from various sources, namely 1000+ ground monitors across India and satellite missions, like . To power our platform, we use TimescaleDB built-in functions to build chunks that allow us to store historical data in an easily accessible, scalable way. For example, we might have 12 values for March 1, 2014, but we only need for that day for historical analysis. Sentinel 5P one value When we started: We had 100s of rasters for all the satellite images we were ingesting. As a result, we wrote a custom Python script for every analysis, which was time-consuming and tedious. Post the TimescaleDB transition: We store all the ingested data in the database and simply extract raster values. We use geospatial JOINs, utilising TimescaleDB and PostGIS to combine geospatial data and time-series data and perform time-slicing. This allows us to gain insights and store them, make complex spatial-temporal queries, and carry out deeper analysis. Simply put, no more reading 100s of raster files or writing custom scripts for each analysis; we just dump into our database, and use SQL to analyze and find solutions. Compression of data and storage complexities are areas we no longer need to stress about. Moreover, we don’t have to worry about storing data in a time-series fashion, or optimization on that time-series data. And, as mentioned earlier, we’re able to efficiently handle null filling, data aggregation, and time slicing, making our development process significantly less complicated. All of this is to say that, if you’re building a data-intensive solution like we are or combining various types of data, TimescaleDB is a great place to start working with time-series data. It’s helped us expedite our mission of commanding the environmental data space, and the possibilities are endless: you can pace it according to your needs, as well as infinitely scale as you grow. Previously published at https://medium.com/blue-sky-thinking/global-environmental-data-stack-for-climate-action-a5c3ed494091

Stories, News & Mentions about DATASTEMA

Datastema’s Stories on HackerNoon