Facet Data

facetdata

Data powers machine learning algorithms and scikit-learn or offers high quality datasets that are widely used by researchers, practitioners and enthusiasts. Scikit-learn (sklearn)  is a Python module for machine learning built on top of SciPy. It is unique due to its wide range of algorithms, ease of use and integration with other Python libraries. sklearn What are “Sklearn Datasets”? Sklearn datasets are included as part of the scikit-learn ( ) library, so they come pre-installed with the library. Due to this, you can easily access and load these datasets, without having to download them separately. sklearn To use a specific dataset, you can simply import it from sklearn.datasets module and call the appropriate function to load the data into your program. These datasets are usually pre-processed and ready to use, which saves time and effort for data practitioners who need to experiment with different machine learning models and algorithms. Complete List of Datasets in the Sklearn Library Iris Diabetes Digits Linnerud Wine Breast Cancer Wisconsin Boston Housing Olivetti Faces California Housing MNIST Fashion-MNIST make_classification make_regression make_blobs make_moons and make_circles Make_sparse_coded_signal Pre-Installed(Toy) Sklearn Datasets 1. Iris This dataset includes measurements of the sepal length, sepal width, petal length and petal width of 150 iris flowers, which belong to 3 different species: setosa, versicolor and virginica. The iris dataset has 150 rows and 5 columns, which are stored as a dataframe, including a column for the species of each flower. The variables include: Sepal.Length - The sepal.length represents the length of the sepal in centimetres. Sepal.Width - The sepal.width represents the width of the sepal in centimetres. Petal.Length - The petal.length represents the length of the petal in centimetres. Species - The species variable represents the species of the iris flower, with three possible values: setosa, versicolor and virginica. You can load the iris dataset directly from sklearn using the function from the sklearn.datasets module. load_iris # To install sklearn
pip install scikit-learn

# To import sklearn
from sklearn.datasets import load_iris

# Load the iris dataset
iris = load_iris()

# Print the dataset description
print(iris.describe()) Code for loading the Iris dataset using sklearn. Retrieved from on 27/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html 2. Diabetes This sklearn dataset contains information on 442 patients with diabetes, including demographic and clinical measurements: Age Sex Body mass index (BMI) Average blood pressure Six blood serum measurements (e.g. total cholesterol, low-density lipoprotein (LDL) cholesterol, high-density lipoprotein (HDL) cholesterol). A quantitative measure of diabetes disease progression (HbA1c). The Diabetes dataset can be loaded using the function from the sklearn.datasets module. load_diabetes() from sklearn.datasets import load_diabetes

# Load the diabetes dataset
diabetes = load_diabetes()

# Print some information about the dataset
print(diabetes.describe()) Code for loading the Diabetes dataset using sklearn. Retrieved from on 28/3/2023. https://scikit-learn.org/stable/datasets/toy_dataset.html#diabetes-dataset 3. Digits This sklearn dataset is a collection of hand-written digits from 0 to 9, stored as grayscale images. It contains a total of 1797 samples, with each sample is a 2D array of shape (8,8). There are 64 variables (or features) in the digits sklearn dataset, corresponding to the 64 pixels in each digit image. The Digits dataset can be loaded using the function from the sklearn.datasets module. load_digits() from sklearn.datasets import load_digits

# Load the digits dataset
digits = load_digits()

# Print the features and target data
print(digits.data)
print(digits.target) Code for loading the Digits dataset using sklearn. Retrieved from on 29/3/2023. https://scikit-learn.org/stable/datasets/toy_dataset.html#optical-recognition-of-handwritten-digits-dataset 4. Linnerud The Linnerud dataset contains physical and physiological measurements of 20 professional athletes. The dataset includes the following variables: Three physical exercise variables - chin-ups, sit-ups, and jumping jacks. Three physiological measurement variables - pulse, systolic blood pressure, and diastolic blood pressure. To load the Linnerud dataset in Python using sklearn: from sklearn.datasets import load_linnerud
linnerud = load_linnerud() Code for loading the linnerud dataset using sklearn. Retrieved from on 27/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_linnerud.html#sklearn.datasets.load_linnerud 5. Wine This sklearn dataset contains the results of chemical analyses of wines grown in a specific area of Italy, to classify the wines into their correct varieties. Some of the variables in the dataset: Alcohol Malic acid Ash Alkalinity of ash Magnesium Total phenols Flavanoids The Wine dataset can be loaded using the function from the sklearn.datasets module. load_wine() from sklearn.datasets import load_wine

# Load the Wine dataset
wine_data = load_wine()

# Access the features and targets of the dataset
X = wine_data.data  # Features
y = wine_data.target  # Targets

# Access the feature names and target names of the dataset
feature_names = wine_data.feature_names
target_names = wine_data.target_names Code for loading the Wine Quality dataset using sklearn. Retrieved from on 28/3/2023. https://scikit-learn.org/stable/datasets/toy_dataset.html#wine-recognition-dataset 6. Breast Cancer Wisconsin Dataset This sklearn dataset consists of information about breast cancer tumours and was initially created by Dr. William H. Wolberg. The dataset was created to assist researchers and machine learning practitioners in classifying tumours as either malignant(cancerous) or benign (non-cancerous). Some of the variables included in this dataset: ID number Diagnosis (M = malignant, B = benign). Radius (the mean of distances from the centre to points on the perimeter). Texture (the standard deviation of gray-scale values). Perimeter Area Smoothness (the local variation in radius lengths). Compactness (the perimeter^2 / area - 1.0). Concavity (the severity of concave portions of the contour). Concave points (the number of concave portions of the contour). Symmetry Fractal dimension ("coastline approximation" - 1). You can load the Breast Cancer Wisconsin dataset directly from sklearn using the function from the sklearn.datasets module. load_breast_cancer from sklearn.datasets import load_breast_cancer

# Load the Breast Cancer Wisconsin dataset
cancer = load_breast_cancer()

# Print the dataset description
print(cancer.describe()) Code for loading the Breast Cancer Wisconsin dataset using sklearn. Retrieved from on 28/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html Real World Sklearn Datasets Real world sklearn datasets are based on real-world problems, commonly used to practice and experiment with machine learning algorithms and techniques using the sklearn library in Python. 7. Boston Housing The Boston Housing dataset consists of information on housing in the area of Boston, Massachusetts. It has about 506 rows and 14 columns of data. Some of the variables in the dataset include: CRIM - Per capita crime rate by town. ZN - The proportion of residential land zoned for lots over 25,000 sq.ft. INDUS - The proportion of non-retail business acres per town. CHAS - Charles River dummy variable (= 1 if tract bounds river; 0 otherwise). NOX - The nitric oxide concentration (parts per 10 million). RM - The average number of rooms per dwelling. AGE - The proportion of owner-occupied units built prior to 1940. DIS - The weighted distances to five Boston employment centres. RAD - The Index of accessibility to radial highways. TAX - The full-value property-tax rate per $10,000. PTRATIO - The pupil-teacher ratio by town. B - 1000(Bk - 0.63)^2 where -Bk is the proportion of blacks by town. LSTAT - The percentage lower status of the population. MEDV - The median value of owner-occupied homes in $1000's. You can load the Boston Housing dataset directly from scikit-learn using the function from the sklearn.datasets module. load_boston from sklearn.datasets import load_boston

# Load the Boston Housing dataset
boston = load_boston()

# Print the dataset description
print(boston.describe()) Code for loading the Boston Housing dataset using sklearn. Retrieved from on 29/3/2023. https://scikit-learn.org/0.15/modules/generated/sklearn.datasets.load_boston.html 8. Olivetti Faces The Olivetti Faces dataset is a collection of grayscale images of human faces taken between April 1992 and April 1994 at AT&T Laboratories. It contains 400 images of 10 individuals, with each individual having 40 images shot at different angles and different lighting conditions. You can load the Olivetti Faces dataset in sklearn by using the function from the datasets module. fetch_olivetti_faces from sklearn.datasets import fetch_olivetti_faces

# Load the dataset
faces = fetch_olivetti_faces()

# Get the data and target labels
X = faces.data
y = faces.target Code for loading the Olivetti Faces dataset using sklearn. Retrieved from on 29/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_olivetti_faces.html 9. California Housing This sklearn dataset contains information on median house values, as well as attributes for census tracts in California. It also includes 20,640 instances and 8 features. Some of the variables in the dataset: MedInc - The median income in block. HouseAge - The median age of houses in block. AveRooms - The average number of rooms per household. AveBedrms - The average number of bedrooms per household. Population - The block population. AveOccup - The average household occupancy. Latitude - The latitude of the block in decimal degrees. Longitude - The longitude of the block in decimal degrees. You can load the California Housing dataset using the function from sklearn. fetch_california_housing from sklearn.datasets import fetch_california_housing

# Load the dataset
california_housing = fetch_california_housing()

# Get the features and target variable
X = california_housing.data
y = california_housing.target Code for loading the California Housing dataset using sklearn. Retrieved from on 29/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html 10. MNIST The MNIST dataset is popular and widely used in the fields of machine learning and computer vision. It consists of 70,000 grayscale images of handwritten digits 0–9, with 60,000 images for training and 10,000 for testing. Each image is 28x28 pixels in size and has a corresponding label denoting which digits it represents. You can load the MNIST dataset from sklearn using the following code: from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784') Note: The MNIST dataset is a subset of the Digits dataset. Code for loading the MNIST dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_openml.html#sklearn.datasets.fetch_openml 11. Fashion-MNIST The Fashion MNIST dataset was created by Zalando Research as a replacement for the original MNIST dataset. The Fashion MNIST dataset consists of 70,000 grayscale images(training set of 60,000 and a test set of 10,000) of clothing items. The images are 28x28 pixels in size and represent 10 different classes of clothing items, including ­T-shirts/tops, trousers, pullovers, dresses, coats, sandals, shirts, sneakers, bags, and ankle boots. It is similar to the original MNIST dataset, but with more challenging classification tasks due to the greater complexity and variety of the clothing items. You can load this sklearn dataset using the fetch_openml function. from sklearn.datasets import fetch_openml

fmnist = fetch_openml(name='Fashion-MNIST') Code for loading the Fashion MNIST dataset using sklearn. Retrieved from__ __ on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_openml.html#sklearn.datasets.fetch_openml Generated Sklearn Datasets Generated sklearn datasets are synthetic datasets, generated using the sklearn library in Python. They are used for testing, benchmarking and developing machine learning algorithms/models. 12. make_classification This function generates a random n-class classification dataset with a specified number of samples, features, and informative features. Here's an example code to generate this sklearn dataset with 100 samples, 5 features, and 3 classes: from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=5, n_informative=3, n_classes=3, random_state=42) This code generates a dataset with 100 samples and 5 features, with 3 classes and 3 informative features. The remaining features will be redundant or noise. Code for loading the make_classification dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html#sklearn.datasets.make_classification 13. make_regression This function generates a random regression dataset with a specified number of samples, features, and noise. Here's an example code to generate this sklearn dataset with 100 samples, 5 features, and noise level of 0.1: from sklearn.datasets import make_regression

X, y = make_regression(n_samples=100, n_features=5, noise=0.1, random_state=42) This code generates a dataset with 100 samples and 5 features, with a noise level of 0.1. The target variable y will be a continuous variable. Code for loading the make_regression dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_regression.html#sklearn.datasets.make_regression 14. make_blobs This function generates a random dataset with a specified number of samples and clusters. Here's an example code to generate this sklearn dataset with 100 samples and 3 clusters: from sklearn.datasets import make_blobs
X, y = make_blobs(n_samples=100, centers=3, random_state=42) This code generates a dataset with 100 samples and 2 features (x and y coordinates), with 3 clusters centred at random locations, and with no noise. Code for loading the make_blobs dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_blobs.html#sklearn.datasets.make_blobs 15. and make_moons make_circles These functions generate datasets with non-linear boundaries that are useful for testing non-linear classification algorithms. Here's an example code for loading the make_moons dataset: from sklearn.datasets import make_moons X, y = make_moons(n_samples=1000, noise=0.2, random_state=42) This code generates a dataset with 1000 samples and 2 features (x and y coordinates) with a non-linear boundary between the two classes, and with 0.2 standard deviations of Gaussian noise added to the data. Code for loading the make_moons dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html#sklearn.datasets.make_moons Here's an example code to generate and load the make_circles dataset: from sklearn.datasets import make_circles

X, y = make_circles(n_samples=1000, noise=0.05, random_state=42) Code for loading the make_circles dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_circles.html#sklearn.datasets.make_circles 16. make_sparse_coded_signal This function generates a sparse coded signal dataset that is useful for testing compressive sensing algorithms. Here's an example code for loading this sklearn dataset: from sklearn.datasets import make_sparse_coded_signal

X, y, w = make_sparse_coded_signal(n_samples=100, n_components=10, n_features=50, n_nonzero_coefs=3, random_state=42) This code generates a sparse coded signal dataset with 100 samples, 50 features, and 10 atoms. Code for loading the make_sparse_coded_signal dataset using sklearn. Retrieved from on 30/3/2023. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_sparse_coded_signal.html#sklearn-datasets-make-sparse-coded-signal Common Use Cases for Sklearn Datasets Pre-Installed(Toy) Sklearn Datasets - This sklearn dataset is commonly used for classification tasks and is used as a benchmark dataset for testing classification algorithms. Iris - This dataset contains medical information about patients with diabetes and is used for classification and regression tasks in healthcare analytics. Diabetes - This sklearn dataset contains images of handwritten digits and is commonly used for image classification and pattern recognition tasks. Digits - This dataset contains physical fitness and medical data of 20 athletes and is commonly used for multivariate regression analysis. Linnerud - This sklearn dataset contains chemical analysis of wines and is commonly used for classification and clustering tasks. Wine - This dataset contains medical information about breast cancer patients and is commonly used for classification tasks in healthcare analytics. Breast Cancer Wisconsin Real World Sklearn Datasets - This sklearn dataset contains information about housing in Boston and is commonly used for regression tasks. Boston Housing - This dataset contains grayscale images of faces and is commonly used for image classification and facial recognition tasks. Olivetti Faces - This sklearn dataset contains information about housing in California and is commonly used for regression tasks. California Housing - This dataset contains images of handwritten digits and is commonly used for image classification and pattern recognition tasks. MNIST - This sklearn dataset contains images of clothing items and is commonly used for image classification and pattern recognition tasks. Fashion-MNIST Generated Sklearn Datasets - This dataset is a randomly generated dataset for binary and multiclass classification tasks. make_classification - This dataset is a randomly generated dataset for regression tasks. make_regression - This sklearn dataset is a randomly generated dataset for clustering tasks. make_blobs and - These datasets are randomly generated datasets for classification tasks and are commonly used for testing nonlinear classifiers. make_moons make_circles - This dataset is a randomly generated dataset for sparse coding tasks in signal processing. make_sparse_coded_signal Final Thoughts Sklearn datasets provide a convenient way for developers and researchers to test and evaluate machine learning models without having to manually collect and preprocess data. They are also available for anyone to download and use freely. The lead image of this article was generated via model using the prompt 'iris dataset'. HackerNoon's AI Stable Diffusion More Dataset Listicles: Excel Datasets Keras Datasets R Datasets

16 Best Sklearn Datasets for Building Machine Learning Models

বিল্ডিং মেশিন লার্নিং মডেলের জন্য 16 সেরা Sklearn ডেটাসেট

Die 16 besten Sklearn-Datensätze zum Erstellen von Modellen für maschinelles Lernen

Los 16 mejores conjuntos de datos de Sklearn para crear modelos de aprendizaje automático

16 meilleurs ensembles de données Sklearn pour créer des modèles d'apprentissage automatique

मशीन लर्निंग मॉडल बनाने के लिए 16 सर्वश्रेष्ठ स्केलेरन डेटासेट

機械学習モデルの構築に最適な 16 の Sklearn データセット

기계 학습 모델 구축을 위한 16가지 최고의 Sklearn 데이터 세트

16 melhores conjuntos de dados do Sklearn para construir modelos de aprendizado de máquina

Makine Öğrenimi Modelleri Oluşturmak için En İyi 16 Sklearn Veri Kümesi

16 bộ dữ liệu Sklearn tốt nhất để xây dựng mô hình học máy

用于构建机器学习模型的 16 个最佳 Sklearn 数据集

Authentication is a very crucial aspect of your web application. If you are offering a service or selling some product to the user, you should keep track of the user for future reference. All this can be only possible if you have an authentication system in place. But using the right tool for the job is also very necessary. In past, email and password-based authentication were most popular and widely used. But with time, a new concept called OAuth was introduced when big techs were popping out with a huge user base. Where you as an owner of the site trust the OAuth provider to authenticate a user, in return, the OAuth provider gives you details of the user. OAuth-based authentication is simple for the user. They only have to maintain one account with an OAuth provider and use this account to log in to all other websites. But there is always trust involved in the OAuth process. After some time, passwordless authentication came into the picture. In this process, when you enter the username or your email, they mail you a link. If you paste the link into the browser, you are automatically authenticated and logged in. The most common example of password-less authentication is when you click on the send magic link to log in to . hackernoon.com With the advancement of Artificial Intelligence (AI) and machine learning (ML), facial recognition techniques are gaining huge popularity. With time, as the datasets will grow bigger, the accuracy of an AI model will also increase. Nowadays, we can also use facial recognition techniques, to authenticate users in our web application. In this article, we are building a simple application, to demonstrate how to authenticate a user using facial recognition. In this process, we are going to use FaceIO APIs. Why we need facial recognition-based authentication The need for facial recognition is multifold. I am trying to put up some points here in a concise way. Make sure to read the article to the end to get a complete conceptual understanding and a detailed implementation walkthrough. : Facial authentication method is very fast than the traditional means of authentication. You just have to click on a button to start the authentication process and within a millisecond, it is done. In the traditional email password-based methods, you have to add your details line by line. Sometimes after successful log-in, you are greeted with a captcha. How irritating it is! Faster than traditional methods : The only requirement of the Facial authentication technique is a camera. All smartphones nowadays have a camera by default. All desktops also have some sort of webcams present. Therefore, users don't need any specialized hardware to use this service. Don't require specialized hardware : The most important feature of facial authentication is that it can prevent impersonation. On social platforms, many people create fake accounts by impersonating someone. This can be very risky if the fake account holder commits some type of digital crime. With the help of facial recognition, social platforms can recognize if the account someone was trying to access, actually belongs to them. Reduce impersonation on social platforms : Bots and automated scripts are introduced to help people get rid of repetitive tasks. But people also used them in a different way to spam others. Daily you come across many bots and automated scripts in your digital life that you don't even notice or realize. To prevent this, some websites use captcha. Using facial recognition, this problem can also be solved as bots have no face to authenticate 😄. Reduce bots and automated scripts : Privacy is a very sensitive topic for all of us. We all become a little bit concerned when someone asks you to authenticate using your face data. But as we are using FaceIO in this tutorial, the authentication process is purely end-to-end encrypted. In the backend, they store only the hash of your facial features. They are completely GDPR and CCPA-compliant. So, you can trust them to store your data safely. Privacy-focused Make a facial authentication project Now we are going to make a facial authentication web application. This project includes all the bits and pieces and all you need to know about how to implement face recognition-based authentication in your web application. I am explaining the process step-by-step. You can get a full source code . Make sure to obtain a free to follow along. here API key Installing the required dependency Create a blank directory and inside it, make an file. You can also add a separate CSS file, but for the sake of simplicity, I will keep it to the bare minimum. index.html If you are using VSCode for development, you can use the live server to serve your static files. Inside your , add this basic HTML markup, to begin with. index.html <!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Document</title>
  </head>
  <body></body>
</html> FaceIO provides a very handy javascript library to interact with their AI models. This makes our life so easier that we can implement the facial recognition feature using only a few lines of code. To add the FaceIO javascript library, we use their CDN(content delivery network) inside the body tag of our HTML document. <body>
	<script src="https://cdn.faceio.net/fio.js"></script>
</body> Now, create a file and link the file inside the body tag after FaceIO CDN. index.js <body>
	<script src="https://cdn.faceio.net/fio.js"></script>
	<script src="./index.js"></script>
</body> Now let's create 2 helper functions to ease our development process. One is for enrolling a user (like the sign-up feature) and the other is for authentication (like the log-in feature). Enrolling a user Enrolling a user is very simple because of the javascript library FaceIO provides. Inside the HTML markup add a button with . We access this button inside our javascript file using the method. id="enroll" getElementbyID Now initialize the FaceIO object inside . You have to add the public ID of your FaceIO project. You can get the public ID listed in your project dashboard. index.js const faceio = new faceIO("<Your Public ID here"); Let's add an event listener in the enroll button. When someone clicks the button, we execute the enroll method of the faceIO object. This enrollment method takes a variety of optional parameters. is the local language of the user. locale corresponds to the number of seconds to wait for the user to grant access to the camera. permissions timeout is the number of seconds to wait for the user to accept the terms and conditions of FaceIO. termsTimeout is the total number of seconds to wait while trying to recognize a face. idleTimeout is the number of seconds to wait to receive processed facial data from the FaceIO node. replyTimeout is a boolean that represents if users give consent to scan their faces. If you have already taken consent from the user then you can set the value to . userConcent true :  Inside the enroll function you can add a payload of your choice. Payload should be a key value object. You can use this payload feature to attach your email address or any other information related to the user. payload In our case, the enroll function looks like this. enroll.addEventListener("click", async () => {
  let response = await faceio.enroll({
    locale: "auto",
    payload: {
      email: "example@gmail.com",
      pin: "12345",
    },
  });

  console.log(` Unique Facial ID: ${response.facialId}
  Enrollment Date: ${response.timestamp}
  Gender: ${response.details.gender}
  Age Approximation: ${response.details.age}`);
}); When you run this function, A popup appears in front of the user. This popup contains terms and conditions. If the user accepts the terms and conditions, it asks for camera access. If the user grants the camera access, FaceIO will scan the face. FaceIO model looks for unique facial features that distinguish the user from others. Once completed, You have to add a PIN which is attached to your facial data. This PIN is very important for the user and keeps it in a safer place. When all these steps are completed, FaceIO returns an object to the user. This object contains a user face ID which is a universally unique identifier, gender, and age. Gender and age are not very accurate, as they are predicted by an AI model. userInfo You can use store this faceID in your backend. When a user wants to log in, you can match this faceID to authenticate the user. During the workflow, if any error occurs, FaceIO has an extensive list of error messages. If the user doesn't allow the camera access then the error is thrown. fioErrCode.PERMISSION_REFUSED If the user doesn't accept the terms and conditions popup then the error is thrown by the server. fioErrCode.TERMS_NOT_ACCEPTED Authenticating a user To initiate the authentication flow, add a button in your HTML markup with . Access this button inside with the help of the method. id="authenticate" index.js getElementbyID Now when the user presses this button, we initiate the authentication flow. The authentication function is very simple. Authenticate function take , , , and parameter like the previous function. The code looks something like this. permissionTimeout idleTimeout replyTimeout locale enroll() authenticate.addEventListener("click", async () => {
  let response = await faceio.authenticate({
    locale: "auto",
  });

  console.log(` Unique Facial ID: ${response.facialId}
      PayLoad: ${response.payload}
      `);
}); When the user presses the authentication button, a similar screen popup like in the case of enrolling function. It takes your camera access and scans your face. Once scanned, it asks for the PIN that you entered during the enrollment time. If you provide the correct pin, FaceIO returned the FaceData and the payload you specified in the enrollment process. You can also double-check the authentication flow by matching the faceID in your server. Now our authentication flow is completed. You can see it is easier than implementing an email password auth flow. All the heavy lifting is done by the FaceIO server and its AI model, as a developer, you only have to add the application logic to modify the authentication flow and its experience. Privacy features FaceIO has a robust privacy protection system. Let me list down some of them. : FaceIO service is completely GDPR and CCPA compliant. GDPR stands for General Data Protection Regulations. It is adopted in 2018 and it requires all businesses to protect the personal data and privacy of the user. It is GDPR and CCPA compliant CCPA stands for California Consumer Protection Act gives more control to the user over their data. If you are operating in those regions, you don't have to worry. : FaceIO only store hashes of your facial features. It doesn't store any plain data and stores as minimum information as possible. The client-side library and widgets don't handle any biometrics data. All the process is done at the backend. It stores only hashes FAQ Is FaceIO browser agnostic? FaceIO is completely browser agnostic. It can run any browser including chrome, firefox, and safari. As all the processing is done at the server, FaceIO needs only your camera access to run. If you are using some privacy browser by disabling javascript, make sure to enable them. As FaceIO needs to use javascript to talk with the server. Can't we build our model from scratch? Yes, you can build your model from scratch and implement the facial authentication feature in your web app. But for this, you should have a deep understanding of artificial intelligence and machine learning processes. As most web developers are not from this background, it is a smart move to use others' services to implement a feature rather than reinvent the wheel. There are other solutions are present in the market that you can explore. One is AWS Rekognition. This service is offered by AWS and works very similarly to FaceIO. The interesting thing is that you can opt for AWS Rekognition inside your FaceIO dashboard. Yes, it is possible. You can now choose what you want and go for it. Hide the public ID As we can see in the coding part, you have to give your public ID during the initiation of the FaceIO object. If you want, you can hide it using environment variables. If you are using a front-end framework or a build system, you can dynamically add these values at the build time. If you are using NextJS, you can use file to save your sensitive credentials. .env.local If you are not using any framework, you can use Vite. Vite supports environment variables. It builds and compresses your vanilla js project for faster deployment. Just create a file, put all your credentials there, and you are done. Don't forget to put your file inside your list. .env .env .gitignore Conclusion If you read the article to this point, I think you like this article. Please share it with your peers. If you want to give any feedback, I am available on Twitter as . Keep reading, keep learning. hrishikshpathak

How to Authenticate a User via Face Recognition in Your Web Application

Cómo autenticar a un usuario a través del reconocimiento facial en su aplicación web

Comment authentifier un utilisateur via la reconnaissance faciale dans votre application Web

अपने वेब एप्लिकेशन में फेस रिकग्निशन के जरिए किसी यूजर को ऑथेंटिकेट कैसे करें

Web アプリケーションで顔認識を介してユーザーを認証する方法

Como autenticar um usuário por meio de reconhecimento facial em seu aplicativo da Web

Cách xác thực người dùng qua nhận dạng khuôn mặt trong ứng dụng web của bạn

如何在 Web 应用程序中通过人脸识别对用户进行身份验证

For the past few years now, the increasing and the exponential improvement of and have invited media groups to rethink the way they do business. . Many of these disruptions have been centered around the mountains of data media groups have access to, and what Artificial Intelligence (AI) (and machine learning more specifically) could do with it. Indeed, while artificial intelligence has been fully embraced by a plethora of pure players ( , , , …) traditional actors are still lagging, and now see the technology as a shortcut to a much-needed renewed growth. digitalization of customer journeys cloud technologies computing capacities If it sounds like I’m using long words to say “digital disruption”, trust your instinct Spotify Netflix Buzzfeed Disney Below is an exhaustive look at the use cases currently being implemented by the old guard throughout the world. 1. Creation process optimisation Metadata creation/indexing automation Using machine learning, artificial intelligence can both translate data and use image recognition to automate the creation of metadata for all types of content (describing an image, for example, to make it easier to find via Google). This makes it easier for both internal and external stakeholders to discover content by allowing searches to be carried out with finer criteria, for more precise results. Automatic indexing (as well as the conversion of a multitude of data formats) speeds up the work of journalists, facilitate the verification of facts, and allows humans to concentrate on tasks with more added value. , , … Dozens of media companies have already implemented these solutions across their operations. Fox The New York Times BBC Automation of article writing and video creation We’re also seeing the emergence of tools automating the writing of articles with low added value, thus allowing journalists to work on subjects that require both more investigation and more specialized expertise. , for example, offers journalists bots to automatically process election results: their tool generates texts as soon as information arrives on the Ministry of the Interior website. The Washington Post’s has been doing about the same with sports since 2016. Other such tools can greatly speed up article writing or video creation by using voice recognition to translate audio information into text. This text is then directly “re-encoded” to the video, which makes it easier to find, verify and potentially modify. Syllabs Data2Content Heliograf Note that we’re discussing very simple tasks : most machine learning tests going beyond factual reporting have failed ( ). often hilariously Identification of emerging trends The first journalist to break a news-story gets all the glory, so the the story goes. As such, it is paramount to control and anticipate information movements, but also track its dissemination in real time to stand out from competitors. To that effect, the R&D teams of the press agency have developed the tool which identifies significant events on Twitter thanks to an algorithm. This tool assigns these events a “media score” which makes it possible to focus primarily on the most significant events. is also able to generates a note of confidence on the veracity of these events. Reuters News Tracer News Tracers This last point is particularly important, as it meets the paramount requirement of reliability of information expected from the media. Fact-checking Indeed, some algorithms can also help journalists verify the reliability and accuracy of images and videos posted on social networks. and , for example, are two start-ups using intelligent algorithms to combat false information and deep fakes. This is done by assigning confidence indices to the content, based on data provided by end users. ClaimBuster FactMata 2. Distribution, personalisation and recommendation of content Although the opportunities presented above will indeed have a disruptive impact on the media market, They merely represent a small fraction of the opportunities offered to the media industry by machine learning. Although the automation of content creation is eagerly awaited by science fiction enthusiasts, the major impact of A.I will be through the process by which the content is adapted and presented to audiences. As the market produces oceans of content , an effective pairing is one that converts this mass into a set of value proposition adapted to each final consumer or group of final consumer. This granularity can vary in several ways: of very variable quality and attractiveness It can mean the right content in the right place Take and its 400 channels of distribution, for example. Using human intelligence to put the right content in the right place would be time-consuming, and any optimisation would be deeply imperfect. As such, the company used artificial intelligence to inform the probability of the virality of an article and promote it on the channel most suitable for the public having the greatest chances of appreciate. This probability is the product of a joint effort by the product, social media, engineering and data science teams, who developed a machine learning model based on historical data of high-performance content. Buzzfeed It can mean the right content at the right time The dead come back to life. , a machine learning solution used by Swiss magazine , is able to identify the magazine’s best articles by cross-referencing its archives with data from (which provides data and analyzes to global publishers) and . The algorithm then assigns a relevance score according to qualitative indicators (reading time, audience history, engagement and debate aroused on social networks …) and advises the best time to republish and reach new audiences. This gives a second life to the content, and the ensuing revenues can go straight to the bottom-line. Zombie Le Temps Chartbeat Google Analytics It can mean the right content for the right group of people has more content than anyone can (or wants to) read or watch. Thanks to the solution from the start-up , the publisher has therefore created profiles based on the history of the content consumed (traced by cookies). Depending on their profile, the readers are offered suitable content, both on the site and in theirs emails. This investment in the segmentation of its readers has enabled to . In addition, return traffic to the site . Business Insider Sailthru Business Insider increase its click-through rates by 60% and its click-through rates in newsletters by 150% jumped by 52% . It is paramount to note here that such tools very much have the power to further polarise our societies by providing content that aligns with existing views, and that their use in the media should not be accepted lightly It cannot (yet) mean the right content for the right person An algorithm can make decisions based on contextual data and customer journeys, but will hardly be able to understand the preferences of each one of those customers : likes and dislikes are rarely stable and are usually highly contextual. The variety of human tastes still baffles machines for the time being. As such, any personalization at the level of the individual must, for now, be assisted by a curator. Nevertheless, many such roles are merely postponing the seemingly unavoidable : , almost three quarters (72%) of media players plan to actively experiment with A.I in order to improve recommendations and increase production effectiveness. according to a study Reuters 3. Monetisation As monetization is one of the most important lever to traditional media’s survival, many use cases are starting to emerge in this area, from customer engagement to advertising space. Paywalls Many major publications use a paywall (a system whereby access is limited to users who have paid to subscribe to the site) in one form or another. It often comes in the form of blocking content after reading a finite number articles. Here too, Artificial intelligence offers a more dynamic and less coercive value proposition. Take for example, the , a Swiss press group that uses an algorithm combining a hundred criteria to determine when the internet user is most likely to trigger his paid engagement. Once the right moment is detected, a personalized landing page appears for the prospect. With a conversion rate multiplied by 5 in three years, the results are convincing. Neue Zürcher Zeitung Real-time bidding We can imagine that Real-Time Bidding (RTB) will find new dimensions when propped up by smart algorithms, particularly on the supplier side: a Machine Learning algorithm could optimize the selection of applicants, manage the costs of data transfer, improve visibility of performance, avoid fraud or even check the quality of advertisements. However, such examples are still very rare on the market today. Conclusion A.I will influence all parts of the media value chain, helping content creators to be more creative, content publishers to be more productive and consumers to find content that suits their interests. Artificial intelligence is nevertheless only a part of a logic of continuity of business optimization strategies and, as such, should not be considered as a revolution. This fact in now way minimizes the work behind its implementation: as scientist Peter Skomoroch mentioned last year : “ “ you can expect your company’s transition to machine learning to be roughly 100 times more difficult as your transition to mobile. This story was originally written for my blog, The Pourquoi Pas. Come say hi!

How AI and Machine Learning Are Infiltrating the Media Industry

Independent, Testable , and Clean After reading the uncle Bob’s Clean Architecture Concept, I’m trying to implement it in Golang. This is a similar architecture that we used in our company, but a little different structure. Not too different, same concept but different in folder structure. Kurio - App Berita Indonesi a, You can look for a sample project here , a sample CRUD management article. https://github.com/bxcodec/go-clean-arch Disclaimer : I’m not recommending any library or framework used here. You could replace anything here, with your own or third party that has the same functions. Basic As we know the constraint before designing the Clean Architecture are : Independent of Frameworks. The architecture does not depend on the existence of some library of feature laden software. This allows you to use such frameworks as tools, rather than having to cram your system into their limited constraints. Testable. The business rules can be tested without the UI, Database, Web Server, or any other external element. Independent of UI. The UI can change easily, without changing the rest of the system. A Web UI could be replaced with a console UI, for example, without changing the business rules. Independent of Database. You can swap out Oracle or SQL Server, for Mongo, BigTable, CouchDB, or something else. Your business rules are not bound to the database. Independent of any external agency. In fact your business rules simply don’t know anything at all about the outside world.More at https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html So, based on this constraint, every layer must independent and testable. If Uncle Bob’s Architecture, has 4 layer : Entities Usecase Controller Framework & Driver In my projects, I’m using 4 too : Models Repository Usecase Delivery Models Same as Entities, will used in all layer. This layer, will store any Object’s Struct and its method. Example : Article, Student, Book. Example struct : type Article struct {
	ID        int64 Title     string Content   string UpdatedAt time.Time CreatedAt time.Time } import "time" `json:"id"` `json:"title"` `json:"content"` `json:"updated_at"` `json:"created_at"` Any entities, or model will stored here. Repository Repository will store any Database handler. Querying, or Creating/ Inserting into any database will stored here. This layer will act for CRUD to database only. No business process happen here. Only plain function to Database. This layer also have responsibility to choose what DB will used in Application. Could be Mysql, MongoDB, MariaDB, Postgresql whatever, will decided here. If using ORM, this layer will control the input, and give it directly to ORM services. If calling microservices, will handled here. Create HTTP Request to other services, and sanitize the data. This layer, must fully act as a repository. Handle all data input - output no specific logic happen. This Repository layer will depends to Connected DB , or other microservices if exists. Usecase This layer will act as the business process handler. Any process will handled here. This layer will decide, which repository layer will use. And have responsibility to provide data to serve into delivery. Process the data doing calculation or anything will done here. Usecase layer will accept any input from Delivery layer, that already sanitized, then process the input could be storing into DB , or Fetching from DB ,etc. This Usecase layer will depends to Repository Layer Delivery This layer will act as the presenter. Decide how the data will presented. Could be as REST API, or HTML File, or gRPC whatever the delivery type. This layer also will accept the input from user. Sanitize the input and sent it to Usecase layer. For my sample project, I’m using REST API as the delivery method. Client will call the resource endpoint over network, and the Delivery layer will get the input or request, and sent it to Usecase Layer. This layer will depends to Usecase Layer. Communications Between Layer Except Models, each layer will communicate through inteface. For example, Usecase layer need the Repository layer, so how they communicate? Repository will provide an interface to be their contract and communication. Example of Repository’s Interface package repository models type ArticleRepository interface {
	Fetch(cursor string, num int64) ([]*models.Article, error)
	GetByID(id int64) (*models.Article, error)
	GetByTitle(title string) (*models.Article, error)
	Update(article *models.Article) (*models.Article, error)
	Store(a *models.Article) (int64, error)
	Delete(id int64) (bool, error)
} import "github.com/bxcodec/go-clean-arch/article" Usecase layer will communicate to Repository using this contract, and Repository layer implement this interface so can used by Usecase MUST Example of Usecase’s Interface package usecase ( )

type ArticleUsecase interface {
	Fetch(cursor string, num int64) ([]*article.Article, string, error)
	GetByID(id int64) (*article.Article, error)
	Update(ar *article.Article) (*article.Article, error)
	GetByTitle(title string) (*article.Article, error)
	Store(*article.Article) (*article.Article, error)
	Delete(id int64) (bool, error)
} import "github.com/bxcodec/go-clean-arch/article" Same with Usecase, Delivery layer will use this contract interface. And Usecase layer implement this interface. MUST Testing Each Layer As we know, clean means independent. Each layer testable even other layers doesn’t exist yet. This layer only tested if any function/method declared in any of Struct. And can test easily and independent to other layers. Models Layer: To test this layer, the better ways is doing Integrations testing. But you also can doing mocking for each test. I’m using github.com/DATA-DOG/go-sqlmock as my helper to mock query process msyql. Repository: Because this layer depends to Repository layer, means this layer need Repository layer for testing . So we must make a mockup of Repository that mocked with mockery, based on the contract interface defined before. Usecase: Same with Usecase, because this layer depends to Usecase layer, means we need Usecase layer for testing. And Usecase layer also must mocked with mockery, based on the contract interface defined before Delivery: For mocking , I use mockery for golang by vektra could be seen here https://github.com/vektra/mockery Repository Test To test this layer, like I said before , I’m using a sql-mock to mock my query process. You can use like what I used here github.com/DATA-DOG/go-sqlmock , or another that have similar function func TestGetByID(t *testing.T) {
 db, mock, := sqlmock.New() err != nil { 
    t.Fatalf(“an error ‘%s’ was not expected when opening a stub  
        database connection”, err) 
  } 
 defer db.Close() 
 rows := sqlmock.NewRows([]string{
        “id”, “title”, “content”, “updated_at”, “created_at”}).   
        AddRow( , “title ”, “Content ”, time.Now(), time.Now()) 
 query := “SELECT id,title,content,updated_at, created_at FROM 
          article WHERE ID = \\?” 
 mock.ExpectQuery(query).WillReturnRows(rows) 
 a := articleRepo.NewMysqlArticleRepository(db) 
 num := int64( ) 
 anArticle, := a.GetByID(num) 
 assert.NoError(t, err) 
 assert.NotNil(t, anArticle)
} err if 1 1 1 1 err Usecase Test Sample test for Usecase layer, that depends to Repository layer. package usecase_test ( models ucase )

func TestFetch(t *testing.T) { := (mocks.ArticleRepository) mockArticle models.Article
	err := faker.FakeData(&mockArticle)
	assert.NoError(t, err)

	mockListArtilce := make([]*models.Article, )
	mockListArtilce = append(mockListArtilce, &mockArticle)
	mockArticleRepo.On( , mock.AnythingOfType( ), mock.AnythingOfType( )).Return(mockListArtilce, nil)
	u := ucase.NewArticleUsecase(mockArticleRepo)
	num := int64( )
	cursor := list, nextCursor, := u.Fetch(cursor, num)
	cursorExpected := strconv.Itoa(int(mockArticle.ID))
	assert.Equal(t, cursorExpected, nextCursor)
	assert.NotEmpty(t, nextCursor)
	assert.NoError(t, err)
	assert.Len(t, list, len(mockListArtilce))

	mockArticleRepo.AssertCalled(t, , mock.AnythingOfType( ), mock.AnythingOfType( ))

} import "errors" "strconv" "testing" "github.com/bxcodec/faker" "github.com/bxcodec/go-clean-arch/article" "github.com/bxcodec/go-clean-arch/article/repository/mocks" "github.com/bxcodec/go-clean-arch/article/usecase" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" mockArticleRepo new var 0 "Fetch" "string" "int64" 1 "12" err "Fetch" "string" "int64" Mockery will generate a mockup of repository layer for me. So I don’t need to finish my Repository layer first. I can work finishing my Usecase first even my Repository layer not implemented yet. Delivery Test Delivery test will depends on how you to deliver the data. If using http REST API, we can use httptest a builtin package for httptest in golang. Because it’s depends to Usecase, so we need a mock of Usecase . Same with Repository, i’m also using Mockery to mock my usecase, for delivery testing. func TestGetByID(t *testing.T) { mockArticle models.Article 
 err := faker.FakeData(&mockArticle) 
 assert.NoError(t, err) 
 mockUCase := (mocks.ArticleUsecase) 
 num := int(mockArticle.ID) 
 mockUCase.On(“GetByID”, int64(num)).Return(&mockArticle, nil) 
 e := echo.New() 
 req, := http.NewRequest(echo.GET, “/article/” +  
             strconv.Itoa(int(num)), strings.NewReader(“”)) 
 assert.NoError(t, err) 
 rec := httptest.NewRecorder() 
 c := e.NewContext(req, rec) 
 c.SetPath(“article/:id”) 
 c.SetParamNames(“id”) 
 c.SetParamValues(strconv.Itoa(num)) 
 handler:= articleHttp.ArticleHandler{ : mockUCase, : httpHelper.HttpHelper{}
 } 
 handler.GetByID(c) 
 assert.Equal(t, http.StatusOK, rec.Code) 
 mockUCase.AssertCalled(t, “GetByID”, int64(num))
} var new err AUsecase Helper Final Output and The Merging After finish all layer and already passed on testing. You should merge into one system in main.go in root project. Here you will define, and create every needs to environment, and merge all layers into one. Look for my main.go as example: package main ( httpDeliver articleRepo articleUcase cfg _ ) config cfg.Config

func init() {
	config = cfg.NewViperConfig() config.GetBool( ) {
		fmt.Println( )
	}

}

func main() { := config.GetString( )
	dbPort := config.GetString( )
	dbUser := config.GetString( )
	dbPass := config.GetString( )
	dbName := config.GetString( )
	connection := fmt.Sprintf( , dbUser, dbPass, dbHost, dbPort, dbName)
	val := url.Values{}
	val.Add( , )
	val.Add( , )
	dsn := fmt.Sprintf( , connection, val.Encode())
	dbConn, := sql.Open( , dsn) err != nil && config.GetBool( ) {
		fmt.Println(err)
	}
	defer dbConn.Close()
	e := echo.New()
	middL := middleware.InitMiddleware()
	e.Use(middL.CORS)

	ar := articleRepo.NewMysqlArticleRepository(dbConn)
	au := articleUcase.NewArticleUsecase(ar)

	httpDeliver.NewArticleHttpHandler(e, au)

	e.Start(config.GetString( ))
} import "database/sql" "fmt" "net/url" "github.com/bxcodec/go-clean-arch/article/delivery/http" "github.com/bxcodec/go-clean-arch/article/repository/mysql" "github.com/bxcodec/go-clean-arch/article/usecase" "github.com/bxcodec/go-clean-arch/config/env" "github.com/bxcodec/go-clean-arch/config/middleware" "github.com/go-sql-driver/mysql" "github.com/labstack/echo" var if `debug` "Service RUN on DEBUG mode" dbHost `database.host` `database.port` `database.user` `database.pass` `database.name` "%s:%s@tcp(%s:%s)/%s" "parseTime" "1" "loc" "Asia/Jakarta" "%s?%s" err `mysql` if "debug" "server.address" You can see, every layer merge into one with its dependencies. Conclusion In short, if drawn in a diagram, can seen below Every library used here you can change by your own. Because the main point of clean architecture is : no matter your library, but your architecture is clean, and testable also independent This is how i organized my projects, you could argue, or agree, or maybe improve this for more better, just leave a comment and share this The Sample Projects The sample project can seen here https://github.com/bxcodec/go-clean-arch Library used for my project: Glide : for package management go-sqlmock from github.com/DATA-DOG/go-sqlmock Testify : for testing Echo Labstack (Golang Web Framework) for Delivery layer Viper : for environment configurations Further Reading about Clean Architecture: Second part of this article: https://hackernoon.com/trying-clean-architecture-on-golang-2-44d615bf8fdf https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html http://manuel.kiessling.net/2012/09/28/applying-the-clean-architecture-to-go-applications/ . Another version of Clean Architecture in Golang If you have a question , or need more explanation, or something, that I can not explain well here, you can ask me from my linkedin or email me. Thank you

Trying Clean Architecture on Golang

Probando la arquitectura limpia en Golang

Hardware acceleration for data processing has a long history. It’s not new. When I was a child, my dad outfitted our 12 MHz 286 system with the optional 80287 Floating Point Unit (FPU) coprocessor, which ran at a whopping 4.7MHz. The thing was a beast. It did wonders for Lotus 1–2–3. Not long after, this specific FPU was integrated into the CPU, reducing the need for the external coprocessor for math operations. However, coprocessor actually stuck around for longer. You know that GPU you’re using to read this article? DSPs in your mobile phone? Sound card? They’re all coprocessors, and there’s a reason they’re still around. Your CPU can’t do everything No, really. It can’t. The actual silicon for the Intel x86 (the heir to the 8086 and my beloved 80286) architecture is already crazy complex. Adding additional operations to the x86 architecture has diminishing returns, according to Rick Merrit of the EETimes. to the point where it may not be worth it In fact, GlobalFoundries, one of the largest silicon chip manufacturers has announced it’s . It’s just not worth it anymore. abandoning plans for 7nm chips Intel Haswell-EP Xeon E5–2699 V3 Die on wccftech Why is that a problem? Well, each processor core in the x86 architecture does so many different things, that it has actually made programmers complacent. Don’t worry about writing bad software — the processor will make sure it runs fast! Of course, Intel already knew this back in the early 2000s. They tried to kill off the x86 architecture with Itanium, a processor series designed for 64-bit computing. However, AMD had a different plan, and released 64-bit capable x86 processors, obstructing Intel’s plans. And that’s how we ended up with this mess. Going multicore The only way to get more oomph out of CPUs was to scale them out. That’s why today’s CPUs are almost always multi-core. They have anywhere from 2 to 28 cores for higher end server chips like the Intel Xeon series. The clever pipelining and vectorization, caching, and some other complex systems found in the x86 architecture allow for pretty good use of the miniature electronics. Some tasks can run in groups, and the processor may even correctly predict what the next actions are, before they happen. However, as I previously mentioned — these processors are already crazy complex. Other architectures do exist. ARM, which you can find in a huge variety of devices — from battery chargers to your phone, and even your car — has a reduced instruction set, also known as RISC. IBM has similarly gone the RISC route. RISC processors have a much simpler core, which can’t do as much as more elaborate CPU cores. Software has to be specifically written to be multi-core to benefit from these kinds of processors. For now, x86 has a stay of execution. It could be another 10, 15 years, but x86 will be going away. If you’re interested in finding out more about what Intel was doing to enable parallel processing, read “The Story of ISPC”, about the Intel SPMD Program Compiler: https://pharr.org/matt/blog/2018/04/18/ispc-origins.html Here come the GPUs and GPGPUs Graphical Processing Units (GPUs) are also not really new. The term has , but tended to focus strictly on graphics. NVIDIA’s first card marketed as a “GPU” was the GeForce 256, from 2009. However, General Purpose GPUs ( s) actually started appearing around 2007, when NVIDIA and ATI (now AMD) started equipping their 3D graphics cards with more and more capabilities, like Unified Pixel Shaders. These capabilities could be repurposed to perform operations like matrix multiplications, Fast Fourier Transform, wavelet transform, and more. been in use since 1986 at least GPGPU NVIDIA Tesla C870 with 128 CUDA cores ( ) NVIDIA Accessing these features meant that new programming constructs had to be invented. So, in 2008, Apple started developing OpenCL. By late 2009 it was already adopted by AMD, IBM, Qualcom, Intel, and even NVIDIA. AMD decided to support OpenCL extensively, instead of their “Close to Metal” framework. AMD’s Radeon Pro V340, with 512GB/s of memory bandwidth. Source: AMD While NVIDIA supported and still supports OpenCL, they didn’t abandon their own framework, known as CUDA. Today, CUDA is the de-facto standard for high-speed, high-throughput GPU computing. It’s essentially a software layer that gives direct access to the GPU’s virtual instruction set and parallel computational elements, for the execution of compute kernels. Newer NVIDIA GPUs have up to 5,760 CUDA cores With 256 cores in 2009, and with up to 5,760 cores in a single modern GPU, I think it’s easy to see why GPUs are useful for operations that require a high degree of parallelism. But it’s not just parallelism — GPUs have extremely large memory bandwidth, which makes it suitable for high-throughput applications. Why don’t we just replace CPUs with GPUs? Not all applications are suitable for GPUs. GPUs are mostly useful if: The actions are repetitive The actions are mostly independant (do not rely on eachother) The actions are computationally intensive GPUs have found some interesting uses outside of graphics (and cryptomining). Today, GPUs are found in a variety of data processing scenarios. They can be used as part of a data processing pipeline, for running machine learning models on, as part of a hardware accelerated relational database, or just render a result nicely. Let’s take a look at three of the main areas where GPUs can help in the data processing pipeline: GPUs for Stream Processing New stream processing solutions, like ’s Plasma Engine can take advantage of GPUs for stream processing data coming in and out of databases (GPU or not). This tool can be used to perform analysis and/or transformation of streaming data on the GPU. FASTDATA.io The main competitor to FASTDATA’s engine is , which is available as an open-source add-on from IBM. GPU-enabled Spark GPU Databases for Analytics Practically all GPU databases are ground-up, purpose built for analytics. GPU Databases. Source: The 2018 Big Data Landscape The reason for building a GPU database from scratch seems to be the difficulty in retrofitting an old, row-based database with a very different technologies. There are several players in the GPU database field, each with their own benefits: OmniSci (formerly MapD) — In-memory GPU database for geospatial use-cases SQream DB — GPU data warehouse for larger-than-RAM data-sets BlazingDB — GPU database based on Parquet Kinetica — In-memory GPU database HeteroDB — Postgres plugin, based on UDFs for fast NVMe access Brytlyt — Postgres plugin for in-memory compute Blazegraph — Graph database GPUs for machine learning Machine learning tends to be highly suitable for GPU architecture. It’s not just about the high level of parallelism, and the multitude of matrix operations. A big part of the reason is actually the aforementioned memory bandwidth. NVIDIA Tesla V100 core-arrangement. Source: ServeTheHome CPUs are what we call latency-oriented processors. Latency-oriented means that they favor lower latency of operations with clock cycles in the nanoseconds (~3 billion operations per second on one piece of data). In contrast, GPUs are throughput oriented. They will perform a single operation over lots of data at once, at a lower clock speed (~1 billion operations per second on multiple chunks of data). GPUs sacrifice latency with a slower clock speed, in order to get a higher throughput on every clock cycle. There is no shortage in different GPU-enabled machine learning frameworks: TensorFlow cuBLAS Caffe Theano Torch7 cuDNN MATLAB cxxnet/mxnet Deeplearning4j Keras Mathematica and many more. Not just GPUs — ASICs, FPGAs and other exotic hardware While CPUs and GPUs are general purpose, we have some other alternatives in hardware, but there are trade-offs. We’ve already discussed CPUs and GPUs, so let’s talk about FPGAs and ASICs specifically. FPGAs Field Programmable Gate Arrays (FPGA) are a high-speed, programmable chip. They’re extremely popular for prototyping and specialist devices. They contain an array of programmable logic blocks, and a hierarchy of reconfigurable interconnects. Because FPGAs are programmable, they can be altered after they’ve been manufactured, but not by anyone. They provide good performance, but can cost quite a bit — which is why they remain mostly in low-yield devices. You can find FPGAs in medical equipment, cars, specialist equipment, etc. FPGAs provide higher memory bandwidth, lower power consumption than CPUs (and GPUs), but they may struggle with floating point computations, and are hard to work with. In fact, they’re quite difficult to program for, even though recent tooling like OpenCL for FPGA exists. In data analytics, FPGAs are suited for simple repetitive tasks. They’re found in modern platforms like , the , the , and devices like . One of the most well-known usage of FPGAs in analytics is , although the FPGA aspect of Netezza seems to have been mostly abandoned. Microsoft’s Project Brainwave Swarm64 Database Accelerator for Postgres Xilinx Alveo Data Center Accelerator Ryft IBM’s Netezza Eventually, Netezza ended up as part of and the FPGA components were used primarily for specific operations that would benefit from FPGAs like decompression and transformations. Other operations still used the CPU. IBM PureData Source: IBM Data Retrieval Technologies RDBMS, BLU, IBM Netezza and Hadoop ASICs Custom designed Application-Specific Integrated Circuits (ASIC) are the fastest option for high-speed operations, in the silicon level. Because these chips are highly customized, they can deliver the best performance with minimal overhead. ASICs are commonly found as digital signal processors (DSPs) — in audio processing, video encoding, and networking among others. These applications are primarily in mass-produced devices like cameras, mobile phones, wireless routers, etc. Because of the high cost and time investment in creating custom ASICs, they are also the most costly option. They are actually the cheaper option if you plan on manufacturing millions of them, but at smaller volumes they are hard to justify. For that reason, they’re not commonly found in analytics or data processing proper. Comparing the tradeoffs Here’s a summary of the different processors and their tradeoffs: Each of the processor types has its own tradeoffs The future of hardware accelerated data processing The future is in hardware diversity, but not all of the coprocessors are suitable for every task. : Keep standard, sequential programs and code running The CPU’s job Run high-throughput parallel code The Coprocessor’s job: Increase the throughput of parallel programs CPU + Coprocessor: It’s clear that the x86 CPU architecture will not last forever. It simply is not sustainable for the long-run. It’s not advancing at the same rate that modern data workloads are. Because they’re not bound to the same backwards compatibility, GPUs are advancing much faster. Modern data workloads are varied, varying, growing, and fast-moving. GPU and other hardware accelerated data processing applications are growing in popularity, with some notable successes in the machine learning space and data warehousing space. Tesla V100 with NVLink GPU-to-GPU and GPU-to-CPU (Power9) connections. Source: NVIDIA FPGA and GPU-enabled hardware, like IBM Power9 with NVLINK and OpenCAPI, as well as Azure and AWS GPU and FPGA instances are just a sign of things to come. Conclusion I believe hardware accelerated data processing is still in it’s infancy, but is likely to become more widespread, as the accelerating technologies continue to make strides over today’s CPU architecture. Thanks for reading this to the end! I know it isn’t easy — there’s a lot of information (and acronyms) to digest. This article is the first in a series of hardware accelerated data articles. In my next article, I will deep dive into how different hardware accelerated databases utilize their respective hardware accelerators. If you enjoyed this article, leave a comment below, or share with your friends and colleagues.

A gentle introduction to hardware accelerated data processing

Since the 2016 US presidential election, the term “fake news” has become part of our daily vernacular. Fake news has become so prevalent in our society that even the most sophisticated of us fall for it sometimes. It has reached such epidemic proportions that French President Macron recently commented, “we must fight against the virus of fake news.” In fact, the ranks this massive digital spread of misinformation among the top future global risks, along with failure to adapt to climate change, organized crime, and the food shortage crisis. World Economic Forum World Economic Forum — Global Risks 2013 Eighth Edition We’re living in an information “ ” where media and technology companies rush to develop new defenses as fake news techniques proliferate. arms race But who should fix these problems? The first thing to establish is who’s responsible for fake news. Some people like to blame social media and their lack of response to this new epidemic. Others like to add that it is in the nature of fake news to be extremely shareable — exactly like the legitimate content the news media has created its new distribution models around. In my opinion, improving the ability of platforms and media to address the fake news phenomenon requires taking a more holistic approach, identifying areas in every part of the value chain — from creation all the way to circulation and discovery — that need to change. As the problem has become more widely acknowledged, fighting fake news has proven to be fertile ground for . There are lots of different startups fighting fake news, and they’re going about it in many different ways. For instance, some aim to tackle the huge challenge of fighting falsehoods on social media platforms, while others aim to protect brands and their reputation from misinformation. As a result, there are a number of different markets in which startups have the opportunity to succeed. startups In order to pinpoint where these opportunities and challenges lie, I’ve broken down fake news into four distinct stages: 1) creation, 2) publication, 3) circulation, and 4) discovery. I’ll discuss those stages below and highlight startups working to address various problems within each stage. Value Chain Startup Map Creation An attacker looking to spread misinformation will need to create content that they can then distribute widely. To do this, they can employ two broad methods: 1) create fake content and/or 2) create fake accounts with which to spread fake content. An example of a company attacking the fake content issue is , an AI powered news site, which generates unbiased news stories. Looking at all sites (left-leaning, right-leaning, etc.), Knowhere writes its own “impartial” version of a story based on what it finds. Tackling fake accounts, another company, , tracks the behavior of website visitors and uses artificial intelligence to determine which are bots and which ones should be blocked. Operating under the thesis that fake news has an army of distribution bots at its core, PerimeterX aims to stop malicious bots in their tracks before they attack. Knowhere PerimeterX Publication Once content has been created, an attacker will need to publish their misinformation online. Most fake-news-fighting companies operate at this stage, looking to tackle the challenge of deciphering between fact or fiction. Although this is technologically a hard problem to solve given the nuances of human understanding, I view the startups operating solely in this stage as projects rather than ventures, as there is no clear business model and it’s very much a game of whack-a-mole that requires automation at scale. As a result, you see grassroots efforts like the , a competition run by volunteers in the AI community, fostering the development of tools that help fact-checkers do their jobs more efficiently. Fake News Challenge Circulation Like any good news story, fake content will be shared, liked, reposted, and distributed across many different platforms and channels. And while the impact on individual consumers might not be great, advertisers can lose business by hosting bad content and brands can be damaged by being displayed next to bad content. In fact, a June 2017 study from Yahoo’s Brightroll found that 96% of programmatic decision-makers are concerned about fake news as it relates to their business. Of those surveyed, 31% said they will respond to the issue by reducing spend with programmatic partners whose inventory associates brands with fake news. As such, startups like Israeli-based are looking to help ad agencies not only identify fake stories to ensure their brand doesn’t appear alongside them, but also defund the bad actors who are producing low-quality content. AdVerif.ai Discovery The more widely a piece of misinformation can be spread, the better the chances that it will capture the public imagination and achieve its objective. In order to prevent the discovery of such content, some startups are taking a decentralized approach whereby an editorial board no longer determines what’s worth reading: instead, users, developers, and publishers run custom rankings to produce search results. One such company is , which is using Blockchain technology to create transparent and publicly auditable content networks and ranking algorithms. A decentralized platform gives publishers and application developers new business models, outside of advertising and subscriptions, and potentially improves “organic” content discovery for audiences that doesn’t rely on easily abused social media signals such as links, likes, and votes. Userfeeds Full-stack Solutions The companies I highlighted above were startups operating in singular stages, but if you take a look at the market map, you’ll notice three companies operating more as a full-stack solution, whereby the output of one stage is fed as inputs into another to create a data feedback loop, a competitive moat. It’s clear that, as attackers continue to evolve their techniques and innovate, so too will defenders, whether it’s: ’s approach of building a fact-checking platform to create content credibility scores that it uses to partner with ad networks or; Factmata ’ approach of using image data to protect consumers from fake news as they browse or; Robhat Labs ’s approach of building a platform to identify the coordination of attacks and providing tools to businesses to best respond. New Knowledge And as I think through the opportunities for why these companies are the most likely to succeed, I boil it down to three core themes: Dual-product business model Similar to the way a computer virus spreads from host to host, misinformation spreads like wildfire from person to person. For example, a recent MIT showed that fake news spreads six times faster than the truth on Twitter. Continuing this virus analogy, I see companies tackling fake news in the same way they responded to anti-virus software back in the 1990s and 2000s; Symantec is an example of an appropriate consumer-plus enterprise business model worth replicating today. study By providing a free consumer service, a company that roots out and identifies fake news can gather data at little to no cost and build up a base of knowledge about where all harmful content resides. This data can then be sold to enterprises, providing a layer of analytics and insights. The consumer product is thus a means to the end of selling to businesses. Once the service compiles this data, one potential commercialization can be around brand security. The social media management / analytics market, of which reputation management is a sub-segment, is estimated to grow to $9.54B by 2022 and is driven largely by the increased access to information. Brands today are unaware of where they are going to appear and know that their reputation can be damaged in a very short time. Their brand may appear next to undesirable or inappropriate content, or other content may spread unfair, deceptive, or non-credible information about them. Companies are increasingly looking for solutions that not only protect their brands but give them insights into how their brands are being portrayed. Coordination over content At this point, I’ve seen too many startups building businesses around a fact-checking service that operates as some extension of Google Chrome. The problem with whitelisting news sources is that you’re handing the advantage to incumbent news sources — which means there can be very little innovation in this space. What I’m much more interested in seeing are startups moving up the stack toward the source of the content and trying to identify the coordination of activities rather than the veracity of the data, which is a much smaller market. If you have a network of things being shared and a network of people sharing them, there are orders of magnitude more people sharing “interesting” things than things being shared. The content itself doesn’t change nearly as fast as users do. Moreover, anybody identifying content runs into a slippery problem: despite its prevalence, identifying fake news is highly subjective. The term “fake news” has so many definitions and requires such nuances of human understanding that competent fact checking becomes a sort of AI-complete problem. As such, the biggest room for improvement is in detecting different types of coordination, the signature of behavior. Due to the sheer scale of information online, people have yet to try all the possible ways to cluster accounts in a network. Imagine identifying clusters not only by the content but also by comparing stylometry, topicality, timing, and other characteristics. By analyzing the network propagation of stories and trying to use that heuristic to classify things as fake, companies can avoid having to determine what is ground truth. Pushing truth to power In my view, companies that aim to create a more holistic, trustworthy news platform have a higher chance of succeeding than those looking to build the next crowdsourced fact-checking service or bias-detection algorithm. Pushing values of truth can take many forms. One example might be building a reputation score that indicates how much the reader can trust a piece of content. Having a reputation system that identifies and differentiates trusted sources of information from bots and from misinformation could be an invaluable tool for consumers trying to make informed decisions. I liken this to New York City’s requirement that restaurants post calorie counts on their menus. This not only encouraged New Yorkers to eat more healthfully, but — an unintended consequence — also forced restaurants to introduce lower-calorie items. Similarly, by having a reputation score on each piece of content, users would be able to see which sources and content they could believe, implicitly putting pressure on the content producers who care about their reputation. Just as the availability of food metrics like calories and nutrients helps drive better decision-making and changes public health at a meta level, metrics that scale up to increase the authority of accurate news will drive more accurate thinking about a multiplicity of information sources. Reputation leads to credibility, which leads to trust. The internet has democratized publishing, but it has not made editing for the common good — that is, optimal consumer education and media literacy — a value or priority. Whether it’s coming up with new models of distribution or building out security products over time, I too resonate with French President Emmanuel Macron when he says, “We must fight against the virus of fake news.” Ultimately, I believe this space will be a blend of media and cybersecurity, in which companies will have rethought the distribution of true news while innovating technologically to stop the spread of fake news. If you think about it, cybersecurity attacks such as viruses and DDoS operate the same way, but instead now leverage the social media networks that have come to fruition over the past eight years. And startups that end up taking a multidisciplinary approach to prevention and look to tackle one or more of the three opportunities discussed above will triumph in the fake news arms race. If you’re building a company that is looking to join the battle or have a startup to recommend in any of the above categories, drop me an email at . jerry.lu@luxcapital.com I’d love to chat.

The Fake News Arms Race

GPU databases are the hottest new thing in the database world, and it’s the most innovative trend since Hadoop showed up over 5 years ago. Below is a guide on how to find the right GPU Database for you. What is a GPU database? To put it simply, a GPU database is a database, relational or non-relational, that uses a GPU (graphical processing unit) to perform some database operations. For example, GPU databases are typically fast. Subsequently, GPU databases are more flexible in processing many different types of data, or much larger amounts of data. Most of the GPU databases tend to focus on analytics, and they’re offering it to a market that was oversold on Hadoop for Big Data analytics. The truth is Hadoop was not designed for relational data analytics, and yet the widespread use of Hadoop led to the creation of many projects that aimed to do exactly that, with mixed results. Hadoop is now past it’s hype cycle and is now entering the “Plateau of productivity” stage. Making the most of GPUs GPUs and CPUs can cooperate quite nicely, when both are taken advantage for their unique benefits, but this is not just for performance’s sake. In reality, the different architecture of the GPU makes it unsuitable for performing specific operations, like text manipulations that may differ based on the content. Because the (thousands of) GPU cores like to work the same way, having contents that will cause the code to behave differently will cause a significant drop in performance — essentially becoming partially sequential and not strictly parallel (in what Nvidia’s CUDA calls branch divergence). A GPU excels at performing (relatively simple), repetitive operations on large amounts of data in many streams. It just wouldn’t function very well if the branches diverge, the operations launch too few streams (kernels), or the operations can’t be parallelized. One of the reasons Nvidia GPUs are so popular among developers is that packages like Thrust, CUB and Modern GPU have made writing performant, parallel GPU code quite easy, in contrast with optimized, vectorized, efficiently threaded CPU code. GPUs for Analytics In general, GPUs can be used for a variety of different stages in the analytics pipeline. It can be used as the main database, as part of the processing pipeline, or just for the resulting analytic dataset — for example, with popular frameworks like TensorFlow. Let’s take a look at two of the main areas where GPUs can help in the analytics pipeline. GPUs for Stream Processing New stream processing solutions, like FASTDATA.io’s Plasma Engine can take advantage of GPUs for stream processing data coming in and out of databases (GPU or not). This tool can be used to perform analysis and/or transformation of streaming data on the GPU. The main competitor to FASTDATA’s engine is GPU-enabled Spark, which is available as an open-source add-on from IBM. GPU Databases for Analytics With the exception of PG-Strom and Brytlyt, which retrofits the open-source Postgres RDBMS by augmenting it with GPU-aware parts, all other GPU databases are purpose built for analytics. Blazegraph is another exception, because it is designed for graph operations. This leaves us with four players, dealing mostly with relational, structured analytics, with an SQL interface. This chart below should help you understand which of these GPU database is right for you: GPU Databases for Analytics SQream DB (formerly MapD) Omnisci Kinetica BlazingDB (formerly PG-Strom) HeteroDB Blazegraph GPUs for Stream Processing FASTDATA.io IBM Spark GPU

Which GPU database is right for me?

Stories, News & Mentions about FACETDATA

Facetdata’s Stories on HackerNoon