Data Oleum

dataoleum

A Machine Learning Text Classification Case Study with a Product-driven Twist

একটি পণ্য-চালিত টুইস্ট সহ একটি মেশিন লার্নিং পাঠ্য শ্রেণিবিন্যাস কেস স্টাডি

Eine Fallstudie zur Textklassifizierung durch maschinelles Lernen mit einer produktgesteuerten Wendung

Un estudio de caso de clasificación de textos con aprendizaje automático con un toque impulsado por el producto

Une étude de cas de classification de textes d'apprentissage automatique avec une touche axée sur le produit

उत्पाद-संचालित ट्विस्ट के साथ एक मशीन लर्निंग टेक्स्ट वर्गीकरण केस स्टडी

製品主導型のひねりを加えた機械学習テキスト分類のケーススタディ

제품 중심 트위스트를 사용한 기계 학습 텍스트 분류 사례 연구

Um estudo de caso de classificação de texto de aprendizado de máquina com um toque orientado ao produto

Пример классификации текста с помощью машинного обучения с учетом особенностей продукта

Ürün Odaklı Bir Yönlü Makine Öğrenimi Metin Sınıflandırması Örnek Olay İncelemesi

Nghiên cứu điển hình về phân loại văn bản học máy với khuynh hướng hướng đến sản phẩm

产品驱动的机器学习文本分类案例研究

I first discovered way back in 2021 when I built a demo of their APIs for the Adobe Developer blog ( ). Diffbot "Natural Language Processing, Adobe PDF Extract, and Deep PDF Intelligence" At that time, I was impressed with how easy Diffbot's API was and also how quickly it responded. I had not looked at their API in a while, but a few days ago, they announced new support for summarizing text. I thought this would be a great thing to combine with the . Here's what I found. Adobe PDF Extract API First off, if you want to try this yourself, you'll need: Adobe PDF Services . These are free, and you get 500 transactions per month for free. For folks who may not know, I work for Adobe, and this is one of the products I cover. credentials Diffbot . They provide a free two-week trial but no free tier. That being said, I've had to reach out to them a few times when building stuff, and they've provided really great support for me, so I definitely think they're worth you checking out. credentials Alright, let's look at how a summary flow might work. Step One - Extract the Text The (sorry, the "Adobe PDF Extract API", wait, this is my blog, I can shorten things!) is pretty powerful. It uses AI to intelligently parse a PDF to correctly find each and every element detail in the document. Extract API So text, fonts, colors, position, and so forth. It can also find images and tabular data as well which leads to some pretty powerful use cases. (For a good example of this, see my where I scan multiple scientific journals to collect and aggregate astronomical data and create reports.) blog post For this demo, we literally just need the text. For that, I'll make use of the REST APIs. The "flow" for nearly all aspects of the PDF services available are: Exchange credentials for an access token Ask to upload a file for input (in this case, a PDF to be extracted) Upload the document Kick off the job Poll for completion Download the bits Note that there are also SDKs you can use, but I've found our REST APIs so simple I just hit the endpoints directly. Here's the script I wrote to do the Extract process. It's basically everything I said above and pointing to a source PDF in my local filesystem. /*
This demo is a two step process. This file, step one, handles extracting and storing the JSON from a PDF.
*/
import 'dotenv/config';
import fs from 'fs';
import { Readable } from 'stream';
import { finished } from 'stream/promises';

const SOURCE_PDF = '../../source_pdfs/boring_adobe_security_doc.pdf';
const REST_API = "https://pdf-services.adobe.io/";

const CLIENT_ID = process.env.CLIENT_ID;
const CLIENT_SECRET = process.env.CLIENT_SECRET;

async function delay(x) {
	return new Promise(resolve => {
		setTimeout(() => resolve(), x);
	});
}

async function getAccessToken(id, secret) {

	const params = new URLSearchParams();
	params.append('client_id', id);
	params.append('client_secret', secret);

	let resp = await fetch('https://pdf-services-ue1.adobe.io/token', { 
		method: 'POST', 
		headers: {
			'Content-Type':'application/x-www-form-urlencoded'
		},
		body:params 
	});

	let data = await resp.json();
	return data.access_token;
}

async function getUploadData(mediaType, token, clientId) {

	let body = {
		'mediaType': mediaType
	};
	body = JSON.stringify(body);

	let req = await fetch(REST_API+'assets', {
		method:'post',
		headers: {
			'X-API-Key':clientId,
			'Authorization':`Bearer ${token}`,
			'Content-Type':'application/json'
		},
		body: body
	});

	let data = await req.json();
	return data;
}

async function uploadFile(url, filePath, mediaType) {

	let stream = fs.createReadStream(filePath);
	let stats = fs.statSync(filePath);
	let fileSizeInBytes = stats.size;

	let upload = await fetch(url, {
		method:'PUT', 
		redirect:'follow',
		headers: {
			'Content-Type':mediaType, 
			'Content-Length':fileSizeInBytes
		},
		duplex:'half',
		body:stream
	});

	if(upload.status === 200) return;
	else {
		throw('Bad result, handle later.');
	}

}

async function pollJob(url, token, clientId) {

	let status = null;
	let asset; 

	while(status !== 'done') {
		let req = await fetch(url, {
			method:'GET',
			headers: {
				'X-API-Key':clientId,
				'Authorization':`Bearer ${token}`,
			}
		});

		let res = await req.json();

		status = res.status;
		if(status === 'done') {
			asset = res;
		} else {
			await delay(2000);
		}
	}

	return asset;
}

async function downloadFile(url, filePath) {
	let res = await fetch(url);
	const body = Readable.fromWeb(res.body);
	const download_write_stream = fs.createWriteStream(filePath);
	return await finished(body.pipe(download_write_stream));
}

async function extractJob(asset, token, clientId) {
	let body = {
		'assetID': asset.assetID
	}

	let resp = await fetch(REST_API + 'operation/extractpdf', {
		method: 'POST', 
		headers: {
			'Authorization':`Bearer ${token}`, 
			'X-API-KEY':clientId,
			'Content-Type':'application/json'
		},
		body:JSON.stringify(body)
	});

	return resp.headers.get('location');

}

let accessToken = await getAccessToken(CLIENT_ID, CLIENT_SECRET);
console.log('Got our access token.');

let uploadedAsset = await getUploadData('application/pdf', accessToken, CLIENT_ID);

await uploadFile(uploadedAsset.uploadUri, SOURCE_PDF, 'application/pdf');
console.log('Source PDF Uploaded.');

let job = await extractJob(uploadedAsset, accessToken, CLIENT_ID);
console.log('Job created. Now to poll it.');

let result = await pollJob(job, accessToken, CLIENT_ID);
console.log('Job is done.'); 

await downloadFile(result.content.downloadUri, 'extract.json');
console.log('All done.'); Ok, hopefully, you're still reading. In general, I try to avoid posting giant blocks of code like that, but if you focus on the lines at the end, you'll see I'm just hitting utility functions that do what I described in the flow above. Authenticate, ask to upload a PDF, kick off a job, check it, and download the result. One note I'll add. Extract returns a zip file containing a JSON result set, and optionally, tables and images. One nice thing about the REST API is that I can get directly to the JSON and just store it. The JSON result can be quite huge. For my source PDF (an incredibly boring Adobe security document) of three pages, the resulting JSON is 4560 lines long. You can find my source PDF , and the raw output from Extract . here here Instead of putting all 4.5k lines here, let me show a snippet - two unique elements found by the API: {
	"Bounds": [
		44.62139892578125,
		756.9429931640625,
		245.0037841796875,
		766.3184967041016
	],
	"Font": {
		"alt_family_name": "* Arial",
		"embedded": true,
		"encoding": "Identity-H",
		"family_name": "* Arial",
		"font_type": "CIDFontType0",
		"italic": false,
		"monospaced": false,
		"name": "*Arial-6539",
		"subset": false,
		"weight": 400
	},
	"HasClip": false,
	"Lang": "en",
	"Page": 0,
	"Path": "//Document/Sect/P",
	"Text": "Adobe Vendor Security Review Program White Paper ",
	"TextSize": 8.5,
	"attributes": {
		"SpaceAfter": 18
	}
},
{
	"Bounds": [
		0.0,
		0.0,
		630.0,
		820.7799987792969
	],
	"ClipBounds": [
		548.72802734375,
		739.1929931640625,
		602.5444488525391,
		820.7799987792969
	],
	"Page": 0,
	"Path": "//Document/Sect/Figure",
	"attributes": {
		"BBox": [
			548.9779999999737,
			739.4429999999993,
			587.61599999998,
			790.920999999973
		],
		"Placement": "Block"
	}
}, In the sample above, you can see the first element is textual and contains a property, while the second one is a figure. For my demo, I just need to use the property when it exists. Let's see that in action. Text Text Step Two - Create the Summary I mentioned earlier that the Diffbot API was fairly simple to use. Let me demonstrate that. First, I'll set up some variables and read in the JSON I got from the first step. To be clear, I could do everything in one process, but there's really no point in running Extract more than once. What's cool is - I could actually do calls on the result. multiple As an example, one other cool feature Diffbot has is to get entities from text, i.e., what a document is speaking about (people, places, etc). Anyway, here's the beginning: /*
In this file, we take the result from our Extract operation and pass it to Diffbot
*/

import 'dotenv/config';
import fs from 'fs';

const DIFFBOT_KEY = process.env.DIFFBOT_KEY;
const SOURCE_JSON = './extract.json';
const data = JSON.parse(fs.readFileSync(SOURCE_JSON, 'utf8'));

console.log(`Read in source data from ${SOURCE_JSON}.`); Next, I need to parse out the text from the Extract result: let text = data.elements.reduce((text, el) => {
	if(el.Text) text += el.Text + '\n';
	return text;
},''); Next, I craft an HTTP request to Diffbot. Check their for more information. docs let fields = 'summary';
let url = `https://nl.diffbot.com/v1/?fields=${fields}&token=${DIFFBOT_KEY}`;
	
let body = [{
	content:text, 
	lang:'en',
	format:'plain text'
}];

console.log('Passing text to Diffbot.'); 

let req = await fetch(url, { 
	method:'POST',
	body:JSON.stringify(body),
	headers: { 'Content-Type':'application/json' }
});

let result = await req.json(); And that's it. As a final step, I simply output it: console.log(`Summary of PDF:\n${result[0].summary}`); Given the source PDF, the final result is: Adobe has a vendor security review program that evaluates vendors that collect, store, process, 
transmit, or dispose of Adobe data outside of Adobe-controlled physical offices or data center 
locations. The VSR program includes requirements for vendors to follow when handling sensitive 
data and assigns a risk level score to vendors based on their compliance with Adobe standards. 
If a vendor fails the VSR program, Adobe holds discussions with the business owner to understand 
the details of the vendor's security practices and determine whether or not to continue working with them. My three-page PDF is now one simple paragraph. You can imagine how useful this would be for organizations with millions of documents. Combine this with other services (like the entities feature I mentioned previously), and it makes working with large libraries that much easier. Try It! If you want to check this out yourself, you can grab all the code here: . As I said, everything here can be tested for free, so give it a shot, and let me know what you think in a comment below. https://github.com/cfjedimaster/document-services-demos/tree/main/random_demos/extract_diffbot_summary

How to Use AI and PDF Services to Automate Document Summaries

নথির সারাংশ স্বয়ংক্রিয় করতে এআই এবং পিডিএফ পরিষেবাগুলি কীভাবে ব্যবহার করবেন

So nutzen Sie KI- und PDF-Dienste zur Automatisierung von Dokumentzusammenfassungen

Cómo utilizar servicios de IA y PDF para automatizar resúmenes de documentos

Comment utiliser les services AI et PDF pour automatiser les résumés de documents

दस्तावेज़ सारांशों को स्वचालित करने के लिए AI और PDF सेवाओं का उपयोग कैसे करें

AI および PDF サービスを使用して文書の要約を自動化する方法

AI 및 PDF 서비스를 사용하여 문서 요약을 자동화하는 방법

Como usar serviços de IA e PDF para automatizar resumos de documentos

Как использовать сервисы AI и PDF для автоматизации сводки документов

Belge Özetlerini Otomatikleştirmek için Yapay Zeka ve PDF Hizmetleri Nasıl Kullanılır?

Cách sử dụng dịch vụ AI và PDF để tự động hóa việc tóm tắt tài liệu

如何使用 AI 和 PDF 服务自动化文档摘要

Machine Learning: Your Ultimate Feature Selection Guide Part 2 - Select the Real Best

Introduction Time series data is a key component in a wide range of domains, including finance and meteorology. Effective feature engineering is critical in preparing time series data for machine learning models. In this article, we delve into advanced techniques for time series feature engineering, such as Fourier transform, wavelet transformation, derivatives, and autocorrelation. These techniques assist in uncovering hidden structures and trends, capturing both time and frequency information, and measuring the linear relationship between data points at varying lags. Techniques Fourier Transform Fourier transform is a mathematical technique that decomposes a time series signal into its frequency components. It's based on the principle that any signal can be broken down into a series of sinusoidal waves with varying amplitudes, frequencies, and phases. By capturing the periodic patterns in the data, the Fourier transform can help identify hidden structures and trends that might be useful for prediction. The Fourier Transform can be categorized into two types: the continuous Fourier Transform (CFT) for continuous signals and the discrete Fourier Transform (DFT) for discrete signals. Fast Fourier Transform (FFT) is an efficient algorithm for computing the DFT of a sequence. The power spectral density, obtained from the squared magnitudes of the Fourier transform, can be used as a feature in machine learning models to improve their performance, and here is a simple usage example for FFT: import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.fft import fft, ifft


time_series = np.random.random(200)

# Perform Fast Fourier Transform (FFT)
fft_values = fft(time_series)

# Get the magnitude and frequencies
fft_magnitude = np.abs(fft_values)
frequencies = np.fft.fftfreq(len(time_series))

# Plot the frequency spectrum
plt.plot(frequencies, fft_magnitude)
plt.xlabel('Frequency')
plt.ylabel('Magnitude')
plt.title('Frequency Spectrum')
plt.show()

# Filter out the low magnitude frequencies
threshold = 7
fft_values_filtered = fft_values.copy()
fft_values_filtered[fft_magnitude < threshold] = 0

# Perform the inverse Fast Fourier Transform (IFFT)
filtered_time_series = ifft(fft_values_filtered)

# Plot the original and filtered time series
plt.plot(time_series, label='Original')
plt.plot(filtered_time_series.real, label='Filtered')
plt.xlabel('Time')
plt.ylabel('Value')
plt.title('Time Series Forecasting')
plt.legend()
plt.show() Wavelet Transformation Wavelet transformation is a mathematical technique used to decompose a signal into different frequency components. It is particularly useful in time series analysis because it allows us to capture both time and frequency information in the data. Unlike a Fourier decomposition which always uses complex exponential (sine and cosine) basis functions, a wavelet decomposition uses a time-localized oscillatory function as the analyzing or mother wavelet. The most common wavelet transformation is the Continuous Wavelet Transform (CWT). CWT is defined as: The wavelet function can be chosen based on the type of data and the desired features. A popular choice is the Morlet wavelet, a product of a complex exponential and a Gaussian function, defined as: Scipy example to compute the CWT using the Morlet wavelet: from scipy import signal
import matplotlib.pyplot as plt

t = np.linspace(-1, 1, 200, endpoint=False)
sig = np.cos(2 * np.pi * 7 * t) + signal.gausspulse(t - 0.4, fc=2)
widths = np.arange(1, 31)

cwtmatr = signal.cwt(sig, signal.morlet2, widths)

real, imag = np.real(cwtmatr), np.imag(cwtmatr)

fig, axes = plt.subplots(ncols=2, figsize=(20, 5))
axes[0].imshow(
    real,
    extent=[-1, 1, 1, 31],
    cmap="PRGn",
    aspect="auto",
    vmax=abs(real).max(),
    vmin=-abs(real).max(),
)
axes[1].imshow(
    imag,
    extent=[-1, 1, 1, 31],
    cmap="PRGn",
    aspect="auto",
    vmax=abs(imag).max(),
    vmin=-abs(imag).max(),
) Derivatives Derivatives can be used to describe the rate of change in time series data. For example, the first derivative represents the velocity, while the second derivative represents acceleration. By incorporating these features, we can better capture the dynamics of the time series data. The first-order derivative represents the instantaneous rate of change of a variable with respect to time. It can help identify the direction and magnitude of the trend in the data. As it’s shown on the slide, it can be used as a marker for detecting changes in the stock or market behavior regimes. The second-order derivative represents the rate of change of the first-order derivative. It can help identify acceleration or deceleration in the trend and detect points of inflection. Also, here it needs to be mentioned, Seasonal derivatives and Cross-derivative: help to capture the seasonal variations in the data and pairwise variables influencing. To compute the derivatives, we can use the finite difference method: It’s the simplest method to compute the first derivative with and use as a feature: pandas time_series = pd.Series(data, index=dates)

# Calculate the first-order derivative
time_series_diff = time_series.diff().dropna()

# Combine the original time series and the first-order derivative
data = pd.concat([time_series.shift(1), time_series_diff], axis=1).dropna()
data.columns = ['y_t', 'first_derivative'] Here's an example of using the first-order derivative as an indicator for regime changes. When the derivative changes its sign, it signifies a change in the regime: Autocorrelation and Partial Autocorrelation Autocorrelation, also known as the autocovariance function or serial correlation, measures the linear relationship between the time series data points at different lags. Partial autocorrelation, also known as the partial autocovariance function, on the other hand, measures the correlation between two data points at different lags after removing the effect of other data points at smaller lags. Code example for autocorrelation and partial autocorrelation: from statsmodels.tsa.stattools 
import acf, pacf 
data = np.array([...])  
# Input time series data 
lags = 10 
autocorr = acf(data, nlags=lags) 
partial_autocorr = pacf(data, nlags=lags)

f, ax = plt.subplots(nrows=2, ncols=1, figsize=(width, 2*height))
plot_acf(data,lags=lag_acf, ax=ax[0])
plot_pacf(data,lags=lag_pacf, ax=ax[1], method='ols')
plt.tight_layout()
plt.show() Conclusion Advanced feature engineering techniques for time series data can significantly improve the performance of machine learning models. Fourier transform, wavelet transformation, derivatives, and autocorrelation each contribute unique insights into the underlying structure and trends of temporal data. By leveraging these techniques, analysts can create more accurate and efficient forecasting models, ultimately leading to better decision-making in various domains.

Advanced Techniques for Time Series Data Feature Engineering

This thread was originally published on Twitter by Aakriti A Zero to SQL 90 Days Roadmap TL;DR: ⛲ 1-15 : Syntax on ⛲ 15-30 : Case Studies on ⛲ 30-45 : DBMS on ⛲ 45-60 : Advanced SQL with Philip ⛲ 60-75 : Portfolio Projects ⛲ 75-90 : Practice on Mode.com selectstarsql.com javatpoint.com Datalemur.com Day 1-15 ⛲ Familiarize yourself with the Theory behind SQL syntax with Mode! I have a explaining solving these: https://youtube.com/playlist?list=PLvFCWsGNF7q6smZl7C7IbwlAwlAyqrfBk Day 15-30 ⛲ Practice SQL with inquisitive Case Studies at https://selectstarsql.com/ Day 30-45 ⛲ SQL is extremely important but knowing the language is only the tip of the iceberg. Learn various Database Management concepts that help you tackle real life problems and commonly asked interview questions! https://www.javatpoint.com/dbms-tutorial Day 45-60 ⛲ Advance into the world of SQL with Transactions and Triggers with Philip. http://philip.greenspun.com/sql/ Day 60-75 ⛲ Create a portfolio to showcase the recruiter with these videos! https://youtu.be/S2zBHmkRbhY https://youtu.be/MiAl2mQ718s Day 75-90 ⛲ Practice questions asked by FAANG on my favorite SQL practice website! https://datalemur.com/?referralCode=UBu6YFRp

The Roadmap to Learning SQL in 90 Days

We cannot solve our problems with the same thinking we used to create them -Albert Einstein If you have ever built a Machine Learning model, you've probably felt, Model deployment is a part of Machine Learning which isn't paid as much attention as it should. well, this was cool, but how will other people be able to see how cool it is? Well, in this article, I will introduce you to a new tool that will help you generate a web app for your Machine Learning model, which you can, of course, share with other individuals to try out. I will be building a simple neural network model using scikit-learn (not TF, sorry😒) and creating a GUI for the model using Gradio (this is the new cool tool I spoke about). Let us Begin What is Gradio? According to the , Gradio allows you to quickly create customizable UI components around your TensorFlow or PyTorch models or even arbitrary Python functions. Not terribly informative ei😅. Gradio website If you have ever used a python GUI library like Tkinter, Gradio is like that. Gradio is a GUI library that allows you to create customizable GUI components for your Machine Learning model. Pre-requisite For you to successfully work through this tutorial You must have python installed Let's Get Building You can check out the GitHub repo for the project here Let us begin Install required packages Packages Let us install the required packages pip install sklearn pip install pandas pip install numpy pip install gradio Getting our data Our data is going to be in the. CSV format. You can get the data by clicking here Importing Packages We are going to import the required packages import numpy as np

import pandas as pd

import gradio as gr Next, we are going to filter the warnings so we don't see them. import warnings

warnings.filterwarnings('ignore') Import the data Next, we are going to import our data data = pd.read_csv('diabetes.csv') Now let's see a little preview of our dataset. data.head() Let us see the feature columns in our dataset. print (data.columns) Getting our Variables Next, we get our X and Y variables x = data.drop(['Outcome'], axis=1)

y = data['Outcome'] Splitting the data Now we are going to split our data using the sci-kit-learn's inbuilt function. train_test_split from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(x,y) Scaling our data Next, we are going to scale our data using sci-kit-learn's inbuilt object. StandardScaler from sklearn.preprocessing import StandardScaler

#instantiate StandardScaler object
scaler = StandardScaler()

#scale data
x_train_scaled = scaler.fit_transform(x_train)

x_test_scaled = scaler.fit_transform(x_test) In the code above, we scaled our data using the StandardScaler object made available to us through sci-kit-learn. To know more about Scaling and why it is done, click here Instantiating and training the model In this section, we are going to create and train our model. The model we are going to make us of will be a Multi-Layer Perceptron Classifier, a neural network built into scikit-learn. #import model object
from sklearn.neural_network import MLPClassifier
model =  MLPClassifier(max_iter=1000,  alpha=1)

#train model on training data
model.fit(x_train_scaled, y_train)

#getting model performance on test data
print("accuracy:", model.score(x_test_scaled, y_test)) Creating the function for Gradio Now is the fun part. Here we are going to create a function that will take in the features of the data set which our model was trained on and pass it as an array to our model to predict. Then we are going to build our gradio web app based on that function. To understand why we have to write a function, you must first understand that gradio builds GUI components for our Machine Learning model based on the function. The function provides a way for gradio to get input from users and pass it on to the ML model, which will then process it and then pass it back to gradio, which then passes the result out. Let's Write... First, we will get the feature columns which we will then pass onto our function. #geting our columns

print(data.columns) Now we will create our function. def diabetes(Pregnancies, Glucose, Blood_Pressure, SkinThickness, Insulin, BMI, Diabetes_Pedigree, Age):
#turning the arguments into a numpy array  

 x = np.array([Pregnancies,Glucose,Blood_Pressure,SkinThickness,Insulin,BMI,Diabetes_Pedigree,Age])

  prediction = model.predict(x.reshape(1, -1))

  return prediction In the code above, we passed the feature columns from our data as arguments into a function we named . Then we turned the arguments into a NumPy array which we then passed onto our model for prediction, and finally, we returned the predicted result of our model. diabetes Creating our Gradio Interface Now we are going to create our Web App interface using Gradio. outputs = gr.outputs.Textbox()

app = gr.Interface(fn=diabetes, inputs=['number','number','number','number','number','number','number','number'], outputs=outputs,description="This is a diabetes model") The first thing we did above was to create a variable named outputs, which hold the GUI component for our model result. The result of our model's prediction will be outputted in a text box. Then we instantiated the Gradio interface object and passed in our earlier function. Then we generated our Inputs GUI component, and then we told the radio to expect 8 inputs in form of numbers. diabetes The inputs represent the feature columns that are present in our dataset, the same 8 feature column names we passed into our function. diabetes Then we passed our earlier output variable into the outputs parameter present in the object. Finally, we passed in the description of our web app into the description parameter. Launching the Gradio Web App Now we're going to Launch our Gradio web app. app.launch() If you are launching the Gradio app as a script from that command line, you will be given a local host link that you will copy and paste in our browser to see your web app. NOTE: If you are launching the app from a jupyter notebook, you will see a live preview of the app as you run the cell, and you will also be provided with a link. Hosting and sharing your Web App If you want to share your web app, all you have to do is put in as a parameter in your launch object. share=True #To provide a shareable link
app.launch(share=True) A link will then be provided for you with a .gradio extension. But this shareable link lasts for only 24 hours and will last if only your system is running. Because Gradio just hosts the web app on your system. In simple words, for your link to work, your system has to be on because gradio uses your system to host the web app and once your system is off. The server connection is severed, and you get a 500😅. Luckily for us, Gradio also provides a way for you to permanently host your model. But the service is subscription-based, so you have to pay $7 monthly to access it. Permanent hosting is way out of the scope o this article (partly because the author is broke😅). But if you are interested in it, click . here Important Links Gradio Website Gradio Documentation Gradio on GitHub Summary The Gradio Library is really cool, and it helps solve a huge problem plaguing the Machine Learning Community, Model Deployment. About 90% of Machine Learning models built are not deployed, and Gradio is working to fix that. It also serves as a way for beginners and experts to show off their models and also test the models in real life. You can't go wrong with the Gradio Library. Also Published Here

Build a GUI for Your Machine Learning Models

Introduction This article talks about the singleton Design Pattern, why it is important and then demonstrates how to build a singleton DBManager (a class that wraps all calls to the database) using dotConnect for PostgreSQL and C#. It also discusses the pros and cons of this approach. Pre-requisites You’ll need the following tools to deal with code examples: Visual Studio 2019 Community Edition ( ) download dotConnect for PostgreSQL ( ) download Introduction to Design Patterns Design Patterns can be used to solve recurring problems or complexities in software development. They are divided into three categories: structural, behavioral, and creational. Creational patterns can be used to create instances of classes and to manage them. However, structural patterns define the types and their relationships and help to understand the relationships between the entities. Behavioral Design Patterns emphasize on how objects collaborate and how responsibilities are delegated between them. What is a Singleton Design Pattern? The Singleton Design Pattern, as the name suggests, restricts the instantiation of a class to one object only. In other words, a class that follows the Singleton Design Pattern will not allow more than one instance of it to be created. Use Cases The typical use cases of the Singleton Design Pattern are: LogManager DbManager ServiceRegistry Create a new ASP.NET Core Web API Project Earlier, we mentioned the necessary tools to proceed to the practical scenarios. The time has come to use those tools. First, we need to create a new ASP.NET Core Web API project: Open Visual Studio 2019 Click . Create a new project Select and click Next. ASP.NET Core Web Application Specify the project name and location to store that project in your system. Optionally, checkmark the checkbox. Place solution and project in the same directory Click . Create In the window, select as the project template. Create a new ASP.NET Core Web Application API Select or as the version. ASP.NET Core 5 later Disable the options (uncheck them). Configure for HTTPS and Enable Docker Support Since we won’t use authentication in this example, specify authentication as No Authentication. Click to finish the process. Create We’ll use this project in this article. Implement a Singleton DBManager in ASP.NET Core and C# In this section we’ll implement a simple DBManager class that follows the singleton Design Pattern. Create a Database Table Create a new database table using the following script: CREATE TABLE books( book_id serial PRIMARY KEY, book_title VARCHAR (255) UNIQUE NOT NULL, book_pages INT NOT NULL ); We’ll use this database table throughout this article to store and retrieve data. Install NuGet Packages To get started you should install the dotConnect for PostgreSQL package in your project. You can install it either from the NuGet Package Manager tool inside Visual Studio or, from the NuGet Package Manager console using the following command: PM>Install-Package Devart.Data.PostgreSql If the installation is successful, you're all set to get started using dotConnect for PostgreSQL in your application. Configuring the Application You should specify the database connection string in a config file, i.e., appsettings.json and then read the connection string in your application. Replace the default generated code of the appsettings.json with the following code: { "PostgreSqlConnectionString": { "DefaultConnection": "UserId = postgres; Password = mypass;host=localhost;database=Test;" }, "AllowedHosts": "*" } You should also add the IConfiguration instance to the services container so that you can access it from anywhere in the application. publicvoid ConfigureServices(IServiceCollection services) { services.AddSingleton(Configuration); services.AddControllers(); } Create the DBManager Class To keep things simple, we'll build a DBManager with minimal features. Now, create a class called DBmanager inside a fie having the same primary name with a .cs extension and write the following code in there: public class DBManager { public int ExecuteNonQuery(string commandText) { return 1; } public int ExecuteReader(string commandText) { return 1; } } Read Data from the Database To read data from the database using dotConnect for PostgreSQL, we can create a PgSqlDataAdapter instance and then use it to fill a data table and return it. The ExecuteReader method shown in the following code snippet illustrates how this can be achieved: public DataTable ExecuteReader(string commandText) { DataTable dataTable = new DataTable(); using (PgSqlConnection pgSqlConnection = new PgSqlConnection(ConnectionString)) { Using (PgSqlCommand pgSqlCommand = new PgSqlCommand() { pgSqlCommand.CommandText = commandText; pgSqlCommand.Connection = pgSqlConnection; if (pgSqlConnection.State != ConnectionState.Open) pgSqlConnection.Open(); PgSqlDataAdapter pgSqlDataAdapter = new PgSqlDataAdapter(pgSqlCommand); pgSqlDataAdapter.Fill(dataTable); if(dataTable.Rows.Count > 0) return dataTable; return null; } } } The ExecuteReader method of the DBManager class returns an instance of a DataTable. Now, add the following property in the DBManager class to store the database connection string: public string ConnectionString { get;set; } Return a List from the ExecuteReader Method You can also return a List from the ExecuteReader method. Create a class named Book with the following content in there: public class Book { public int book_id { get; set; } public string book_title { get; set; } public int book_pages { get; set; } } Replace the source code of the ExecuteReader method of the DBManager class with the following code: public List<Book> ExecuteReader(string commandText) { DataTable dataTable = new DataTable(); using (PgSqlConnection pgSqlConnection = new PgSqlConnection(ConnectionString)) { using (PgSqlCommand pgSqlCommand = new PgSqlCommand()) { pgSqlCommand.CommandText = commandText; pgSqlCommand.Connection = pgSqlConnection; if (pgSqlConnection.State != ConnectionState.Open) pgSqlConnection.Open(); PgSqlDataAdapter pgSqlDataAdapter = new PgSqlDataAdapter(pgSqlCommand); pgSqlDataAdapter.Fill(dataTable); return dataTable.ToList<Book>(); } } } Note that the return type has been changed from DataTable to List<Book> and how an instance of List<Book> is created using an extension method called ToList<T>(). The following code listing illustrates a class named Extensions that contains the ToList extension method. public static class Extensions { public static List<T> ToList<T>(this DataTable dataTable) where T : new() { List<T> data = new List<T>(); foreach (DataRow row in dataTable.Rows) { T item = GetItemFromDataRow<T>(row); data.Add(item); } return data; } public static T GetItemFromDataRow<T>(DataRow dataRow) { Type temp = typeof(T); T obj = Activator.CreateInstance<T>(); foreach (DataColumn column in dataRow.Table.Columns) { foreach (PropertyInfo propertyInfo in temp.GetProperties()) { if (propertyInfo.Name == column.ColumnName) propertyInfo.SetValue(obj, dataRow[column.ColumnName], null); } } return obj; } } Note how an instance of a DataTable is converted to an instance of List<T>. The GetItemFromDataRow method returns an instance of the Book class. Insert Data to the Database To insert data to the PostgreSQL database using dotConnect for PostgreSQL, you can use the following method: public int ExecuteNonQuery(string commandText) { using (PgSqlConnection pgSqlConnection = new PgSqlConnection(ConnectionString)) { using (PgSqlCommand pgSqlCommand = new PgSqlCommand()) { pgSqlCommand.CommandText = commandText; pgSqlCommand.Connection = pgSqlConnection; if (pgSqlConnection.State != ConnectionState.Open) pgSqlConnection.Open(); return pgSqlCommand.ExecuteNonQuery(); } } } The Complete Source Code The complete source code of the DBManager class is given below: public class DBManager { public string ConnectionString { get;set; } public List<Book> ExecuteReader(string commandText) { DataTable dataTable = new DataTable(); using (PgSqlConnection pgSqlConnection = new PgSqlConnection(ConnectionString)) { using (PgSqlCommand pgSqlCommand = new PgSqlCommand()) { pgSqlCommand.CommandText = commandText; pgSqlCommand.Connection = pgSqlConnection; if (pgSqlConnection.State != ConnectionState.Open) pgSqlConnection.Open(); PgSqlDataAdapter pgSqlDataAdapter = new PgSqlDataAdapter(pgSqlCommand); pgSqlDataAdapter.Fill(dataTable); return dataTable.ToList<Book>(); } } } public int ExecuteNonQuery(string commandText) { using (PgSqlConnection pgSqlConnection = new PgSqlConnection(ConnectionString)) { using (PgSqlCommand pgSqlCommand = new PgSqlCommand()) { pgSqlCommand.CommandText = commandText; pgSqlCommand.Connection = pgSqlConnection; if (pgSqlConnection.State != ConnectionState.Open) pgSqlConnection.Open(); return pgSqlCommand.ExecuteNonQuery(); } } } } The Singleton DBManager Class To make the DBManager class a singleton class, you should have a private constructor. A private constructor would prevent the class from being extended or instantiated. Since we don't want more than one instance of this class, a private constructor is needed. You should then create an instance of the DBManager inside a static property as shown in the code snippet given below: public class DBManager { private static DBManager instance; private DBManager() { } public static DBManager Instance { get { if (instance == null) { instance = new DBManager(); } return instance; } } //Other methods removed for brevity } Since the DBManager instance is static, you would have only one instance throughout the lifetime of the application. You can access the DBManager class, and its methods as shown in the following code snippet: DBManager.Instance.ConnectionString = connectionString; return DBManager.Instance.ExecuteReader("select * from public.books"); Make the DBManager Class ThreadSafe But what if you would like to make it thread safe so that no two threads can have access to the critical section? To make this class thread safe, you can take advantage of the lock keyword as illustrated in the following code snippet: public class DBManager { private static object lockObj = new object(); private static DBManager instance; private DBManager() { } public static DBManager Instance { get { lock(lockObj) { if (instance == null) { instance = new DBManager(); } } return instance; } } //Other methods removed for brevity } Use the DBManager Class in Controller Classes You can now use the DBManager instance in your controller class like this: [Route("api/[controller]")] [ApiController] public class BooksController : ControllerBase { readonly IConfiguration _configuration; readonly string connectionString; public BooksController(IConfiguration configuration) { _configuration = configuration; connectionString = _configuration["PostgreSqlConnectionString:DefaultConnection"]; } [HttpGet] public List<Book> Get() { DBManager.Instance.ConnectionString = connectionString; return DBManager.Instance.ExecuteReader("select * from public.books"); } } Summary A DBManager class serves as a helper to connect to databases and perform required CRUD operations. Ideally, it should be a singleton since you don’t need multiple instances of it in your application. This article has demonstrated how we can build a thread-safe DBManager class in C# and use it in an ASP.NET Core application.

Implement a Singleton DBManager in ASP.NET Core and C#

Building a machine learning model requires a series of steps, from data preparation, data cleaning, feature engineering, model building to model deployment. Therefore, it can take a lot of time for a data scientist to create a solution that solves a business problem. To help speed up the process, you can use Pycaret, an open-source library. Pycaret can help you perform all the end-to-end processes of ML faster with few lines of code. What is Pycaret? Pycaret is an open-source, low code library in python that aims to automate the development of machine learning models. This library is useful for any data scientist, analyst, ML engineer, or anyone learning machine learning to be more productive and reach conclusions faster. The library has 70+ automated open-source algorithms and over 25+ pre-processing techniques that can help you build machine learning models with good performance. It supports supervised learning (classification and regression), clustering, anomaly detection, and natural language processing tasks. PyCaret is a Python wrapper around several libraries and frameworks such as and many more. machine learning scikit-learn, XGBoost, Microsoft LightGBM, spaCy, XGBoost, Optuna, Hyperopt, Ray, You don't have to worry about data preparation, feature engineering, , or hyperparameter tuning. Pycaret can perform all these tasks automatically with just a few lines of code. feature selection Another benefit of the library is that after building your machine learning model you can directly deploy the transformation pipeline and trained model on Amazon Web Service (AWS), Microsoft Azure, or Google Cloud Platform(GCP). For classification and regression problems, Pycaret uses the following evaluation metrics: Accuracy, AUC, Recall, Precision, F1, Kappa For Classification: MAE, MSE, RMSE, R2, RMSLE, MAPE For Regression: In this article, you will learn how to use the Pycaret library to automate the end-to-end process of machine learning faster with little manual configuration. How to Install Pycaret Installation is easy and takes only a few minutes. All dependencies are also installed with PyCaret.You can view a list of dependencies . here pip install pycaret Load Data In this tutorial, we will use "mobile price datasets" and the goal is to predict a price range indicating how high the price is. You can download the dataset : here You can load the dataset by using the pandas library. pandas pd numpy np data = pd.read_csv( )
data.head() # import packages import as import as #load data "/train.csv" Let's check the shape of the dataset. data.shape (2000,21) As you can see, the dataset has 20 features and 1 target. Prepare the Environment The first step you need to do is preparing the environment to run your machine learning experiments. You need to initialize the setup() function from pycaret.classification module. In the setup function, you need to define the dataframe for your dataset and the target variable, for this problem is price_range. You can also set the experiment name and other settings. pycaret.classification * grid = setup(data=data, target=data.columns[ ], html= , silent= , verbose= ,log_experiment = , experiment_name = ) from import # setup the environment -1 False True True True 'mobile_prices' As I have said before, Pycaret handles all data preprocessing automatically and these steps are applied within setup() and all the operations performed in PyCaret are sequentially stored in a Pipeline. Note: Create a Model. To create a model in pycaret is very simple and straightforward. You need to add only one parameter i.e the model name in the create_model() function. The create_model will train the algorithm and return a table with k-fold cross-validated scores and the means from different evaluation metrics such as accuracy and F1. In this example, we can train K Neighbors classifier by passing the string input called “knn”. You can to see a complete list of more than 60 estimators available in the Pycaret library. click here knn = create_model( ) #create model 'knn' As you can see the mean accuracy is 91.85%. Compare Different Models With Pycaret, you can train and evaluate the performance of all estimators available in the model library using K-fold validation. The compare_models() function will return a score grid with average cross-validated scores from all estimators. best = compare_models() The table above is sorted by using accuracy metric and the estimators that perform well is followed by Linear Discriminant Analysis. K Neighbors Classifier Sometimes accuracy is not a good evaluation metric depending on the nature of your dataset. You can choose other evaluation metrics to determine which model performs better than others. Tune Model You can also improve the performance of your model by tuning its hyperparameters. The tune_model() function from Pycaret can automatically tune the hyperparameter of a machine learning model by using different search algorithms such as: Random Search Grid Search Bayesian search Tree-structured Parzen Estimator search Now we can tune the KNN model to improve its performance. tuned_knn = tune_model(knn) #tune model The output of the function is a score grid with CV scores and the trained model object. After tuning the hyperparameters of the KNN model, the performance has improved from to . 91.85% 93.00% Model Evaluation You can evaluate your trained model by using the evaluate_model() function from Pycaret. The function displays a user interface for analyzing the performance of a trained model. evaluate_model(tuned_knn) You can view the following plots and other performance details such as: Hyperparameters of the trained models. confusion matrix. Precision Recall Class report (for classification problem) Learning curve Decision Boundary Error Validation Curve This function only works in IPython enabled Notebook. Note: Make Prediction To make a prediction on unseen data, you can use the predict_model() function. For a classification problem, the function predicts Label and Score (probability of predicted class) using a trained model. When data is none, it predicts label and score on the test or holdout set which is 30% of the dataset (by default). holdout_prediction = predict_model(tuned_knn) The tuned KNN model still performs well on the test set with an accuracy of . 93.84% Save Trained Model After training and doing a lot of machine learning experiments to get the best performance, you can save the entire pipeline containing all preprocessing steps and trained model object as a binary pickle file by using the save_model() function. You need to pass the trained model object and the name of the model that will be used to create a pickle file. save_model(tuned_knn, model_name = ) # saving model 'knn_model' Final Thoughts on Pycaret Library In this article, you have learned the most important steps to build machine learning models by using the Pycaret library. The library has a lot of modules and examples to help you build machine learning models in different cases. Check the following resources if you are looking to go deeper. Regression Clustering Anomaly Detection Natural Language Processing Association Rule Mining If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in python? in the footer below. Subscribe to our newsletter

Pycaret: A Faster Way to Build Machine Learning Models

Feature Selection is the process where you automatically or manually select those features which contribute most to your prediction variable or output in which you are interested. One major reason is that machine learning follows the rule of and that is why you need to be very concerned about the features that are being fed to the model. “garbage in garbage out” Having irrelevant features in your data can both increase the computational cost of modeling and the of the models and make your model learn based on irrelevant features. This means you need to select only to be presented during model training. decrease accuracy important features Top reasons to apply feature selection: It enables the machine learning algorithm to train faster. It reduces the complexity of a model and makes it easier to interpret. It improves the accuracy of a model if the right subset is chosen. It reduces overfitting. “I prepared a model by selecting all the features and I got an accuracy of around which is not pretty good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to which is quite impressive”- By Raheel Shaikh 65% 81% are intended to reduce the number of features to those that are believed to be most useful/important to a model in order to predict the target feature. There are different feature selection methods available that can help you to select important features. The most popular methods are. Feature selection methods .Spearman’s rank coefficient. Pearson’s correlation coefficient ANOVA correlation coefficient. Chi-Squared test. Machine Learning Algorithms (Random Forest and Extra Tree). Correlation Matrix with Heatmap. Mutual Information. You can read more about feature selection methods from . Scikit learn library One of the challenges of these methods is to which method(s) should you apply in your dataset to select important features. Each method has its own way to identify the important features. For example, a certain feature can be selected as an important feature in one method and not selected as an important feature in another method. identify Xverse package can help you to solve this problem. “At the end of the day, some machine learning projects succeed and some fail.What makes the difference? Easily the most important factor is the features used.” — Pedro Domingos What is Xverse? Xverse stands for X Universe which is the python package for machine learning to assist Data Scientists with feature transformation and feature selection. Xverse is created by . Sundar Krishnan How does it work? Xverse applies a variety of techniques to select features. When an algorithm picks a feature, it gives a vote for that feature. In the end, Xverse calculates the total votes for each feature and then picks the best ones based on votes. This way, we end up picking the best variables with minimum effort in the features selection process. Xverse uses the following methods to select important features. Information Value using Weight of evidence. Variable Importance using Random Forest. Recursive Feature Elimination. Variable Importance using Extra trees classifier. Chi-Square best variables. L1-based feature selection. Installation The package requires Numpy, Pandas, Scikit-learn, Scipy and Statsmodels. In addition, the package is tested on Python version 3.5 and above. Run the following command to install Xverse. pip xverse install I will use the Loan dataset to find the best features that can help to get good accuracy when predicting if a customer deserves to get a loan or not. You can download the dataset . here Import important packages for this problem. pandas pd numpy np xverse.ensemble VotingSelector sklearn.preprocessing StandardScaler, MinMaxScaler warnings warnings.filterwarnings( ) import as import as from import from import import # To ignore any warnings "ignore" Load the Loan dataset. data = pd.read_csv( )

data.columns "data/loans_data.csv" Loan_ID
Gender
Married
Dependents
Education
Self_Employed
ApplicantIncome
CoapplicantIncome
LoanAmount
Loan_Amount_Term
Credit_History
Property_Area
Loan_Status We have 12 independent features and a target (Loan_Status). You can read the description of each feature here. I have created a simple python function to handle missing data and feature engineering. data[ ].replace( , ,inplace= )
	data[ ].replace( , ,inplace= )
	data[ ].replace( , ,inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].median(), inplace= ) data = data.drop( ,axis= ) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data : def preprocessing (data) # replace with numerical values 'Dependents' '3+' 3 True 'Loan_Status' 'N' 0 True 'Loan_Status' 'Y' 1 True # handle missing data 'Gender' 'Gender' 0 True 'Married' 'Married' 0 True 'Dependents' 'Dependents' 0 True 'Self_Employed' 'Self_Employed' 0 True 'Credit_History' 'Credit_History' 0 True 'Loan_Amount_Term' 'Loan_Amount_Term' 0 True 'LoanAmount' 'LoanAmount' True # drop ID column 'Loan_ID' 1 #scale the data "ApplicantIncome" "ApplicantIncome" -1 1 "LoanAmount" "LoanAmount" -1 1 "CoapplicantIncome" "CoapplicantIncome" -1 1 "Loan_Amount_Term" "Loan_Amount_Term" -1 1 return Let’s preprocess the loan dataset. data = prepocessing(data) Split into independent features and target. X = data.drop( ,axis = )
y = data.Loan_Status 'Loan_Status' 1 Now is time to call VotingSelector from Xverse and train it to find the best features by using the voting approach. clf = VotingSelector(minimum_votes= )
clf.fit(X, y) #train to find the best features 2 I have set , this means the feature to be selected must have at least a total of from the six feature selection methods presented in Xverse. Features below a total of will be neglected. minimum_votes= 2 2 votes 2 votes After training, we can see the feature importance in each feature selection method used during the training. clf.feature_importances_ #show important features The output shows all the features and their values of importance in each method. Now let’s observe the votes from these feature selection methods. clf.feature_votes_ # votes The output shows the and at last, it shows for each feature. It starts by show features with many votes to the features with low or zero votes. variable name, list of feature selection methods and their votes total votes You can see that has a total of which means credit_History is a very important feature for this Loan problem. But both and features have which means we can neglect these two features because of the very low contribution to the prediction if a customer deserves to get a loan or not. Credit_History 6 votes, Gender Self_employed 0 votes, Now we can transform our data to remain with only important selected features. X = clf.transform(X) # transform your data into important features Conclusion Xverse is under active development. Currently, xverse package handles only binary targets. The code for this post is available on . Github If you learned something new or enjoyed reading this article, please share it so that others can see it. Feel free to leave a comment too. Till then, see you in the next post! I can also be reached on Twitter @Davis_McDavid Previously published . here

Features Selection by Using Xverse Package

This is the second and last part of my series which focuses on Anomaly Detection using Machine Learning. If you haven't already, I recommend you read my first article which will introduce you to Anomaly Detection and its applications in the business world. here In this article, I will take you through a case study focus on . It is important that credit card companies are able to recognize fraudulent credit card transactions so that customers are not charged for items that they did not purchase. So the main task is to identify fraudulent credit card transactions by using Machine learning. We are going to use a Python library called which is specifically developed for anomaly detection purposes. Credit Card Fraud Detection PyOD What is the PyOD Library? PyOD is a comprehensive and scalable for in multivariate data. It has around 20 outlier detection algorithms (supervised and unsupervised). PyOD is developed with a comprehensive API to support multiple techniques and you can take a look at the official documentation of PyOD . Python toolkit detecting outlying objects here If you are an anomaly detection professional or you want to learn more about anomaly detection then I recommend you try using the PyOD Toolkit. Features of PyOD PyOD has useful features such as : Unified APIs, detailed documentation, and interactive examples across various algorithms. Advanced models, including Neural Networks/Deep Learning and Outlier Ensembles. Optimized performance with JIT and parallelization when possible, using and . numba joblib Compatible with both Python 2 & 3.(support for python 2 ended January 2020). Installing PyOD in Python Let’s first install PyOD on our machines. pip install pyod pip install --pre pyod # normal install # pre-release version for new features Alternatively, you could clone and run the setup.py file. git clone https://github.com/yzhao062/pyod.git
cd pyod
pip install . If you plan to use Neural Network-based Models in Pyod, you have to install Keras and other libraries manually in your machine. Credit Card Fraud Detection Case Study The dataset we will use contains transactions made by credit cards in September 2013 by European cardholders. The dataset has been collected and analyzed during a research collaboration of Worldline and the of ULB (Université Libre de Bruxelles) on big data mining and fraud detection. Machine Learning Group Now let us see how we can use the PYOD library in this case study. We will start by importing important packages such as pandas, numpy,sklearn and pyod. pandas pd numpy np scipy sklearn matplotlib.pyplot plt seaborn sns sklearn.model_selection train_test_split sklearn.metrics classification_report, accuracy_score,confusion_matrix sklearn.ensemble IsolationForest sklearn.neighbors LocalOutlierFactor pyod.models.knn KNN pyod.models.ocsvm OCSVM pyod.utils.data evaluate_print pyod.utils.example visualize sklearn.preprocessing StandardScaler cf_matrix make_confusion_matrix 
%matplotlib inline warnings 
warnings.filterwarnings( ) np.random.seed( ) # Import important packages import as import as import import import as import as from import from import from import from import # Importing KNN module from PyOD from import from import # Import the utility function for model evaluation from import from import from import from import import 'ignore' # set seed 123 The dataset for this case study can be downloaded . here Let's load the dataset. data = pd.read_csv( ) # Load the dataset from csv file by using pandas "creditcard.csv" Check columns in the dataset. data.columns # show columns The dataset contains 31 columns, only 3 columns make sense which are Time, Amount, and Class (fraud or not fraud). The rest of the 28 columns were transformed using PCA dimensionality reduction in order to protect user identities. data.shape

( , ) # print the shape of the data 284807 31 The dataset contains 284807 rows and 31 columns as explained before. data.head() # show the first five rows You can see all transformed columns are named from V1 to V28. Let's check if we have any missing values in our dataset. data.isnull().sum() #check missing data We don't have any missing values in our dataset. Our target column is Class contains two classes which are fraud labeled as and not fraud labeled as 1 0. data.Class.value_counts(normalize= ) # determine number of fraud cases in our  file True In this dataset, there are only 0.173% (total of 492 )of fraud transactions and 99.82% (total of 284,315) of valid transactions. We can observe if variables in the dataset are correlated to each other by using the heatmap plot implemented in the seaborn library. corr = data.corr() 

fig = plt.figure(figsize=( , ))

sns.heatmap(corr, vmax= , square= ,annot= ) #find the correlation betweeen the variables 30 20 .8 True True The above correlation graph shows that has a strong positive correlation to the while the has a strong negative correlation to the V11 variable Class variable V17 variable Class variable. Because we have many valid transactions, we will use all 10,000 valid cases and 492 fraud cases to create our models. positive = data[data[ ]== ]
negative = data[data[ ]== ]


print( .format(len(positive)))
print( .format(len(negative)))

new_data = pd.concat([positive,negative[: ]]) new_data = new_data.sample(frac= ,random_state= )


new_data.shape # use sample of the dataset "Class" 1 "Class" 0 "positive:{}" "negative:{}" 10000 #shuffling our dataset 1 42 : 492 Positive : 284315 Negative (10492,31) Now we have a total of 10492 numbers of rows. We will standardize the Amount variable by using the method from sklearn. StandardSclaer transforms the data to where there is a mean of 0 and a standard deviation of 1, which means standardizing the data into a normal distribution. standardScaler new_data[ ] = StandardScaler().fit_transform(new_data[ ].values.reshape( , )) #Normalising the amount column. 'Amount' 'Amount' -1 1 Separate the dataset into independent variables and target variable (class variable). NB. we are not going to use the time variable in this article. X = new_data.drop([ , ], axis= ) 
y = new_data[ ] print( .format(X.shape)) 
print( .format(y.shape))
X shape: ( , )
y shape: ( ,) # split into independent variables and target variable 'Time' 'Class' 1 'Class' # show the shape of x and y "X shape: {}" "y shape: {}" 10492 29 10492 Split the dataset into train and test sets. We will only use 20% of the dataset for the test set and the rest will be the train set. X_train, X_test, y_train,y_test = train_test_split(X,y, test_size = , stratify=y, random_state= ) #split the data into train and test 0.2 42 Creating Models We will create two outlier detectors from PyOD library which are K-Nearest Neighbors Detector and One-class SVM detector. 1. k-Nearest Neighbors Detector In KNN detector for any observation, its distance to its kth nearest neighbor could be viewed as the outlying score. PyOD supports three detectors: kNN Uses the distance of the kth neighbor as the outlier score. Largest: Uses the average of all k neighbors as the outlier score. Mean: Uses the median of the distance to k neighbors as the outlier score. Median: clf_knn = KNN(contamination= , n_neighbors = ,n_jobs= )

clf_knn.fit(X_train) # create the KNN model 0.172 5 -1 The two parameters we passed into KNN() are The amount of anomalies in the data which for our case = 0.0172 contamination: Number of neighbors to consider for measuring the proximity. n_neighbors: After training our KNN Detector model, we can get the prediction labels on the training data and then get the outlier scores of the training data. The higher the scores are, the more abnormal. This indicates the overall abnormality in the data. These features make a great utility for anomaly detection tasks. PyOD y_train_pred = clf_knn.labels_ y_train_scores = clf_knn.decision_scores_ # Get the prediction labels of the training data # binary labels (0: inliers, 1: outliers) # Outlier scores We can evaluate with respect to the training data. provides a handy function for this task called The default metrics include and Precision @ n. We will pass class name, y_train values and y_train_scores(outlier scores as returned by a fitted model.) KNN() PyOD evaluate_print(). ROC valuate_print(‘KNN’, y_train, y_train_scores) # Evaluate on the training data e KNN ROC: 0.9566, precision @ rank n:0 0.5482. We see that the model has a good performance on the training data. Let’s plot the confusion matrix for the train set. KNN() scikitplot skplt skplt.metrics.plot_confusion_matrix(y_train,y_train_pred, normalize= ,title=”Consfusion Matrix on Train Set”)
 plt.show() import as # plot the comfusion matrix in the train set False 372 fraud cases were predicted correctly and only 22 cases were predicted incorrectly as valid cases in the train set. We will use decision_function to predict anomaly scores of the test set using the fitted detector(KNN Detector) and evaluate the results. y_test_scores = clf_knn.decision_function(X_test) evaluate_print( , y_test,y_test_scores) # outlier scores # Evaluate on the training data 'KNN' KNN ROC:0.9393, precision @ rank n:0.5408 Our model continues to perform well on the test set. Let’s plot the confusion matrix for the test set. KNN() y_preds = clf_knn.predict(X_test)

skplt.metrics.plot_confusion_matrix(y_test,y_preds, normalize= ,
                                    title= )
plt.show() # plot the comfusion matrix  in the test set False "Consfusion Matrix on Test Set" 87 fraud cases were predicted correctly and only 11 cases were predicted incorrectly as valid cases in the test set. 2. One-class SVM detector This is an unsupervised Outlier detection algorithm and a wrapper of scikit-learn one-class SVM Class with more functionalities. Let's create a Once-class SVM model. clf_ocsvm = OCSVM(contamination= )

clf_ocsvm.fit(X_train) # create the OCSVM model 0.172 After training our OCSVM Detector model, we can get the prediction labels on the training data and then get the outlier scores of the training data. y_train_pred = clf_ocsvm.labels_ clf_name = y_train_scores = clf_ocsvm.decision_scores_ evaluate_print(clf_name, y_train, y_train_scores) # Get the prediction labels of the training data # binary labels (0: inliers, 1: outliers) 'OCSVM' # Outlier scores # Evaluate on the training data OCSVM ROC:0.9651, precision @ rant n:0.7132 OCSVM model performs better than KNN model on the train set. Let’s plot the confusion matrix for the train set. skplt.metrics.plot_confusion_matrix(y_train,y_train_pred, 
                                    normalize= ,
                                    title= )
plt.show() # plot the comfusion matrix  in the train set False "Consfusion Matrix on                 
                                    Train Set" 373 fraud cases were predicted correctly and only 21 cases were predicted incorrectly as valid cases in the train set. We will use decision_function to predict anomaly scores of the test set using the fitted detector(OCSVM Detector) and evaluate the results. y_test_scores = clf_ocsvm.decision_function(X_test) evaluate_print(clf_name, y_test,y_test_scores) # outlier scores # Evaluate on the training data OCSVM ROC: 0.9571, precision @ rank n:0.6633 Our OCSVM model continues to perform well on the test set. Let’s plot the confusion matrix for the test set. y_preds = clf_ocsvm.predict(X_test) 
skplt.metrics.plot_confusion_matrix(y_test,y_preds, normalize= , title=”Consfusion Matrix on Test Set”, ) plt.show() # plot the comfusion matrix in the test set False 92 fraud cases were predicted correctly and only 6 cases were predicted incorrectly as valid cases in the test set. In general, when you compare these two models, we observed that the OCSVM model performs better than the KNN model. There is more that can be done to increase the performance of the best model (OCSVM) for detecting fraud transactions. You can also try to use other detector algorithms found on . PyOD documentation Conclusion It is important that credit card companies are able to recognize fraudulent credit card transactions so that customers are not charged for items that they did not purchase. As a business owner, you can avoid serious headaches and unwanted publicity by recognizing potentially fraudulent use of credit cards in your payment environment. The source code for this article is available on Github. https://github.com/Davisy/Credit-Card-Fraud-Detection-using-PYOD-Library If you learned something new or enjoyed reading this article, please share it so that others can see it.I look forward to hearing your experience using PyOD Library as well. I can also be reached on Twitter @Davis_McDavid Also published at https://medium.com/analytics-vidhya/introduction-to-anomaly-detection-using-machine-learning-with-a-case-study-part-two-f78243f74d2f

Credit Card Fraud Detection via Machine Learning: A Case Study

Have you ever wondered how the machine learning models are constructed? Today we will explore this and learn some quick techniques on how to find out which variables are influencing the model results and by how much. We will use the FIFA 2018 on Kaggle and explore the following models: dataset Decision Tree model Random Forest model This will be the agenda for today: Use the FIFA dataset to train the decision tree model Use the FIFA dataset to train the random forest model Explore the influential variables in the models Find the threshold of the influential variables So without further ado let's get started. 1. Use the FIFA dataset to train the decision tree model Let us first talk a bit about decision trees. Decision tree algorithms start with a root node from a data sample and then select features based on metrics like Gini impurity or information gain and splits the root nodes into leaf nodes/end nodes till no more split is possible. This is illustrated in the diagram below with a sample tree. After the data and the libraries have been imported, the following lines of code will help to train the decision tree model. y = (data[ ] == ) feature_names = [i i data.columns data[i].dtype [np.int64]]
x = data[feature_names] train_x, test_x, train_y, test_y = train_test_split(x,y, random_state= )
dt_model = DecisionTreeClassifier(random_state= , max_depth= , min_samples_split= ).fit(train_x,train_y)

pred_y = dt_model.predict(test_x)

cm = confusion_matrix(test_y,pred_y)
print(cm)
accuracy_score(test_y,pred_y) #Create the dependent variable 'Man of the Match' "Yes" #Create the independent variable for in if in #Train the decision tree model 1 0 5 5 We will get the following output from the confusion matrix: [[ ]
 [ ]] 9 7 6 10 0.59375 The accuracy of the decision tree model is moderate at 59.38% with (10+9) targets being corrected predicted and (7+6) being false positives and false negatives respectively. 2. Use the FIFA dataset to train the random forest model Let us now learn a bit about the random forest model and then train the data with it. is an ensemble learning algorithm that works by constructing multiple decision trees and outputs the class that is either the mode or the mean prediction of the individual decision trees. Random forest An illustration is given below: We will now use the code below to train the random forest model. # Train the RF model

rf_model = RandomForestClassifier(n_estimators= , random_state= ).fit(train_x,train_y)

pred_y = rf_model.predict(test_x)

cm = confusion_matrix(test_y,pred_y)
print(cm)
accuracy_score(test_y,pred_y) 100 1 The output of the Random forest model is given below: [[ ]
 [ ]] 10 6 3 13 0.71875 The random forest model has a better accuracy at 71.88% with (10+13) targets identified correctly and (6+3) targets mis-classified-6 being false positives and 3 being false negatives. 3. Explore the influential variables in the models We will now look at the most influential variables in both the models and how they are affecting the accuracy. We will use ' ' from the ' ' library for this purpose. We can do this with only one line of code as given below. PermutationImportance eli5 # Import PermutationImportance the eli5 library eli5.sklearn PermutationImportance

# Influential variables Decision Tree model

eli5.show_weights(perm, feature_names = test_x.columns.tolist()) from from import for The influential variables in the decision tree model is : The most influential variables in the decision tree model is 'Goal scored', 'On-target', 'Distance Covered (Kms)' and 'Off-Target'. There are also variables that influence the accuracy negatively like 'Attempts' and 'Corners' - hence we can drop these variables from the model to increase the accuracy. Some variables like 'Red', 'Ball Possession %' etc has no influence on the accuracy of the model. The weights indicate by how much percentage the model accuracy is impacted by the variable when the variables are re-shuffled. For eg: By using the feature 'Goal Scored'  the model accuracy can be improved by 14.37% in a range of (+-) 11.59% . The influential variables in the random forest model is : As you can observe there are significant differences in the variables that influence the 2 models and for the same variable like say 'Goal Scored' the percentage of change in accuracy also differs. 4. Find out the threshold of the influential variable at which the changes to model accuracy is happening Let us now take one variable say 'Distance Covered (Kms)' and try to find out the threshold at which the accuracy increases. We can do this easily with Partial dependence plots (PDP). A partial dependence (PD) plot depicts the functional relationship between input variables and predictions. It shows how the predictions partially depend on values of the input variables. For eg: We can create a partial dependence plot of the variable 'Distance Covered (Kms)' to understand how changes in the values of the variable 'Distance Covered (Kms)' affects overall accuracy of the model. We will start with the decision tree model first. matplotlib pyplot plt pdpbox pdp, get_dataset, info_plots feature_to_plot = pdp_dist = pdp.pdp_isolate(model=dt_model,dataset=test_x,
                           model_features=feature_names,
                           feature= feature_to_plot)

pdp.pdp_plot(pdp_dist, feature_to_plot)
plt.show() # Import the libraries from import as from import # Select the variable/feature to plot 'Distance Covered (Kms)' # PDP plot for Decision tree model The plot will look like this: If distance covered is 102 KM, then that influences the model positively, but if >102 Km is covered or <102 Km then that does not influence the model. The PDP (Partial dependence plot) helps to provide an insight into the threshold values of the features that influence the model . Now we can use the same code for the random forest model and look at the plot : For the random forest model, the plot looks a bit different and here the performance of the model increases when the distance covered is 99 till about 102; post which the variables has little or no influence on the model as given by the declining trend and the flat line henceforth. Summary: This is how we can use simple PDP plots to understand the behaviour of influential variables in the model. This information can not only draw insights about the variables that impact the model but is especially helpful in training the models and for selection of the right features. The thresholds can also help to create bins that can be used to sub-set the features that can further enhance the accuracy of the model. Please refer to this on Github for the full code. link Do reach out to me in case of any questions/comments. References: [1] Abraham Itzhak Weinberg, (Feb 2019), Springer Selecting a representative decision tree from an ensemble of decision-tree models for fast big data classification [2] Leo Breiman, (Oct 2001), Springer Random Forests [3] Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin, (Mar 2004), The Wharton School of the University of Pennsylvania, arxiv.org Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation

Partial Dependence Plots: How to Discover Variables Influencing a Model

Spark is the name of the engine to realize cluster computing while PySpark is the Python's library to use Spark. PySpark is a great language for performing exploratory data analysis at scale, building machine learning pipelines, and creating ETLs for a data platform. If you’re already familiar with Python and libraries such as Pandas, then PySpark is a great language to learn in order to create more scalable analyses and pipelines. The goal of this post is to show how to build a machine learning model using PySpark. How To Install PySpark PySpark installing process is very easy as like others python's packages.(eg.Pandas,Numpy,scikit-learn). One important thing is, firstly ensure java has installed in your machine. then you can run PySpark on your jupyter notebook. Exploring The Data We will use the same data set when we , and it is related to diabetes diseases of a National Institute of Diabetes and Digestive and Kidney Diseases. The classification goal is to predict whether the patient has diabetes (Yes/No). The dataset can be downloaded from . built machine learning models in Python Kaggle pyspark.sql SparkSession
spark = SparkSession.builder.appName( ).getOrCreate()
df = spark.read.csv( , header = , inferSchema = )
df.printSchema() from import 'ml-diabetes' 'diabetes.csv' True True The datasets consists of several medical predictor variables and one target variable, Outcome. Predictor variables includes the number of pregnancies the patient has had, their BMI, insulin level, age, and so on. Input variables: Glucose,BloodPressure,BMI,Age,Pregnancies,Insulin,SkinThikness,DiabetesPedigreeFunction. Output variables: Outcome. Have a peek of the first five observations. Pandas data frame is prettier than Spark DataFrame.show(). pandas pd
pd.DataFrame(df.take( ), columns=df.columns).transpose() import as 5 In PySpark you can show the data with Pandas' DataFrame using toPandas() df.toPandas() Checking the classes are perfectly balanced!! df.groupby( ).count().toPandas() 'Outcome' Statistics Summary numeric_features = [t[ ] t df.dtypes t[ ] == ]
df.select(numeric_features).describe().toPandas().transpose() 0 for in if 1 'int' Correlations between independent variables pandas.plotting scatter_matrix
numeric_data = df.select(numeric_features).toPandas()

axs = scatter_matrix(numeric_data, figsize=( , )); n = len(numeric_data.columns) i range(n):
    v = axs[i, ]
    v.yaxis.label.set_rotation( )
    v.yaxis.label.set_ha( )
    v.set_yticks(())
    h = axs[n , i]
    h.xaxis.label.set_rotation( )
    h.set_xticks(()) from import 8 8 # Rotate axis labels and remove axis ticks for in 0 0 'right' -1 90 Data preparation and feature engineering In this part, we will remove unnecessary columns and fill the missing values. Finally, selecting features for machine learning models. These features will be divided into two parts train and test. Lets starting mission 👨‍🚀 Missing Data Handling: pyspark.sql.functions isnull, when, count, col

df.select([count(when(isnull(c), c)).alias(c) c df.columns]).show() from import for in Wow!! 👌 That's great in this datasets haven't any missing values.😀 Unnecessary columns dropping dataset = dataset.drop( )
dataset = dataset.drop( )
dataset = dataset.drop( )
dataset = dataset.drop( )

dataset.show() 'SkinThickness' 'Insulin' 'DiabetesPedigreeFunction' 'Pregnancies' Features Convert into Vector VectorAssembler — a feature transformer that merges multiple columns into a vector column. required_features = [ , , , ] pyspark.ml.feature VectorAssembler

assembler = VectorAssembler(inputCols=required_features, outputCol= )

transformed_data = assembler.transform(dataset)
transformed_data.show() # Assemble all the features with VectorAssembler 'Glucose' 'BloodPressure' 'BMI' 'Age' from import 'features' Done!!✌️ Now features converted into a vector. 🧮 Train and Test Split Randomly split data into train and test sets, and set seed for reproducibility. (training_data, test_data) = transformed_data.randomSplit([ , ], seed = )
print( + str(training_data.count()))
print( + str(test_data.count())) # Split the data 0.8 0.2 2020 "Training Dataset Count: " "Test Dataset Count: " Training Dataset Count: 620 Test Dataset Count: 148 Machine learning Model Building Random Forest Classifier Random forest is a supervised learning algorithm which is used for both classification as well as regression. But however, it is mainly used for classification problems. As we know that a forest is made up of trees and more trees means more robust forest. Similarly, random forest algorithm creates decision trees on data samples and then gets the prediction from each of them and finally selects the best solution by means of voting. It is an ensemble method which is better than a single decision tree because it reduces the over-fitting by averaging the result. pyspark.ml.classification RandomForestClassifier

rf = RandomForestClassifier(labelCol= , 
                            featuresCol= ,
                            maxDepth= )
model = rf.fit(training_data)
rf_predictions = model.transform(test_data) from import 'Outcome' 'features' 5 Evaluate our Random Forest Classifier model. pyspark.ml.evaluation MulticlassClassificationEvaluator

multi_evaluator = MulticlassClassificationEvaluator(labelCol = , metricName = )
print( , multi_evaluator.evaluate(rf_predictions)) from import 'Outcome' 'accuracy' 'Random Forest classifier Accuracy:' Random Forest classifier Accuracy: 0.7945205479452054
(79.5%) Decision Tree Classifier Decision trees are widely used since they are easy to interpret, handle categorical features, extend to the multiclass classification setting, do not require feature scaling, and are able to capture non-linearities and feature interactions. pyspark.ml.classification DecisionTreeClassifier

dt = DecisionTreeClassifier(featuresCol = , labelCol = , maxDepth = )
dtModel = dt.fit(training_data)
dt_predictions = dtModel.transform(test_data)
dt_predictions.select( , , , , ).show( ) from import 'features' 'Outcome' 3 'Glucose' 'BloodPressure' 'BMI' 'Age' 'Outcome' 10 Evaluate our Decision Tree model. pyspark.ml.evaluation MulticlassClassificationEvaluator

multi_evaluator = MulticlassClassificationEvaluator(labelCol = , metricName = )
print( , multi_evaluator.evaluate(dt_predictions)) from import 'Outcome' 'accuracy' 'Decision Tree Accuracy:' Decision Tree Accuracy: 0.7876712328767124
(79.7%) Logistic Regression Model Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary).  Like all regression analyses, the logistic regression is a predictive analysis.  Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables. Logistic Regression is used when the dependent variable(target) is categorical. pyspark.ml.classification LogisticRegression

lr = LogisticRegression(featuresCol = , labelCol = , maxIter= )
lrModel = lr.fit(training_data)
lr_predictions = lrModel.transform(test_data) from import 'features' 'Outcome' 10 Evaluate our Logistic Regression model. pyspark.ml.evaluation MulticlassClassificationEvaluator

multi_evaluator = MulticlassClassificationEvaluator(labelCol = , metricName = )
print( , multi_evaluator.evaluate(lr_predictions)) from import 'Outcome' 'accuracy' 'Logistic Regression Accuracy:' Logistic Regression Accuracy: 0.7876712328767124
(79.7%) Gradient-boosted Tree classifier Model Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. pyspark.ml.classification GBTClassifier
gb = GBTClassifier(labelCol = , featuresCol = )
gbModel = gb.fit(training_data)
gb_predictions = gbModel.transform(test_data) from import 'Outcome' 'features' Evaluate our Gradient-Boosted Tree Classifier. pyspark.ml.evaluation MulticlassClassificationEvaluator
multi_evaluator = MulticlassClassificationEvaluator(labelCol = , metricName = )
print( , multi_evaluator.evaluate(gb_predictions)) from import 'Outcome' 'accuracy' 'Gradient-boosted Trees Accuracy:' Gradient-boosted Trees Accuracy: 0.8013698630136986(80.13%) Conclusion PySpark is a great language for data scientists to learn because it enables scalable analysis and ML pipelines. If you’re already familiar with Python and Pandas, then much of your knowledge can be applied to Spark. To sum it up, we have learned how to build a machine learning application using PySpark. We tried three algorithms and gradient boosting performed best on our data set. I got inspiration from @ 's Github repository ' '. Favio André Vázquez first_spark_model Source code can be found on . I look forward to hearing feedback or questions. Github Machine learning models sparking when PySpark gave the accelerator gear like the need for speed gaming cars. References: 1. PySpark Tutorial for Beginners: Machine Learning Example 2. Apache Spark 2.1.0 3. Machine Learning with PySpark and MLlib — Solving a Binary Classification Problem

Building A Machine Learning Model With PySpark [A Step-by-Step Guide]

Water your plants, get alarms, and analyze the data Having plants at home is a luxury that only households that  always count with someone's presence can enjoy. If you live alone and travel for a few days, you are likely to return home and find a mumified version of your beloved basil plant. One may argue: "Why not buy a timer on Home Depot for 5 bucks?". That's an excellent point, but why not make it overly complicated and learn a couple of things during the process? In this article I will show you how I created an IoT application that controls the irrigation of the plants in my house. This application independently irrigates 5 pots: 1 lemon tree, 2 passion fruit plants, 1 gardenia and 1 pot with herbs (basil, fennel and oregano). Because I'm planting these plants on the balcony of my apartment, I don't have running water, so I'm using a 40 liters tank. This tank has an ultrasonic sensor that monitors the volume and once the water level is below 30%, my Losant app sends me an email telling me to fill the tank. Well, to be honest, it sends an email to my wife, I hate filling that tank. Besides that, the system gets the current temperature and humidity here in Houston, TX and based on that increases the volume of irrigation water to compensate for the extra heat. I realized that I needed to implement this feature once I was done with the hardware, so I'm pulling the temp info from a free weather API. It's pretty cool that Losant allows you to integrate info from so many different sources. The hardware The general system diagram is shown below: The electronic enclosure The electronics are very simple, these are the components of this project: Microprocessor: Particle Photon (https://www.particle.io) Power supplies (5 and 12 Volts) 6 Relays. Caveat: I have used mechanical relays because I've also used a switching power supply. SSD relays can switch as fast as your switching power supply and mess up with the output. Wifi antenna The basic idea is the following: 5V feed the Particle Photon Photon's outputs connect to the low power side of the relays driving 12V to the valve solenoids upon activation. Similarly, the pump (underwater aquarium pump) is controlled by the Photon which drives AC current through its motor. Lets put everything together in a nice, presentable box. Remember, if you use a metallic enclosure like I did, you need to ground the box. I can't stress this enough! The Water Manifold I got the cheapest solenoid valves I could get from AliExpress.com, I think I paid 3 bucks for each. This is not bad at all, in the US they go for 10 bucks each. I also got some valves, tubing and an aquarium manifold. I've put everything together in a fancy plastic box. Note that the electrical connection from this wet module to the electronics in done by a multi-core cable and a connector. The final result is pretty neat. Firmware The Particle Photon, like Arduino, uses C++. It has a some conveniences like Particle.function() and Particle.publish(). These features allow the user to trigger functions from external events and publish local events respectively. In order to trigger functions remotely you'll need to declare them in the void setup. In my system I have 7 Particle functions, they are: tooHot: changes the value of the boolean variable hot to TRUE if the local temperature is greater than 30 Celsius. The data is retrieved from the weather API and the logic is executed in Losant. mode: activates the manual mode. pf1, lemon, herbs, gardenia, pf2: water each plant in manual mode. These Particle functions only accept one argument and it must be a String. Conversaly, you can publish events with Particle.publish(). Please check out my code below and next we will see how to create an interface to interact with the microcontroller. V1 = ; V2 = ; V3 = ; V4 = ; V5 = ; pump = ; trigger = ; echo = ; volume; hot = FALSE; manual_mode = FALSE; { Particle.function( , tooHot);
    Particle.function( , mode);
    Particle.function( , pf1);
    Particle.function( , lemon);
    Particle.function( , herbs);
    Particle.function( , gardenia);
    Particle.function( , pf2); pinMode(trigger, OUTPUT);
    pinMode(echo, INPUT);
    pinMode(pump, OUTPUT);
    pinMode(V1, OUTPUT);
    pinMode(V2, OUTPUT);
    pinMode(V3, OUTPUT);
    pinMode(V4, OUTPUT);
    pinMode(V5, OUTPUT); Time.zone( ); digitalWrite(V1, HIGH);
    digitalWrite(V2, HIGH);
    digitalWrite(V3, HIGH);
    digitalWrite(V4, HIGH);
    digitalWrite(V5, HIGH);
    digitalWrite(pump, HIGH);
    
    Serial.begin( );
} { STARTUP(WiFi.selectAntenna(ANT_EXTERNAL)); (manual_mode == FALSE){ (Time.hour() == && Time.minute() == ){
            auto_irrigation();
        } (hot == TRUE){ (Time.hour() == && Time.minute() == ){
                auto_irrigation();
            }
        }
    } ((Time.hour() == || Time.hour() == ) && Time.minute() == ){
        volume = measure_level();
        Particle.publish( , (String)volume);
    }

} {
    digitalWrite(V1, LOW);
    digitalWrite(pump, LOW);
    delay( );
                
    digitalWrite(V1, HIGH);
    digitalWrite(V2, LOW);
    delay( );
                
    digitalWrite(V2, HIGH);
    digitalWrite(V3, LOW);
    delay( );
                
    digitalWrite(V3, HIGH);
    digitalWrite(V4, LOW);
    delay( );
                
    digitalWrite(V4, HIGH);
    digitalWrite(V5, LOW);
    delay( );
                
    digitalWrite(V5, HIGH);
    digitalWrite(pump, HIGH);
} { result; duration = ; ( i = ; i <= ; i++)
    {
        digitalWrite(trigger, LOW);
        delayMicroseconds( );
        digitalWrite(trigger, HIGH);
        delayMicroseconds( );
        digitalWrite(trigger, LOW);
        duration = duration + (pulseIn(echo, HIGH)/ );
        delay( );
    }
    result = *( -duration* )/ ; result;
} { (command == ){
        hot = TRUE; ;
    } (command == ){
        hot = FALSE; ;
    } { ;
    }
    
} { (command == ){
        manual_mode = TRUE; ;
    } (command == ){
        manual_mode = FALSE; ;
    } { ;
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V1, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V1, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V2, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V2, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V3, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V3, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V4, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V4, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V5, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V5, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} // ports to control and read hardware const int 0 const int 1 const int 2 const int 3 const int 4 const int 5 const int 6 const int 7 // tank volume measurement int // temperature alarm bool // manual mode bool void setup () // declaring particle functions "tooHot" "mode" "pf1" "lemon" "herbs" "gardenia" "pf2" // defining i/o's // setting time zone -5 // initiallizing actuators 9600 void loop () // selects the u.FL antenna if // morning irrigation schedule if 7 00 // if it's warm, the system water the plants again at 3pm if if 15 00 // measure water tank level twice a day if 8 16 00 "tank_volume" /* FUNCTIONS */ // irrigation int auto_irrigation () 80000 50000 80000 40000 40000 // measure the tank level, averages 100 measurements int measure_level () int long 0 for int 0 100 2 10 100 50 100 25.4 0.017 25.4 return /* this function changes the boolean value of the variable hot and its logic is executed on Losant. 
the weather api provides the temp and if greater than 30C, it will trigger toohot. */ int tooHot (String command) if "yes" return 1 else if "no" return 0 else return -1 // this function changes the logic value of manual_mode and it's triggered on losant. int mode (String command) if "true" return 1 else if "false" return 0 else return -1 // passion fruit 1 manual irrigation int pf1 (String command) if if "true" else if "false" // lemon tree manual irrigation int lemon (String command) if if "true" else if "false" // herbs manual irrigation int herbs (String command) if if "true" else if "false" // gardenia manual irrigation int gardenia (String command) if if "true" else if "false" // passion fruit 2 manual irrigation int pf2 (String command) if if "true" else if "false" IoT Workflow Receive an email notification Losant is an IoT platform that allows the developper to create a workflow logic that will receive and trigger events. It's beyond the scope of this article to provide a detailed Losant tutorial, however I'll explain the intuition behind it and give you references in case you decide to develop your own IoT app. Let's start checking how to implement an email notification to alert the user that the water level is getting low. First you'll need to create a device and a particle integration in Losant. Upon doing that, go to your workflow and build the following block logic. Let's examinate how to set the nodes to implement this logic. This node establishes communication with your device. Click on it and select the integration you have created. 1. Particle: Remember Particle.publish()? We will set up this node to listen for events published from the Photon. In order to do that, we need to define the structure of the payload with some javascript. The event published by the microcontroller will be transmitted in a json package, more precisely inside data. In our case, the variable being passed is 'volume', so we can retrieve this data by accessing 'data.volume'. 2. Function: Click on the function node and add the following code. parts = payload.data.data.split( );
payload.data.volume = parts[ ]; var ':' 0 : This node will attribude the data from you payload to a device and the attributed you've defined upon its creation. This will allow us to create a dashboard later. 3. Device Get Select the query method to: 'Match all tags query' Set 'Key Templates' to 'particle_device_id' and 'Value Templates' to '{{ data.coreid }}' Set 'Result Path' to 'device_to_update' : Stores the state in a device attribute. 4. Device State 'Device ID JSON path' = 'device_to_update.id' 'Attribute' = '{{data.volume}}' This block allows you to debug and understand what is going on with you payload. You can trigger events from the workflow and see how your payload is being constructed. This block is not necessary to make the app work. 5. Debug: Executes a conditional logic. Remember you'll need to refer to the variable according to the way that your JSON payload was defined, in our case {{data.volume}}. 6. Conditional: This one is pretty self explanatory. Upon the activation of the condition previously defined, this block will send an email. 7. Email: To summarize, we will retrieve the variable 'volume' by setting up a logic to listen to the events being published from the IoT microcontroller. Upon getting this data from a JSON package we will compare it to a threshold and if the value is less than 30% of the total volume, an email will be sent asking the user to re-fill the tank. Get local temperature from an API First you'll need an API key, go to and create an account. You can use their free plan for this project. If I'm not wrong, the free plan includes 1000 api calls a day. If you're not in the agricultural business, this should be more than enough. https://openweathermap.org/api Build the following workflow: The http block is responsible from calling the API, let's check on how to set it up. In the 'URL template', add the following: http://api.openweathermap.org/data/2.5/weather?q='YOUR CITY'&appid='YOUR API KEY' The 'request method' should be set to GET and the API response should be stored in the 'Payload path to store response' field. Set it to 'working.weather'. Set the timer to call the API every hour so you don't exceed the daily limit of calls. Now your payload should look like this: Now the temperature should be accessible through: {{working.weather.body.main.temp}} Triggering Functions Remotely Now it's time to finally trigger our Particle functions previously defined in our firmware. Let's start by building a workflow: Basically we have a 'Virtual Button' node triggering a 'Particle Call' node for each one of our remotely operated functions. In the 'Particle Call' node, we need to: Inform the node which firmware function we will call by adding its name to the field 'Function Name'. Tell the node what argument we will be passing by specifying it directly or adding a payload path. We will pass a payload path since we will be using a toggle switch to activate the system. We will create variables for each plant and add them to the payload. They will be located at the root data, so they will be accessible as data.pf1, data.lemon, data.herbs, data.gardenia and data.pf2. Now, we need to create an interface to trigger this workflow we've just created. Go to Dashboard and add an input control. Add toggles and buttons like so: The toggle state is sent via payload to the microcontroller upon triggering the workflow via the button. Set the button like so (note, the value between the double brackets should be the same of your toggle, my toggle is also named 'lemon'). Do the same for all the other inputs and test it: Data visualization Add some charts and link then to your devices attributes. Conclusion I hope you guys enjoyed this article. This project has several moving parts and I know that I have covered some of them too fast without providing the level of detail that the reader deserves, for that reason do not hesitate to contact me if you'd like to get more info or simply ask a question. Moving forward I'd like to add a strain gauge to monitor the weight of the plant. Over 98% of the mass of plants comes from the air (interesting fact isn't it?). That would be an interesting step into starting optimizing fertilization and irrigation. If you are interested in learning more about Losant, check this tutorial out: https://www.losant.com/blog/how-to-integrate-particle-with-losant See y'all soon!

IoT Home Irrigation System Using Losant and Particle Photon

Detecting Fraudulent Transactions

Before any predictive ML models can be applied to a data set it is important to thoroughly understand the data. Using visualizations is a very effective method of digging deeper into what the data. This is important prior to using any ML predictive models, in order to ensure meaningful and actionable outcomes. The goal of this case study is to illustrate how to use python to better understand the . In this case we are going to use State Drug Utilization data sets which can be found from the website. Creating visually rich images of tabular data can provide insights that are often difficult to see in tabular format. data Medicaid.gov Examples of what can be created include displaying the total number of hydrocodone(a narcotic analgesic) units that were reimbursed by Medicare in 2017. The size of the blue circles are representative of the number of units (based on Unit Type) reimbursed by Medicaid sorted by each state. According to the Kaiser Family Foundation there were a total of deaths in 2016 related to natural and semi synthetic opioids such as hydrocodone. 14,487 Henry J Kaiser Family Foundation Another example includes comparing the number of prescriptions Medicaid paid for in full or in portion in 2008 and 2018. (fig 2) Number of Hydrocodone prescriptions for which Medicaid paid a portion in 2008 and in 2018 It is clear to see that there is a decreasing trend in the number of hydrocodone prescriptions for which Medicaid paid a portion in 2018 compared to 2008. This is in line with most states adopting tougher prescription size limits as well as some states requiring prior authorizations before approval. Most states show decreasing numbers. However in 2018, even with a 15% decrease in the number of prescriptions paid for by Medicaid, California still stands out by far for having the most number of hydrocodone prescriptions paid for in part or full by Medicaid. % Difference between the Number of Hydrocodone Prescriptions paid for in part or in full by Medicaid 2008–2018 The percentage difference in the number of prescriptions varied from -91% to 1570%. Top 6 States with highest % increase in the Number of Hydrocodone Prescriptions paid for in part or in full by Medicaid 2008–2018 6 states showed increases in the number of hydrocodone prescriptions over the 10 year period; New Mexico 1570% increase, Iowa 738%, Maryland 402%, Michigan 207%, Oregon 86%, Nevada, 10%. The International Narcotics Control Board reported that in 2015, Americans represented about 99.7% of the world’s hydrocodone consumption ( ) CNN **Step 1: Accessing the data**You can access the data from the Medicaid.gov website . The data sets are broken down by year. I used the State Drug Utilization 2017 data set for fig 1 and the 2008 and 2018 data sets for the comparisons. here Publication of these drug utilization data sets have been available since the start of the , where states report drug utilization for covered outpatient drugs paid for by state Medicaid agencies. Medicaid Drug Rebate Program More information about the data set and an explanation of the data field descriptions can be found in this FAQ here . Step 2: Install and import dependencies ou can access the Jupyter Notebook : Y here import numpy as npimport pandas as pd from pandas_summary import DataFrameSummaryfrom IPython.display import displayfrom sklearn import metrics 2. Load the data set PATH = "Utilization/"df_data = pd.read_csv(PATH+'State_Drug_Utilization_Data_2017.csv')df_data.head() The output above results in the data being loaded with the following warning message: This is easily fixed by including ,this warning occurs because guessing dtypes for each column is very memory demanding. low_memory=False How much data are we dealing with here? returns 4.7 million rows and 20 columns df_data.shape Review the data df_data.columns Here is the first issue, notice the gap between the column names for example needs to be re-named to , having a gap within a column name will only give us errors further down. Utilization Type Utilization_Type Renaming columns is done by using rename df_data.rename(columns={'Utilization Type':'Utilization_Type','State':'State','Labeler Code':'Labeler_Code','Product Code':'Product_Code','Package Size':'Package_Size','Year':'Year','Quarter':'Quarter','Product Name':'Product_Name','Suppression Used':'Suppression_Used','Units Reimbursed':'Units_Reimbursed','Number of Prescriptions':'Number_of_Prescriptions','Total Amount Reimbursed':'Total_Amount_Reimbursed','Medicaid Amount Reimbursed':'Medicaid_Amount_Reimbursed','Non Medicaid Amount Reimbursed':'Non_Medicaid_Amount_Reimbursed','Quarter Begin':'Quarter_Begin','Quarter Begin Date':'Quarter_Begin_Date','Latitude':'Latitude','Longitude':'Longitude','Location':'Location','NDC':'NDC'}, inplace=True) And now df_data.columns should now look like this: The following commands , and help with getting additional information: df_data.info() df_data.count() df_data.describe(include='all') prints out information about the data frame; number of entries, number of columns, column data types and memory usage. Knowing data types is important especially if we are merging .csv files and also for memory error issues. df_data.info() Working with large data sets can result in memory error issues and reviewing the dtypes can be very helpful in resolving these memory issues. I did not encounter any memory issues until merging the different year data sets. As an example changing all int64 dtypes into int32 dtypes results in a saving of 180 MB (from 689 MB to 509 MB). However you have to be careful when changing dtypes, if you have large numbers you need to be aware of the storage capacity range of each for example: int Int16 -- (-32,768 to +32,767)Int32 -- (-2,147,483,648 to +2,147,483,647)

Int64 -- (-9,223,372,036,854,775,808 to +9,223,372,036,854,775,807) prints out the number of entries in each column, here we can see that there are missing values in 8 columns (circled in blue). df_data.count() generates descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding values. Here we can see the number of unique values in each column, the value most commonly found in that column and more. However here is another problem… looking under the top state is df_data.describe NaN State XX! Its also sometimes annoying to have to scroll through the describe output, I want the ability to see all the data together. This can be done by using transpose. So what does state ‘XX’ mean? According to the FAQ it refers to any data where the are aggregated for example for National Totals so the code ‘XX’ represents multiple states. For the sake of this project we are going to remove any entries where the State is ‘XX’ here , National Drug Code (NDC) There are 20 columns in this data set and we do not need to use all of them so I am going to get rid of a number of columns and save the new file. We now have the file we need to create Fig 1. 4 Create the Visualization The visualization was created using Basemap which is a great for creating maps using geographical projections. You will have to download the shape files for all the state boundaries. You can download the shape files from , you should download the .shx .dbf and .shp files. tool here You can find the Jupyter Notebook for this project here If you are interested in reading more about AI and healthcare here are my other articles. Thanks for reading! _Experimenting with simple data augmentation parameters to get the best results_towardsdatascience.com Data Augmentation Experimentation _“If you always do what you’ve always done, you’ll always get what you’ve always got.” — Henry Ford_becominghuman.ai Not Your Typical Health IT Company

Visualizing State Drug Utilization Data Sets

This tutorial is part of this book and through it, we are going to discover how to use Rancher in order to create a scalable application. Wordpress Rancher Architecture In , everything (like containers, networks or images ) is an resource with a process lifecycle. Containers, images, networks, and accounts are all resources with their own process lifecycles. Rancher API API Rancher is built on the top of containers: A web UI An API A server that manages Rancher Agents A database A machine microservice The binary docker-machine Rancher Server Architecture When you run using the + + The Database + are processes that live inside this container. Rancher docker run rancher/server ... Rancher API Rancher Process Server Machine Microservice Note that the docker-machine binary is also living in the same container but only runs when it is called by the . API has also an Agent part that manages the life cycle of containers. Rancher If creates a machine successfully, some events are exchanged between the and the microservice. A event is created and a command is executed to get the details needed to connect to the machine’s Docker daemon. docker-machine docker-machine bootstrap docker-machine config If everything run without problems, the service fires up a on the machine via docker Rancher Agent run rancher/agent … open a connection to the server in order to establish a 2-way communication. The manage its containers and reports every change using the Docker . Rancher Agents WebSocket Rancher Agent API During this tutorial, we are going to use an machine and this is how a different view of the layers of our installation: EC2 Rancher RancherOS Architecture RancherOS is a small distribution released by team. It is an easy way to run containers at scale in production and includes only the services needed to run Docker. RancherOS Rancher It only includes the latest version of Docker and removes any unneeded library that a “normal” distribution could have. Linux In , everything is a container, the traditional system is replaced so that Docker run directly on the . RancherOS init Kernel A special component in this system is called which is the daemon that allow a user (a non-system user) runs its containers. User Docker RancherOS Architecture We are going to run in an machine using : RancherOS EC2 AWS CLI aws ec2 run-instances --image-id ami-ID --count 1 --instance-type t2.micro --key-name MySSHKeyName --security-groups sg-name This is the list of AMI by region: Now you can login to your machine using the common command : AWS ssh ssh -i "MySSHKeyName" rancher@xxxx.xx-xxxx-x.compute.amazonaws.com Running Rancher Since Docker is installed by default, we can start using it directly. Let’s start a server then use it with . MariaDB Rancher Server docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=cattle -e MYSQL_USER=cattle -e MYSQL_PASSWORD=cattle -p 3306:3306 -d mariadb Run the and change 172.31.0.190 by your : Rancher Server MariaDB IP docker run --name rancher_server -p 8080:8080 -e CATTLE_DB_CATTLE_MYSQL_HOST=172.31.0.190 -e CATTLE_DB_CATTLE_MYSQL_PORT=3306 -e CATTLE_DB_CATTLE_MYSQL_NAME=cattle -e CATTLE_DB_CATTLE_USERNAME=cattle -e CATTLE_DB_CATTLE_PASSWORD=cattle -v /var/run/docker.sock:/var/run/docker.sock -d rancher/server Now you can go to your and add a host with a supported version of Docker: <host_ip>:8080 Linux Adding A Rancher Host We are not going to use the public address but the private ( ) one. IP eth0 Adding A Rancher Host Let’s add a custom host. A command is given to run on any reachable host in order to let it join the server.At this step, I already created an machines (with as an operating system) and I am going to use it and run this command: EC2 RancherOS sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.1 http://172.31.0.190:8080/v1/scripts/02048221239BE78134BB:1483142400000:i5yrQRE4kFo7eIKz9n1o5VFTew Make sure any or allow traffic from and to all other hosts on ports 500 and 4500. Security Groups firewalls UDP After the installation, we can notice that some containers are running in each machine like the , , ..: Agent Agent dns healthcheck network-manager CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS               NAMES8339a8af3bb8        rancher/net:v0.9.4               "/rancher-entrypoint."   About a minute ago   Up About a minute                       r-ipsec-ipsec-router-1-19bd25e395f3d3afdb7c        rancher/net:holder               "/.r/r /rancher-entry"   About a minute ago   Up About a minute                       r-ipsec-ipsec-1-90844ad45643089ca485        rancher/healthcheck:v0.2.3       "/.r/r /rancher-entry"   2 minutes ago        Up 2 minutes                            r-healthcheck-healthcheck-1-91ae658dfec564c53f13        rancher/dns:v0.14.1              "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-metadata-dns-1-471d9deac162c07d07f6        rancher/net:v0.9.4               "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-ipsec-ipsec-cni-driver-1-e02f6a010313ff5cb812        rancher/scheduler:v0.7.5         "/.r/r /rancher-entry"   2 minutes ago        Up 2 minutes                            r-scheduler-scheduler-1-5ca518560f6f62a190c7        rancher/network-manager:v0.5.3   "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-network-manager-1-03d3f619ec270864ff07        rancher/metadata:v0.8.11         "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-metadata-1-4be6ebc698a78058bc3c        rancher/agent:v1.2.1             "/run.sh run"            2 minutes ago        Up 2 minutes                            rancher-agent Running A Wordpress Service If everything is ok, we are going to create a service. Go to menu and click . Wordpress Stack User Create the database, name it (we are going to use it in order to link it to the container) MariaDB wordpressdb Wordpress Then add your environment variables: Now, using the same way, create the container, map host port 80 to the container port 80 and link the container as a instance. Wordpress mariadb mysql Then add your environment variables: At this step, we can check the running services, use -> then to see the containers: Stack User default What we started is the equivalent of 2 docker commands, one to start the container and the other one to start the container. mariadb wordpress You can visit the fresh installation using its address. Wordpress IP We can see the details of the different configurations a running container could have if you choose the container from the same view then use the “ ” menu. View in API This is an example of the container: wordpress {"id":"1s7","type":"service","links":{"self":"…/v2-beta/projects/1a5/services/1s7","account":"…/v2-beta/projects/1a5/services/1s7/account","consumedbyservices":"…/v2-beta/projects/1a5/services/1s7/consumedbyservices","consumedservices":"…/v2-beta/projects/1a5/services/1s7/consumedservices","instances":"…/v2-beta/projects/1a5/services/1s7/instances","networkDrivers":"…/v2-beta/projects/1a5/services/1s7/networkdrivers","serviceExposeMaps":"…/v2-beta/projects/1a5/services/1s7/serviceexposemaps","serviceLogs":"…/v2-beta/projects/1a5/services/1s7/servicelogs","stack":"…/v2-beta/projects/1a5/services/1s7/stack","storageDrivers":"…/v2-beta/projects/1a5/services/1s7/storagedrivers","containerStats":"…/v2-beta/projects/1a5/services/1s7/containerstats"},"actions":{"upgrade":"…/v2-beta/projects/1a5/services/1s7/?action=upgrade","restart":"…/v2-beta/projects/1a5/services/1s7/?action=restart","update":"…/v2-beta/projects/1a5/services/1s7/?action=update","remove":"…/v2-beta/projects/1a5/services/1s7/?action=remove","deactivate":"…/v2-beta/projects/1a5/services/1s7/?action=deactivate","removeservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=removeservicelink","addservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=addservicelink","setservicelinks":"…/v2-beta/projects/1a5/services/1s7/?action=setservicelinks"},"baseType":"service","name":"wordpress","state":"active","accountId":"1a5","assignServiceIpAddress":false,"createIndex":1,"created":"2017-04-04T21:05:53Z","createdTS":1491339953000,"currentScale":1,"description":"wordpress files","externalId":null,"fqdn":null,"healthState":"healthy","instanceIds":["1i14"],"kind":"service","launchConfig":{"type":"launchConfig","capAdd":[ ],"capDrop":[ ],"dataVolumes":[ ],"dataVolumesFrom":[ ],"devices":[ ],"dns":[ ],"dnsSearch":[ ],"environment":{"WORDPRESS_DB_NAME":"wordpress","WORDPRESS_DB_USER":"wordpress","WORDPRESS_DB_PASSWORD":"wordpress"},"imageUuid":"docker:wordpress","instanceTriggeredStop":"stop","kind":"container","labels":{"io.rancher.container.pull_image":"always"},"logConfig":{"type":"logConfig","config":{ },"driver":""},"networkMode":"managed","ports":["80:80/tcp"],"privileged":false,"publishAllPorts":false,"readOnly":false,"secrets":[ ],"startOnCreate":true,"stdinOpen":true,"system":false,"tty":true,"version":"0","dataVolumesFromLaunchConfigs":[ ],"vcpu":1},"lbConfig":null,"linkedServices":{"mysql":"1s6"},"metadata":null,"publicEndpoints":[{"type":"publicEndpoint","hostId":"1h1","instanceId":"1i14","ipAddress":"54.246.163.197","port":80,"serviceId":"1s7"}],"removed":null,"retainIp":null,"scale":1,"scalePolicy":null,"secondaryLaunchConfigs":[ ],"selectorContainer":null,"selectorLink":null,"stackId":"1st5","startOnCreate":true,"system":false,"transitioning":"no","transitioningMessage":null,"transitioningProgress":null,"upgrade":null,"uuid":"0b8a8290-0d10-47be-b182-45f1e57e80bc","vip":null} We can also inspect other services and hosts configurations in the same way. Cattle: The Rancher Container Orchestrator What we started below is powered by its own orchestration tool called . Rancher offers the possibility to use other orchestration tools like , or . Rancher Cattle Kubernetes Docker Swarm Mesos is a container orchestration and scheduling framework, in its beginning, it was designed as an extension to but since continues to develop, and started to diverge. is used extensively by itself to orchestrate infrastructure services as well as setting up, managing, and upgrading , , and clusters. Cattle Docker Swarm Docker Swarm Cattle Swarm Cattle Rancher Swarm Kubernetes Mesos application deployments are organized into which can be used to or stacks. A is a collection of and the latter is primarily a Docker image with its networking, scalability, storage, health checks, service discovery links, environment and all of the other configurations. Cattle stacks User Infrastructure Cattle Stack Services A could be a load balancer or an external service. A can be launched using a or file or just start containers like we did for the stack. Cattle Service Stack docker-compose.yml rancher-compose.yml Wordpress In addition to this, you can start several applications using an application catalog. More than 50 different apps are available in the app catalog. An application is defined by a and a file and can be deployed easily with the default configurations. docker-compose rancher-compose Let’s take the example of : Portainer Portainer is a lightweight management UI which allows you to easily manage your Docker host or Swarm cluster.Portainer is meant to be as simple to deploy as it is to use. It consists of a single container that can run on any Docker engine (Docker for Linux and Docker for Windows are supported).Portainer allows you to manage your Docker containers, images, volumes, networks and more ! It is compatible with the standalone Docker engine and with Docker Swarm. This is the file of : docker-compose.yml Portainer portainer:  labels:    io.rancher.sidekicks: ui    io.rancher.container.create_agent: true    io.rancher.container.agent.role: environment  image: rancher/portainer-agent:v0.1.0  volumes:  - /config

ui:  image: portainer/portainer:pr572  command: --no-auth --external-endpoints=/config/config.json --sync-interval=5s -p :80  volumes_from:  - portainer  net: container:portainer This is its file: rancher-compose.yml .catalog:  name: "Portainer"  version: "1.11.4"  description: Open-source lightweight management UI for a Docker host or Swarm cluster  minimum_rancher_version: v1.5.0-rc1 Starting Portainer Rancher offers the possibility to start infrastructure . If you go to -> , you can see a catalog of tools like , , ..etc Stacks Stacks Infrastructure Bind9 Portainer Rancher vxlan Infrastructure Catalog Scaling Wordpress Using Rancher Now we want to scale our frontal app in order to handle more traffic. In this case, you should click -> then the name of the (it should be the default one if you followed this tutorial as it is). Now click on and scale your app to 3 containers: Wordpress Stack User Stack Wordpress At this step, you should see how won’t scale the right way. Wordpress This is normal since all of our containers are mapped to the same port 80 (that can handle only one container). Wordpress In order to scale the right way, we should tell Rancher to create a load balancer service with auto-discovery and let choose the container port automatically. Wordpress Rancher Wordpress This part is detailed in . Painless Docker Book Connect Deeper This article is part of . Painless Docker Book: Unlock The Power Of Docker & Its Ecosystem is a practical guide to master and based . Painless Docker Docker its ecosystem on real-world examples Painless Docker tends to be a complete and detailed guide to create, deploy, optimize, secure, trace, debug, log, orchestrate & monitor Docker and Docker clusters. Through this book, you will learn how to use Docker in development and production environments and the DevOps pipeline between them in order to build modern microservices applications. If you resonated with this article, please subscribe to : An Online Community Of Diverse & Passionate DevOps, SysAdmins & Developers From All Over The World. DevOpsLinks You can find me on and you can also check my books: & . Twitter SaltStack For DevOps Painless Docker If you liked this post, please recommend and share it with your followers.

Stories, News & Mentions about DATAOLEUM

Dataoleum’s Stories on HackerNoon