At Axlr Data, we help automate your accelerated brand growth through innovative technology

AXLR DATA

axlrdata

A Comprehensive Guide to UAV Digital Forensics With DJI Drones

Storybook V2 Comic Book Builder: Part 2, The Document Generator

An Elegant Three Button HTML and JavaScript Audio Animation Player

Fully Covering I/O File Based Applications in .NET C# Using Unit Tests

In this post, we are going to scrape Yahoo Finance using python. This is a great source for stock-market data. We will code a scraper for that. Using that scraper you would be able to scrape stock data of any company from yahoo finance. As you know I like to make things pretty simple, for that, I will also be using a which will increase your scraping efficiency. web scraper This tool will help us to scrape dynamic websites using millions of rotating proxies so that we don’t get blocked. It also provides a captcha clearing facility. It uses headerless chrome to scrape dynamic websites. Why this tool? Requirements Generally, web scraping is divided into two parts: Fetching data by making an HTTP requestExtracting important data by parsing the HTML DOM Libraries & Tools is a Python library for pulling data out of HTML and XML files. allow you to send HTTP requests very easily. to extract the HTML code of the target URL. Beautiful Soup Requests web scraping tool Setup Our setup is pretty simple. Just create a folder and install Beautiful Soup & requests. For creating a folder and installing libraries type below given commands. I am assuming that you have already installed Python 3.x. pip install beautifulsoup4
pip install requests mkdir scraper Now, create a file inside that folder by any name you like. I am using scraping.py. Firstly, you have to sign up for the . It will provide you with 1000 FREE credits. Then just import Beautiful Soup & requests in your file. like this. scrapingdog API from bs4 import BeautifulSoup import requests What we are going to scrape Here is the list of fields we will be extracting: 1. Previous Close 2. Open 3. Bid 4. Ask 5. Day’s Range 6. 52 Week Range 7. Volume 8. Avg. Volume 9. Market Cap 10. Beta 11. PE Ratio 12. EPS 13. Earnings Rate 14. Forward Dividend & Yield 15. Ex-Dividend & Date 16. 1y target EST Preparing the Food Now, since we have all the ingredients to prepare the scraper, we should make a GET request to the to get the raw HTML data. If you are not familiar with the scraping tool, I would urge you to go through its . Now we will scrape Yahoo Finance for financial data using requests library as shown below. target URL documentation r = requests.get( "https://api.scrapingdog.com/scrape?api_key=<your-api-key>&url=https://finance.yahoo.com/quote/AMZN?p=AMZN&.tsrc=fin-srch" ).text this will provide you with an HTML code of that target URL. Now, you have to use BeautifulSoup to parse HTML. soup = BeautifulSoup(r,’html.parser’) Now, on the entire page, we have four tags. We are interested in the first two because we currently don’t need the data available inside the third & fourth “tbody” tags. “tbody” First, we will find out all those “tbody” tags using variable “ soup”. alldata = soup.find_all(“tbody”) As you can notice that the first two “ ” has 8 “tr” tags and every “tr” tag has two “td” tags. tbody try : table1 = alldata[ 0 ].find_all(“tr”) except : table1= None try : table2 = alldata[ 1 ].find_all(“tr”) except : table2 = None Now, each “tr” tag has two “td” tags. The first td tag consists of the name of the property and the other one has the value of that property. It’s something like a key-value pair. At this point, we are going to declare a list and a dictionary before starting a loop. for l={} u= list () For making the code simple I will be running two different “for” loops for each table. First for “ table1” table1_td = table1[i].find_all(“td”) u.append(l)
 l={} for i in range ( 0 , len (table1)): try : except : table1_td = None l[table1_td[ 0 ].text] = table1_td[ 1 ].text Now, what we have done is we are storing all the td tags in a variable “table1_td”. And then we are storing the value of the first & second td tag in a “dictionary”. Then we are pushing the dictionary into a list. Since we don’t want to store duplicate data we are going to make dictionary empty at the end. Similar steps will be followed for “ . table2” table2_td = table2[i].find_all(“td”) u.append(l)
 l={} for i in range ( 0 , len (table2)): try : except : table2_td = None l[table2_td[ 0 ].text] = table2_td[ 1 ].text Then at the end when you print the list “ ” you get a JSON response. u “Yahoo finance”: [
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 { },
 {
   “Forward Dividend & Yield”: “N/A (N/A)”
 },
 {
   “Ex-Dividend Date”: “N/A”
 },
 { }
 ]
} { “Previous Close”: “ 2 ,317.80” “Open”: “ 2 ,340.00” “Bid”: “ 0.00 x 1800 ” “Ask”: “ 2 ,369.96 x 1100” “Day’s Range”: “ 2 ,320.00–2,357.38” “52 Week Range”: “ 1 ,626.03–2,475.00” “Volume”: “ 3 ,018,351” “Avg. Volume”: “ 6 ,180,864” “Market Cap”: “ 1.173 T” “Beta (5Y Monthly)”: “ 1.35 ” “PE Ratio (TTM)”: “ 112.31 ” “EPS (TTM)”: “ 20.94 ” “Earnings Date”: “Jul 23 , 2020 — Jul 27, 2020” “1y Target Est”: “ 2 ,645.67” Isn’t that amazing. We managed to scrape Yahoo finance in just 5 minutes of setup. We have an array of python Object containing the financial data of the company Amazon. In this way, we can scrape the data from any website. Conclusion In this article, we understood how we can scrape data using & regardless of the type of website. data scraping tool BeautifulSoup Feel free to comment and ask me anything. You can follow me on and . Thanks for reading and please hit the like button! 👍 Twitter Medium Additional Resources And there’s the list! At this point, you should feel comfortable writing your first web scraper to gather data from any website. Here are a few additional resources that you may find helpful during your web scraping journey: List of web scraping proxy services Web Scraping Tools BeautifulSoup Documentation Scrapingdog Documentation Guide to Web Scraping

Web Scraping with python

Project Link: https://github.com/Joker666/goworkerpool In modern programming languages concurrency has become an unequivocal need. Almost all programming languages today have some method of concurrency. Some have really powerful constructs, that can offload works to different threads in the OS, e.g., Java, some mimic this behavior in the same thread, e.g., Ruby. Golang has a very powerful concurrency model called CSP (communicating sequential processes), which breaks a problem into smaller sequential processes and then schedules several instances of these processes called Goroutines. The communication between these processes happens by passing immutable messages via Channels. Here we will explore, how we can take advantage of concurrency in golang and how we can limit its usage with worker pools. In the second part, we will explore how to build a robust solution for concurrency around this concept. A Simple Case Let's imagine we have an external API call which takes about 100ms to get done. If we have 1000 of these calls, and we call these synchronously, it would take about to complete 100s model SimpleData {
	ID } basic ( ) {
	start := time.Now() i, _ := allData {
		Process(allData[i])
	}
	elapsed := time.Since(start)
	fmt.Printf( , elapsed)
} {
	fmt.Printf( , data.ID)
	time.Sleep( * time.Millisecond)
	fmt.Printf( , data.ID)
} main ( ) { allData []model.SimpleData i := ; i < ; i++ {
		data := model.SimpleData{ ID: i }
		allData = (allData, data)
	}
	fmt.Printf( ) basic.Work(allData)
} //// model/data.go package type struct int //// basic/basic.go package import "fmt" "github.com/Joker666/goworkerpool/model" "time" func Work (allData []model.SimpleData) for range "Took ===============> %s\n" func Process (data model.SimpleData) "Start processing %d\n" 100 "Finish processing %d\n" //// main.go package import "fmt" "github.com/Joker666/goworkerpool/basic" "github.com/Joker666/goworkerpool/model" "github.com/Joker666/goworkerpool/worker" func main () // Prepare the data var for 0 1000 append "Start processing all work \n" // Process Start processing all work 
Took ===============> 1m40.226679665s Here, we have a simple model that holds a data struct with just an integer value. We process an array of this data synchronously. This is obviously not an optimum solution since these tasks can be processed concurrently. Let's turn this into an asynchronous process with goroutines and channels. Asynchronous {
	start := time.Now() wg sync.WaitGroup

	dataCh := ( model.SimpleData, )

	wg.Add( ) { wg.Done() data := dataCh {
			wg.Add( ) { wg.Done()
				basic.Process(data)
			}(data)
		}
	}() i, _ := allData {
		dataCh <- allData[i]
	} (dataCh)
	wg.Wait()
	elapsed := time.Since(start)
	fmt.Printf( , elapsed)
} worker.NotPooledWork(allData) //// worker/notPooled.go func NotPooledWork (allData []model.SimpleData) var make chan 100 1 go func () defer for range 1 go func (data model.SimpleData) defer for range close "Took ===============> %s\n" //// main.go // Process Start processing all work 
Took ===============> 101.191534ms Here, we create a buffered channel of 100 and add all the data passed to to the channel. Since it is a buffered channel, it would not be able to enter any more data than 100 until data is extracted from it. Which we do it inside a goroutine. We range over the channel, extract data from it, add a goroutine, and process. There's no limit to how many goroutines we are creating here. This can process as many tasks as passed. Theoretically, it is possible to handle as much data as possible given the resources needed. If we run this, . Crazy right. Not quite. NoPooledWork we complete 1000 tasks around 100ms The Problem Unless we have all the resources on earth, we have limited allocations we can do for a given period of time. The minimal size of a goroutine object is 2K, but it can reach up to 1GB. The above solution runs all the tasks concurrently and let's say for a million of these tasks, it can quickly run out of memory and CPU of the machine. Either we have to upgrade our machine or find a different solution. Computer scientists have thought about this problem long before and came up with a brilliant solution called Thread Pool or Worker Pool. The idea is to have a limited pool of workers to process the tasks. Once a worker is done processing one task, it goes for handling the next task. So tasks keep waiting to be processed. This reduces the burst on CPU and memory and rather distributes the tasks over time. The Solution: Worker Pool Let's fix the earlier problem with worker pool implementation {
	start := time.Now() wg sync.WaitGroup
	workerPoolSize := dataCh := ( model.SimpleData, workerPoolSize) i := ; i < workerPoolSize; i++ {
		wg.Add( ) { wg.Done() data := dataCh {
				basic.Process(data)
			}
		}()
	} i, _ := allData {
		dataCh <- allData[i]
	} (dataCh)
	wg.Wait()
	elapsed := time.Since(start)
	fmt.Printf( , elapsed)
} worker.PooledWork(allData) //// worker/pooled.go func PooledWork (allData []model.SimpleData) var 100 make chan for 0 1 go func () defer for range for range close "Took ===============> %s\n" //// main.go // Process Start processing all work 
Took ===============> 1.002972449s Here, we have a limited number of worker size, 100. We spin up exactly 100 goroutines to process the tasks. We can think of channels as queues and each worker goroutine is a consumer. Multiple goroutines can listen to the same channel but each item on a channel would be processed just once. Channels work like queue in Golang This is a nice solution, if we run it, we see it takes to finish all the tasks. Not quite 100ms, but we don't need it. We get a much better solution that distributes the load over time. 1s Handling Error We are not quite done yet. This looks like a complete solution except it isn't. We are not handling errors here. So let's create a scenario where we have to handle errors and let's see how we can go about it. {
	start := time.Now() wg sync.WaitGroup
	workerPoolSize := dataCh := ( model.SimpleData, workerPoolSize)
	errors := ( error, ) i := ; i < workerPoolSize; i++ {
		wg.Add( ) { wg.Done() data := dataCh {
				process(data, errors)
			}
		}()
	} i, _ := allData {
		dataCh <- allData[i]
	} (dataCh)

	wg.Add( ) { wg.Done() { { err := <-errors:
				fmt.Println( , err.Error()) <-time.After(time.Second * ):
				fmt.Println( ) }
		}
	}() (errors)
	wg.Wait()
	elapsed := time.Since(start)
	fmt.Printf( , elapsed)
} {
	fmt.Printf( , data.ID)
	time.Sleep( * time.Millisecond) data.ID % == {
		errors <- fmt.Errorf( , data.ID)
	} {
		fmt.Printf( , data.ID)
	}
} worker.PooledWorkError(allData) //// worker/pooledError.go func PooledWorkError (allData []model.SimpleData) var 100 make chan make chan 1000 for 0 1 go func () defer for range for range close 1 go func () defer for select case "finished with error:" case 1 "Timeout: errors finished" return defer close "Took ===============> %s\n" func process (data model.SimpleData, errors <- error) chan "Start processing %d\n" 100 if 29 0 "error on job %v" else "Finish processing %d\n" //// main.go // Process We have modified our method to handle some randomized errors and add it to the passed channel. So, to handle errors in a concurrent model, we need channel to hold the error data. After all the tasks are done processing, we check the channel for errors. The error object holds ID of the task so that if needed, we can process those again. process errors errors errors This is a better solution than the one which was not handling errors at all. But we are not done with it yet, in the second part, we will go over how we can make a dedicated worker pool package that is robust and can handle concurrent tasks with a limiting worker pool number. Exercise We don't always want to keep processing our tasks if multiple tasks keep failing. Indicates that we have a deeper issue to resolve first. For exercise, you can try to improve the last solution to stop processing tasks, when there are more than 2 tasks that have resulted in an error. Send pull request in the original when you find a solution and I will send a review for that. repo Conclusion Golang's concurrency model is powerful enough to just build a worker pool solution without much overhead, that's why it's not included in the standard library. But we can spin up our own solution that fits our needs. I will write the next article very soon. Til then, stay tuned.

Concurrency in Golang And WorkerPool [Part 1]

Let us learn about MongoDB, Mongoose, Node, and other tech by building a simple URL shortener project. Have you ever wondered how you could create a quick URL shortener for yourself? Like how twitter shortens your links when you share them? Or how bit.ly works? Sure enough, these are complicated companies, but the concept for URL shorteners is simple. Let's learn about MongoDB and other backend tools by actually building this project in 7 steps. Introduction to Project We will be using this from codedamn to really, like really create and evaluate our hands-on project and see the feedback. free URL shortener classroom We will be using the following technologies: Mongoose as the ORM MongoDB as the backend database Node.js as the backend A simple embedded JS file as the frontend Lab 1: Setting up the Express server The link of this lab is here It's a fairly straightforward lab. We have to just create a route /short which should respond appropriately. This code would let us pass: express = ( ) app = express()

app.get( , (req, res) => {
	res.send( )
})

app.get( , (req, res) => {
	res.send( )
})

app.listen(process.env.PUBLIC_PORT, () => { .log( )
}) // Initialize express server on PORT 1337 const require 'express' const '/' 'Hello World! - from codedamn' '/short' 'Hello from short' console 'Server started' Lab 2: Setting up our view engine The link of this lab is here We're using a single .ejs file, so let's explore that a little. Again, a very simple lab because we just have to change the name of the variable. This should get us done: express = ( ) app = express()

app.set( , )

app.get( , (req, res) => {
	res.render( , { : })
})

app.listen(process.env.PUBLIC_PORT, () => { .log( )
}) const require 'express' const 'view engine' 'ejs' '/' 'index' myVariable 'My name is John!' console 'Server started' Lab 3: Setting up MongoDB The link of this lab is here In this lab, we'll connect to MongoDB correctly and insert a record, just for the record This is the solution which should get us to the next lab: app.post( , (req, res) => { db = mongoose.connection.db db.collection( ).insertOne({ : })

	res.json({ : })
}) mongoose.connect( , { : , : })
mongoose.connection.on( , () => { app.listen(process.env.PUBLIC_PORT, () => { .log( )
	})
}) '/short' const // insert the record in 'test' collection 'test' testCompleted 1 ok 1 // Setup your mongodb connection here 'mongodb://localhost/codedamn' useNewUrlParser true useUnifiedTopology true 'open' // Wait for mongodb connection before server starts console 'Server started' Lab 4: Setting up a Mongoose schema The link of this lab is here Finally, we define a schema in the models/url.js file for proper handling with Mongoose, and here's the code for that: mongoose = ( ) shortId = ( ) shortUrlSchema = mongoose.Schema({ : { : , : }, : { : , : , : shortId.generate
  }, : { : , : , : }
}) .exports = mongoose.model( , shortUrlSchema) const require 'mongoose' const require 'shortid' const new full type String required true short type String required true default clicks type Number required true default 0 module 'ShortUrl' Also, as part of the challenge, we update the `/short` route now. app.post( , (req, res) => { record = ShortURL({ : }) record.save()
	res.json({ : })
}) '/short' async // insert the record using the model const new full 'test' await ok 1 Lab 5: Linking the frontend, backend, + MongoDB This is also a simple lab. We just have to update the route to extract the URL passed and store it in the database using our schema. app.use(express.urlencoded({ : }))

app.post( , (req, res) => { fullUrl = req.body.fullUrl .log( , fullUrl) record = ShortURL({ : fullUrl
	}) record.save()

	res.redirect( )
}) extended false '/short' async // Grab the fullUrl parameter from the req.body const console 'URL requested: ' // insert and wait for the record to be inserted using the model const new full await '/' Lab 6: Displaying short URLs on the frontend Now, we display the URL set on our website using the .ejs variables passed. app.get( , (req, res) => { allData = ShortURL.find()
	res.render( , { : allData })
}) '/' async const await 'index' shortUrls Lab 7: Making the redirection work Finally, we link up the redirection scheme using dynamic express routes and correct status codes. app.get( , (req, res) => { shortid = req.params.shortid rec = ShortURL.findOne({ : shortid }) (!rec) res.sendStatus( ) rec.clicks++ rec.save() res.redirect(rec.full)
}) '/:shortid' async // grab the :shortid param const // perform the mongoose call to find the long URL const await short // if null, set status to 404 (res.sendStatus(404)) if return 404 // if not null, increment the click count in database await // redirect the user to original link Conclusion And we can call it a day! You just built a fully working URL shortener by yourself using Express + Node + MongoDB. Give yourself a pat on back! The final source code is available on GitHub If you have any feedback on this article or codedamn classrooms, feel free to reach out to me on and let's discuss :) twitter Also published at https://dev.to/mehulmpt/create-a-url-shortener-project-using-mongodb-node-but-hands-on-1ai2

How To Create a URL Shortener using MongoDB And Node

K Nearest Neighbor Predictive Classification Javascript implementation Utilizing D3 Visualizations At its heart, the k Nearest Neighbor technique treats the problems it receives as issues of pattern recognition in order to properly classify data. It does this by ‘learning’ the graphical space in which the different categories reside; implicit is the assumption that each classes will be clustered distinctly. Computers learn similarly to living things, based on as much previous data as is available, predictions are made in order to do work more efficiently. Typically, has not been thought of as a good language for the implementation of algorithms, and for good reason. When first developed, the language was incredibly inefficient and a misunderstanding of underlying concepts or blatant disregard for best practices led to a proliferation of terrible libraries and examples. Further, the community had been leaning towards OOP, but there has slowly been a movement to a more functional style. Over the years optimizations have been made, libraries have been refactored or newly created in ways that lend to performing complex processing, the underlying language implementation has had several iterations, and computing machines are also equipped with much greater processing capabilities. Javascript machine learning All of this leads to an interesting space in which we, as Javascript developers, find ourselves. On the cusp of a rapidly transforming widespread technology, the possible applications of which are expanding, we have the opportunity to contribute to and improve a technology which is becoming increasingly fundamental to our daily experience. D3 has been one of the libraries most widely used with the goal of data visualization, and is depended upon several widespread visualization tools today; this is what will be used to plot out k Nearest Neighbors algorithm. If you need an introduction to the library before applying it to something more involved, an introducing it might of use. earlier post of mine Now that we’re all excited about our role in this field, lets explore a simple implementation of arguably the most fundamental machine learning algorithm, k Nearest Neighbor. While on the less complex side of predictive computation algorithms, it is widely used in the data analysis industry, and is one of the best performing for several problems, one such example is the prediction of behavior between proteins, or protein-protein interaction (PPI): “In spite of its simplicity to implement, when a large data set or numerous features are used the computational cost and memory requirement grows rapidly. This method has been used (not widely) in PPI prediction problem.” - , 2013 Zahiri, Bozorgmehr, Masoudi-Nejad To illustrate this, lets explore the example pictured to the left, for which can be cloned down and explored, or you can play with it in ! Note that the dummy data is dumb; as should be inferred, they are not from a valid scientific study nor do the predictions represent a medical diagnosis in reality. It is loosely based on the findings of , and related published research outlining the correlation. which indicates that high levels of the proteins CA15–3 and CA27–29 could indicate the presence of a tumor in the breast; it should also be noted that these tracers also get spiked in cases of ovarian, liver, and other breast diseases. Units are calculated in units per milliliter of blood, with levels over 30U/mL considered high-risk. The training data can be changed in the source code, and the graph will handle the new data by fitting it to the graph as long as it is in the correct format: {x: num, y: num, neighbors: array, identity: num}. the repo this CodePen this medical study To create this graph, there will be a series of functional tasks to accomplish, each of which my be solved differently, and modularity may be implemented in the functions to various degrees or with a different opinions with regards to the ideal separation of concerns. The following is my own opinion on how to best organize the program. The pieces we will address as individual problems in the construction of our larger program include: Elements on the page: A board Training data points Input data points Classification functions: Calibrate(smooth) data to fit to graph Find k nearest neighbors Classify each datum Placement functions: Place training data Place input data OPTIONAL: place circle containing input’s k nearest neighbors To create the board, simply located the element on the page to which you want to append: var = 300;var = 300;var = d3.select(document.getElementById(‘body’)).append(“svg”).attr(“width”, w).attr(“height”, h); h w svgSelection Don’t forget to add a border: var = svgSelection.append(‘rect’).attr(‘x’, 0).attr(‘y’, 0).attr(‘height’, h).attr(‘width’, w).style(“stroke”, ‘black’).style(“fill”, “none”).style(“stroke-width”, 4); border At the top of the js page, include the following variables: and are the dimensions of the board, is an object which holds the highest values for the each of the training variables, into , each data point will be pushed it so that future inputs includes training and input data, and the correspond with the various possible states of classification. h w max addData riskColors To figure out where to place the points on the graph, we need to perform several operations in order to fit the data’s range within the size of the graph. The helper functions and will be utilized in our placement functions. Lets build those out: findMax smooth var = (data) => {var results = {x: 0, y: 0};data.map(datum => {let x = datum.x;let y = datum.y;x > results.x ? results.x = x : false;y > results.y ? results.y = y : false;});return results;} findMax var = (datum, max) => {datum.x = (datum.x/max.x) * h;datum.y = (datum.y/max.y) * h;return datum;} smooth Next, we need a function that will draw the points of data on the graph: var = [];//global container for training and input datavar = [‘mediumturquoise’, ‘crimson’, ‘rebeccapurple’];//indices of riskColors correspond with a classification allData riskColors var = (datum, encircle) => {svgSelection.append('circle').attr('cx', datum.x).attr('cy', datum.y).attr('r', 5).attr('fill', riskColors[datum.identity])encircle ? drawHalo(datum) : false;//draw halo implemented below!}var ;var = (data, adjust=true, encircle=false) => {max === undefined ? max = findMax(data) : max = max;data.map(datum => {allData.push(datum);adjust ? smooth(datum, max) : null;placeDatum(datum, encircle);});} placeDatum max populateGraph will be used set to true for input data, invoking in the , which draws a circle around the data point containing its nearest neighbors. At this point, all that will be on the screen is a blank rectangle. All this needs now is a set of training data and to be invoked. At the bottom of the file, place: Encircle drawHalo placeDatum k //this may be replaced with any data in which there are//identified objects with the three properties found below var = [{x: 32,y: 40, identity: 1},{x: 5,y: 27, identity: 0},{x: 17,y: 9, identity: 0},{x: 50, y: 29, identity: 1},{x: 22,y: 46, identity: 1},{x: 10,y: 27, identity: 0},{x: 5,y: 9, identity: 0}]; trainingData populateGraph(trainingData); The dummy data should now render to the page, clustered and colored. This is still pretty boring, to make it interesting some interface for user interactivity needs to be created. In the body of the HTML, we’ll be adding a form with the id which contains input bars for each of the variables: submitData <div id=”submitData”><form onsubmit=”handleSubmit(event)”><b>Add data:</b> <br/>x:<input id=”x” type=”text” placeholder=”CA15–3 in U/mL”/><br/>y:<input id=”y” type=”text” placeholder=”CA27–29 in U/mL”><button>Submit</button></form></div> The handler should prevent the page from re-rendering, get the inputs from the page, correctly format them to an object to then be passed into a drawing function as a data point: var = function(e) {e.preventDefault();var x = document.getElementById(‘x’).value;var y = document.getElementById(‘y’).value;var = {x: x,y: y,neighbors: [],identity: 0};classifyAndPlot([datum]);//classify and plot not yet defineddocument.getElementById(‘x’).value = ‘’;document.getElementById(‘y’).value = ‘’;//clear the input bars} handleSubmit datum The formatted data then is passed into a function to categorize them based on the points closest to them, which is calculated by looping through all of the data points, calculating distance using good old Pythagoras’s formula. Lets all say it together: “a squared plus b squared equals c squared.” Good! Except this time, it’ll be (xDiff²+yDiff²): √ var = function(data, k) {return data.map(datum => {allData.map(neighbor => {var xDiff = Math.abs(datum.x — neighbor.x);var yDiff = Math.abs(datum.y — neighbor.y);var distance = Math.sqrt(Math.pow(xDiff, 2) + Math.pow(yDiff, 2));var currentFartherNeighbor = datum.neighbors.find(dataNeighbor => dataNeighbor[0] > distance)datum.neighbors.length < k ?datum.neighbors.push([distance, neighbor.identity]) :currentFartherNeighbor ?datum.neighbors.splice(datum.neighbors.indexOf(currentFartherNeighbor), 1, [distance, neighbor.identity]) :false;datum.neighbors.sort((a, b) => b[0] — a[0]);return datum;});});} findNeighbors The above functions loops over the input data, and compares each datum to each of the array, pushing into the neighbors array if there are less then neighbors or if a comparison yields a distance smaller than any currently in the array. In the case of the latter, the farthest neighbor also needs to be spliced out. Then the datum is returned with the neighbors array populated as expected. However, one more component is necessary in order to paint them on the page. We have functions to do the later which may be reused, but before passing data into them to render onto the page, we should consider what the input looks like when this function is invoked, and how to format the input to be successfully accepted by the function. allData k populateGraph var = (data) => {//fit the data to the graph rangedata.map(datum => smooth(datum, max));//populate neighborsfindNeighbors(data, 3);var formattedData = data.map(datum => {var neighborsIdentity = datum.neighbors.reduce((acc, neighbor) => {acc[neighbor[1]] === undefined ? acc[neighbor[1]] = 1 :acc[neighbor[1]] = acc[neighbor[1]] + 1;return acc;}, []);var maximum = Math.max(…neighborsIdentity.filter(neighbor => neighbor));datum.identity = neighborsIdentity.indexOf(maximum);return datum;});populateGraph(formattedData, false, true);} classifyAndPlot That’s it! We have everything we need to train our algorithm and predict unclassified input results based on present clustering trends. While it has been noted that this is not the most efficient solution to most machine learning problems, but is useful in cases where there are discrete categories with clear distinctions. As someone interested in medicine and public health, I’ve learned that cancer diagnosis is one of those cases, in which you are assigned either a positive or negative diagnosis based on various biometric measures related to understood oncological trends. This is the idea that the example data is modeled after, but it can be modified for any training data in which each datum has been classified. What are some ideas for application that come to mind for you? Let me know below in the comments. Thanks for reading, now go make your own improved implementations! Jacob

Stories, News & Mentions about AXLRDATA

Axlrdata’s Stories on HackerNoon