Creating neural networks without human intervention

…And where is the blockchain in it?

Scientists have long since theorized about machines, capable of simulating human thinking. And following decades of research and development, we have finally reached a point where AI is not only a reality - it’s hard to imagine living without it.

But what is AI anyway? When we’re speaking about modern AI, one of the most popular approaches is called Deep Learning. Inspired by the way biological brains process information, deep learning involves the development of deep neural networks.

What’s so Deep about Neural Networks?

A deep neural network is a pipeline of operations that processes data to identify some pattern as a solution to a specific problem. Deep neural nets have a layered structure, and the ‘thinking’ part happens inside the hidden layers between the input and output. The ‘deep’ indicates that there are several internal layers, so the incoming data goes through more complex transformations, as opposed to simpler artificial neural networks.

There are many applications of deep neural networks, and a very basic example is image recognition. In this case, the neural net would take an image as an input and would try to guess with some probability what objects are on it.

However, to make any neural network good in its job, it first has to be trained. In other words, the algorithm has to fine-tune its inner workings, i.e. the parameters that assign values of connections between its neurons. Training requires the neural network to be fed with data and then, as it returns some output, to have its parameters corrected in a way that would make its next prediction better.

So usually a deep neural network would initially spit some nonsense result, that is very far from the truth. Typically, a human researcher will try to adjust the architecture of the algorithm by adding a layer, changing the type of another, etc. This process goes on until we’re happy with the AI accuracy.

So adjusting the architecture is the only part that requires human intervention. But it usually is a very tedious and time-consuming process. A hit-and-miss, so to say.

The good news is that computer science is on the way to automate this, too. The answer is a technique called Neural Architecture Search (NAS).

How a NAS works, courtesy of Google Blog

A NAS utilizes an algorithm called a controller (most often a neural network). It creates a child neural network and proceeds to optimize its architecture until it finds the best solution for a particular problem, i.e. until the child network achieves its highest accuracy.

As you see, a well-working NAS eliminates the need of humans from the process of deep neural network creation.

So do the NAS algorithms really work?

Our experience concurs. In our experiments, we manually created a deep neural network that would predict price movements of ETH/USD. And then we used the Auto-Keras NAS algorithm to find the best configuration for this same task.

Guess what? The NAS was able to do almost the same job as we did in 6 months of manual labour! In fact, the NAS-created AI performed slightly better than the human-made one.

If that’s not convincing enough, check out what Google achieved with NASNet. In a paper published in the summer of 2017, Google described how their NAS algorithm was able to outperform their human data scientists in creating highly accurate deep neural networks.

Let that sink in for a moment. We have reached a point where deep learning can be automated entirely… Does this put humans out of the equation?

As impressive as NAS algorithms are, they come with a caveat. Their main limiting factor is computational power. The more you have, the faster the controller algorithm would iterate through architectures before reaching the best fit solution. And since most machine learning problems require a fair bit of ‘searching,’ using NAS turns into an arms race for computing power.

Thus, being resource-heavy makes the NAS algorithms relatively inaccessible to everyone, but a few technological giants with vast access to computing power and data.

However, a project called ScyNet is going down a path that could substantially lower the barrier to the access of NAS algorithms. To do that, we create a decentralized open network where people are rewarded for sharing computational resources and data.

Decentralize Deep Learning with blockchain

Yes, we said it. Blockchain.

For ten years has this technology been proving its utility. Bitcoin is an example of how a completely decentralized network of computers could establish consensus on money transactions without the need for a central authority.

So what if we use blockchain to make a decentralized group of computers collaborate in creating AI?

In other words, could we create something that resembles a NAS algorithm, being simultaneously executed in a whole blockchain network?

How is that possible?

In basic terms, a NAS requires data and computation to create its solutions and end clients that are willing to pay for said solutions to offset the resource investment costs. One way to allow everyone to invest their resources in running a blockchain-wide NAS would be to introduce a protocol that governs the interactions between the following three types of players.

The first type is the Data node. Once set up on ScyNet, these nodes can sell data to other participants in the network.

Who would buy the data? Meet the NAS node. Similarly to the Bitcoin miners, these are computational nodes. But instead of calculating hashes, the ScyNet miners execute NAS algorithms with the help of their video cards.

And the last type is the Client node. These are the participants who are ready to pay for the machine learning solutions, created by the NAS nodes.

Tournaments for validation

A network that relies heavily on demand for AI solutions would suffer a lot from bad rap. And the worst reputation you’ll ever get is if the AI you’re selling to customers turns out to be crap. Especially if you pretended it isn’t.

This is where our blockchain protocol comes to the rescue. It provides a way to validate the performance of AI solutions. Furthermore, it does so without the need of the NAS nodes to reveal the exact functionality of their configurations, i.e. what’s under the hood. Hence, the ScyNet blockchain is designed as what some call a Zero-Knowledge Proof protocol.

The validation process happens through tournaments. These are regularly-scheduled events, where NAS nodes pay a fee to have their AI agents compete against each other in solving a certain problem. 

Then, to evaluate performance, some of the nodes on ScyNet act as validators.

Let’s take, for example, real-time price predictions. An AI configuration is trained to give hourly predictions for the BTC/USD pair. During the validation period, the NAS nodes send their agents’ real-time forecasts to validators.

However, the predictions are AES encrypted, meaning for the time being, validators are unable to actually read and take advantage of the information. As you probably guessed, the encryption is to prevent role abuse. Otherwise, people could become validators only to get real-time predictions for free.

After the validation period expires, the NAS nodes eventually send the decryption key to the validators. Thus, the latter could compare the predictions with historical price action and calculate the AI accuracy.

Going further, every validator sends their evaluations to the blockchain. However, assessments could be subjective. That’s why the blockchain consensus takes the median of all the calculated performances per agent. There is a penalty for deviating from the median, so the validators are better of keeping it tidy with their performance evaluations.

And finally, we have it - the full tournament ranking! Anyone can see which agents have performed better or worse and exactly how accurate have they been on the task. 

This validation process gives the client nodes peace of mind about the marketed accuracy by providing a solid proof the NAS nodes aren’t selling snake oil.

Last but not least, having a fair ranking means we can reward the best AIs with some portion of the tournament fees. This provides another financial incentive for the NAS nodes to compete and have their AI agents validated, in addition to the bragging rights for a high finish.

What’s good about this blockchain network?

So let’s summarize. First, a network like ScyNet gives a financial incentive for every participant to share resources for the decentralized creation of AI.

Second, there’s open access to these machine learning solutions. They aren’t developed and locked behind closed doors, as is the standard case. And since it’s a free, decentralized marketplace, ScyNet has the potential to bring prices down for the end clients. Plus, all the agent performance is validated, so the client nodes know what they are paying for.

The other good thing here is that such a network could be completely autonomous. These NAS nodes, these clients, etc. don't require human intervention. So the blockchain network could run on its own. And that's entirely possible because NAS algorithms are wholly automated themselves - give them data and video cards and they'll do the job. 

However, it’s worth noting that such a network would still benefit from humans working on new kind of NAS algorithms and developing new kind of applications. So don’t be wary of us primates becoming obsolete just yet - we’ll still be needed, although moving our contributions to another level of abstraction in the Machine Learning world.

Finally, such a protocol could work for numerous problems. Real-time predictions, text-to-speech, neural translations could be just the beginning, followed by all types of classifications and regression problems. Or even reinforcement problems - a neural network that learns how to play a game or drive an autonomous car, for example.

All these are possible applications for ScyNet. Will they be built eventually? It’s up to you!

For now, the core dev team is focused on launching the first use-case, which is in the crypto trading domain. However, the project is open-source, so anyone, who wants to harness the network resources in a NAS algorithm, will be able to develop and define their particular problem type.

This article is based on a presentation we had before the blockchain community of dev.bg. For more details and a technical explanation, watch the (with added English subs).

Stay in touch

If you want to keep up with the latest around ScyNet, feel welcome to join our Discord server. This is the ground zero for all ScyNet-related updates before they spread to our other social media channels (namely Twitter and Facebook).