The are a series of interviews featuring various important voices and perspectives in the Bitcoin ecosystem. Each interviewee was carefully selected and asked the same group of questions. The main goal was to provide the world with a collection of commentary and opinions on the state of the current Bitcoin and crypto market. Some of the names you will recognize while others don’t wish for the limelight but have great insight and experience we all can learn from. Bitcoin Bear Market Diaries These interviews are raw and unfiltered with no agenda other than giving each individual their opportunity to speak their mind. If you like what you see, please share with your friends. If something offends you, you should probably X out and find another crypto fluff piece that gives you warm fuzzies. If you don’t know who Max Keiser or at least heard his name before, you really need to get out more. Max has been into Bitcoin since 2011 when it was $1 and has been making bold market predictions since. Max is also co-founder of Heisenberg Capital one of the first crypto VC funds and has invested in companies like Kraken, ₿itstamp, Shapeshift, Bitfinex, Bitso, BitPesa, BitPay, CasaHODL, Unocoin, Uphold, Minds, BitRefill, ABRA, Kim and the list goes on. Name Max Keiser Country Various How do people know you? Keiser Report How long have you been into Bitcoin? 2011 ($1) Best Bitcoin experience? Satoshi Roundtable Worst Bitcoin experience? Craig Wright What do you think is Bitcoin’s biggest threat? Bitcoin has no known predators. It is the ultimate apex predator. All fiat money will die. What are your thoughts on the various Bitcoin forks? Imposters. What are you optimistic about in this space? Banks, and central banks, will continue failing. As they crumble, Bitcoin will engulf the global economy. Jack Dorsey, Bitcoin maximalist, is a huge positive. Biggest regret during the last Bull market? Not writing a book. What have you learned during the current bear market? Never too late to write a book. What is the biggest fail you have seen during the bear market? BitTorrent. What do you think helped trigger the current bear market? CNBC. What kind of damage do you think latest price drop has done to buyer sentiment? None. When prices start going up, they’ll all be back, and many more. How do you feel about the current state and future of lightning network? It’s the future. CryptoSprings in Palm Springs last fall, helped launch Lighting in a big way. Crypto Springs is back again this year. It’s the most important event of the year, IMO. Elizabeth Stark and Jack Dorsey will blow people’s minds this year. What are your thoughts on HODLing Bitcoin? It’s the number-1 use case. What are you thoughts on alt-coins? I don’t spend much time thinking alt-coins. Do you hold any? If not, why? Nope. Waste of time. What kind of impact do you think the radical drop in alt-coins will have on their future? None. Thoughts on the notion of bitcoinization? It’s coming. What Bitcoin startups are you excited about? Our VC fund, Heisbenberg Capital was, I think, the very first VC fund in this space, (making investments in BTC startups at about the same time as Roger Ver). We are very excited about our investment in Kraken, a massive Unicorn. We just invested in CASAhodl recently. We’ll be making an investment in BitRefill very soon. And Samson Mow’s “Infinite Fleet.” We are focusing on Lighting right now and have talked to a few startups in that space that we’ll be funding. What “crypto influencers” do you think get it wrong and why? Crypto influencers very quickly become boiler rooms. Keep your head up, or you’ll get REKT. What “crypto influencers” do you think get it right and why? He keeps it real. @cryptoblood. What’s it going to take for this Bear market to turn around? It bottomed when the Fed signaled ‘Infinite QE’ recently at around $3,300 BTC. Global Central Banking is about to go bust. How bullish are you on Bitcoin despite this recent pullback? My position from 2011, when I recommended BTC at $1 has not changed, and has never varied. The price is going to $100,000. We’re still on track for that. Any tips you want to give to people new to Bitcoin? Read the white paper. Study it. It has 90% of everything you need to know, with one caveat. Replace the word ‘cash’ with ‘Gold.’ It’s clear to anyone who knows anything about the history of cypherpunks, or who has read Satoshi’s writings and chat logs, that Bitcoin is meant to be digital Gold, not digital fiat. This is where I differ from Roger Ver. I believe he fundamentally misreads the White Paper and after a few years now, we see my position on this is winning in the market. Not to take anything away from Roger. He’s a pioneer and has probably got as many people interested in Bitcoin as Keiser Report has. Side note: We think we’ve created more than 300,000 or so Bitcoin millionaires around the world who have been watching our show starting when BTC was $1. Best tips for storing Bitcoin? CASAhold/ Trezor Name some of your favorite information sources and/or podcasts in the space. There are two sources of info that are best in class and worthwhile: Jameson Lopp and Trace Meyer. Any last words of wisdom? If you’re not prepared to hold for 10 years or more, you’re gambling, not investing. Stay tuned for the next Bear Market Bitcoin Diaries. You can find more volumes and articles here!

BitTorrent

Funding

Trace

Bitcoin and Tacos for the win!

Bitcoin Bear Market Diaries Volume 19 Max Keiser

It always seems impossible until it’s done. — Nelson Mandela

Ethereum

2022 - HackerNoon Contributor of the Year - How To

2022 - HackerNoon Contributor of the Year - Smart Contracts

2022 - HackerNoon Contributor of the Year - Solidity

Ask me a question! 

Nominated for 2022 - HackerNoon Contributor of the Year - How To

Nominated for 2022 - HackerNoon Contributor of the Year - Smart Contracts

Nominated for 2022 - HackerNoon Contributor of the Year - Solidity

Fixed point math in Solidity

Flow

In my previous article ‘ ’ I highlighted the size of the global freelancing economy — over one hundred million freelancers and counting. Moreover, the large super-platforms have millions of users apiece. Decentralised Freelance Platforms are Coming After the Large Incumbents The many freelance platforms are a magnet for both freelancers and clients seeking jobs to be done, the reason, traffic generation (a work supply for freelancers). In this article I will explore how and why freelancers may be trapped on the super-platforms, and where they may be able to save themselves. Its all about the Money So on the surface, everything looks rosy for both the clients and the freelancers. The freelancers can find work and clients can quickly post a job/project for completion by a global workforce. Do not get me wrong, these super-platforms (such as Freelancer, Upwork, Fiverr, and TaskRabbit) provide work for millions of independent workers, however the platform fees levied by these corporations are not paid for by the clients! Freelance platform fees are absorbed by the freelancer. One could argue without such platforms the consistency of work for a freelancer to undertake may/will diminish. Yes. But, the truth of the matter is that the vast majority of freelancers have little choice. To mitigate an inconsistent pay-check, freelancers are drawn to the super-platforms. I dare to say that the choice to be a freelancer in the modern economy is not a choice at all. Furthermore, the cost of living pressures required to keep food on the table, and pay your rent, is not a seldom struggle. What if I were to tell you that all the power is actually in the hands of the client? Think about it, it’s the client (a freelancers de facto employer) who makes the market. Freelancers are coming to the platforms to get paid by the client. It is the client who posts the project/job. No clients mean freelancers with no food on the table. As long as there are hungry freelancers the super-platforms can charge what ever they wish. How did we get here? I would argue that the growth of the super-platforms is a spawn of the VC market, the global casualisation of the workforce, and the adoption-rate of internet usage. Obviously global technology businesses have capital and operational costs that only increase with growth and scale. And any risk-taking entrepreneur should be able to be rewarded for his/her endeavours. I have zero issue with platform innovators being successful. So why are the super-platforms charging such ? Simple — VC money demands a return on investment higher than inflation, plus the risk premium self-assigned by the king makers. large fees Mira Muurinen has written an interesting , interviewing Fabrice Grinda (co-founder of venture fund FJ Labs) talking about what VC’s look for when investing in marketplaces. With investment being contingent on such large unit cost metrics it is no wonder the platform fees for the super-platforms are so high. article I am not advocating for a freemium system, just a more even-handed power balance. A mutiny is rumbling What if freelancers had the power to direct their clients to an unchartered nirvana? A place where the super-platforms do not ‘tax’ them with high fees. What if clients could recognise the hardships of inconsistent work, including the mental health anxieties, and venture over to said freelance nirvana? I have two magical words for you — ‘ ’. decentralised freelance , in his article ‘ ’, outlines the current decentralised freelance platforms operating. He also believes — Michiel Mulders Working Remotely For Crypto — Is it possible? the market still needs to grow to find a larger variety of services asked and offered Freelancers need to jump-ship and explore the decentralised world. Then once there, they need to invite, talk, beg, and plead their clients to join them. Clients need to start making some ethical choices when it comes to how and where they engage with the freelance talent they rely upon. We now have a zero marginal cost solution to the super-platform fee problem, we just need a freelance mutiny en masse to pull clients over. <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

fees

Super

Read My Stories

Why Does the Global Freelance Economy Need a Mutiny?

You’ve added your favorite products to the shopping cart. Now, it’s time to take your credit card out of your wallet and make the payment. As the site is secure and you have already done shopping on this e-commerce website in the past, you don’t think much before entering your credit/debit card details during the checkout. But have you ever imagined that your card details can be stolen when you are making the payment to your favorite online store? Yes, my friend, hackers can steal your card details and make you broke. In fact, stealing card details have become hackers’ new tool to swipe online shoppers’ money. And the term used for this unethical, malicious technique is . Formjacking What exactly is formjacking? Formjacking is almost the same as ATM skimming, the only difference is it happens online. After entering credit or debit card details, when a user of an e-commerce website clicks on submit or its equivalent button, a malicious javascript code injected by a hacker to the e-store collects the credit card details of the user and send this information to the hacker’s server. Once the hacker has the credit card details of users, he/she can use the details to do credit card frauds or sell the details to other cybercriminals on the dark web. Symantec has created an infographic to explain the process. Fromjacking is not a small threat. According to , formjacking attacks soared with an average of 4,800 websites compromised each month. Symantec Popular brands like British Airways, Ticketmaster, Newegg, and Feedify publically reported formjacking reported by the hacker group . Magecart Who is Magecart? Magecart is a common name given to a group of seven major card-skimming cybercriminals. Magecart threat group was responsible for the recent attack against Ticketmaster, Feedify, British Airways, Everlast, the National Republican Senate Committee, Newegg, Oxo, and Groopdealz. All these seven use the same toolset version, but their tactics and techniques differ. RiskIQ and Flashpoint created a detailed document on different groups of Magecart. You can access the PDF from . From this report, you can learn about each group’s tactics, targets, and victims and what makes these groups different from each other. You will also know about growing criminal underworld that helps these groups work and monetize their campaigns. here How to avoid formjacking Magecart formjacking cybercriminals use malicious JavaScript to steal customers’ credit card information. As they exploit customer payment forms, a web browser-based script blocker can provide protection against formjacking attack. Following are some effective script blockers for popular browsers: ScriptSafe is good for blocking script for Google Chrome and Chrome-based browser like Vivaldi NoScript is effective for blocking intrusive script for Firefox Safari users can use JS blocker As smaller online shopping sites might not have high protection level as the bigger sites have, avoiding shopping from small players can be a proactive way to stay safe online. But attacks on leading brands like British Airways proves that none is safe. But still, any small site is more likely to get attacked than a bigger site. Many times attackers go through the third-party tools and applications. If you’re a site owner, you should test any software update before installing it. Also, you should look for tools that check your website realtime and inform you if there is any change. <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Google

Founder of QuickScream

Formjacking: Hackers’ New Favorite Way to Steal Credit Card Information (And How to Avoid it)

Gandcrab — the Most Prevalent Ransomware in 2019 File-encrypting ransomware has gone through significant ups and dramatic downs over the past few years. Overshadowed by the influx of malicious cryptocurrency mining applications in late 2017, this area of cybercrime took a nosedive only the most durable strains could survive. A sample called GandCrab made its debut in the midst of this hiatus and became a game changer. With the first infection instances documented in January 2018, this lineage quickly gained traction and came to dominate the extortion landscape. Its original variant was crafted competently enough to prevent free decryption, so the users who suddenly discovered their personal files being appended with the bizarre .GDCB extension had no chances to restore hostage data beyond the ransom way. Then came editions that blemished files with the .CRAB and .KRAB strings, superseded by a series of the pest’s personas using random victim-specific extensions. As of March 2019, the GandCrab family has spawned 9 distinct variants along with subversions and . Courtesy of Bitdefender that recently supporting iterations up to 5.1, numerous victims were able to reinstate their valuable data assets. However, the infection in its current form is not decryptable this way, proving that researchers are always one step behind. reached v5.2 released a free decryptor Distribution and other tricks Here’s an overview of the mechanisms employed by the GandCrab crew in the course of the culprit’s evolution to spread it on a large scale and diversify the shady monetization portfolio. · An intricate phishing campaign that took root in late January 2019 delivered GandCrab payloads disguised as an . The emails looked like a fine-tuned instruction for the recipients to leave their building in the case of fire or any other disaster requiring evacuation. Such a ticklish subject encouraged most users to open the attached Word document and enable macros to see the contents. Doing so launched an inconspicuous PowerShell script that downloaded the ransomware behind the scenes. emergency exit map update · In February 2019, security analysts an unethical partnership between GandCrab operators and sketchy data recovery companies. These firms claimed to provide seamless file decryption for a fee higher than the actual size of the ransom. Lots of users who were at their wit’s end trying to get their digital lives back opted for such intermediary services, only to pay more than the extortionists wanted. It turned out that the unscrupulous middlemen could access a hidden discount feature on the ransomware Tor payment page, thus earning some dirty extra funds. unveiled · Although largely malevolent, the GandCrab campaign had minor shades of sympathy for the plagued users. In mid-October 2018, the cybercriminals decided to for victims living in Syria. The motivation was a by a Syrian man stating that the ransomware had locked down the photos of his deceased children. The extortionists uploaded a ZIP file containing the keys to an underground hacking forum and wrote they should have excluded Syria from the list of targeted countries in the first place. Security vendors shortly updated their free decrypt tools, thus automating the recovery for users from the Middle Eastern country. release the decryption keys tweet · The very first outbreak of GandCrab that it wasn’t a run-of-the-mill strain. It was the only sample that accepted ransoms in the privacy-centric Dash cryptocurrency alongside Bitcoin. Furthermore, contrary to the majority of ransom Trojans, it was doing the rounds via the Rig and GrandSoft exploit kits that used flaws in unpatched software to drop the deleterious binary. All the elements of this campaign were competently designed from the ground up, suggesting that the adversaries were skilled cybercrooks. demonstrated · The white hats have made a few breakthroughs in fending off the GandCrab epidemic. The earliest of these initiatives in late February last year. Romanian law enforcement orchestrated an operation that resulted in the seizure of some Command & Control servers used by the ransomware distributors. The obtained cryptographic keys enabled Bitdefender to create a decryptor for GandCrab v1. Unfortunately, the malefactors launched a new variant a week after that could no longer be cracked by the tool. hit the headlines · One more anti-ransomware move by South Korean security software provider AhnLab entailed a sort of by GandCrab makers. In July 2018, the analysts came up with a kill switch that prevented the crypto virus from mutilating one’s files. The app made it look as if the computer were already infected with this sample, which forced GandCrab to terminate the raid. vengeance The perpetrators responded to this maneuver by equipping the new ransomware variant with a zero-day exploit targeting AhnLab Lite antivirus suite. In theory, this could expose the AV to denial-of-service issues. In practice, the security tool detected and blocked the malicious code before the bug could cause BSOD or any other damage. · GandCrab v5.0 leveraged a in Windows Task Scheduler’s ALPC (Advanced Local Procedure Call) component in September 2018. By taking advantage of this zero-day vulnerability, the ransomware achieved a privilege escalation effect and could execute arbitrary processes on a contaminated computer. In particular, it deleted Shadow Volume copies of the victim’s data and generated the desktop background warning this way. Although Microsoft patched this exploit the same month, numerous machines remained exposed as users lingered with applying the update. flaw · In early September 2018, the threat actors behind GandCrab dubbed Fallout to their distribution repertoire. Although this malicious framework had made its appearance only a week before, it already gained notoriety for spreading a malware downloader called SmokeLoader and a number of nuisance apps. Fallout used known loopholes in out-of-date versions of Adobe Flash Player and Windows VBScript engine for remote code execution. The exploit kit deposited GandCrab v4 (.KRAB extension) as a second stage payload onto a vulnerable computer. added another exploit kit · The of security researchers in their battle against GandCrab took shape in February 2019. Bitdefender updated their decryption tool that now supports all major versions of the ransomware up to v5.1. This became possible due to the vendor’s collaboration with Europol and law enforcement entities from different countries. The joy was short-lived, though. A couple of days afterward, the felons released GandCrab 5.2 variant that cannot be decrypted. biggest win Conclusion GandCrab is currently the world’s top ransomware menace that’s thriving and evolving despite all efforts of security think tanks. Although the achievements in restoring hostage data are significant and absolutely commendable, the malefactors always have new tricks up their sleeve to outstrip the analysts. Under these circumstances, the proactive defense is the best way to stay safe: · Users should regularly back up their most valuable files. · Keep the operating system and third-party software up to date · Treat dodgy-looking email attachments and links with caution. · Use antivirus and . VPN software <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Dash

Adobe

Alongside

Microsoft

Twitter

Gandcrab — the Most Prevalent Ransomware in 2019

Amazon

Ciklum

Facebook

Slack

Target

Choosing the right platform for chatbot development: UX/UI perspective

What is Spark? Let’s take a look under the hood In we introduced a problem: copious, never ending streams of data, and it’s solution: Apache Spark. Here in Part II we’ll focus on Spark’s internal architecture and data structures, and in Part III we’ll focus more on Spark’s available APIs and Functions in Java. my last post In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers — Grace Hopper With the scale of data growing at a rapid and ominous pace, we needed a way to process potential petabytes of data quickly, and we simply couldn’t make a single computer process that amount of data at a reasonable pace. This problem is solved by creating a cluster of machines to perform the work for you, but how do those machines work together to solve the common problem? Meet Spark Photo by on Jez Timms Unsplash Spark is the cluster computing framework for large-scale data processing. Spark offers a set of libraries in 3 languages (Java, Scala, Python) for its unified computing engine. What does this definition actually mean? Unified: With Spark, there is no need to piece together an application out of multiple APIs or systems. Spark provides you with enough built-in APIs to get the job done Computing Engine: Spark handles loading data from various file systems and runs computations on it, but does not store any data itself permanently. Spark operates entirely in memory — allowing unparalleled performance and speed Libraries: Spark is comprised of a series of libraries built for data science tasks. Spark includes libraries for SQL (SparkSQL), Machine Learning (MLlib), Stream Processing (Spark Streaming and Structured Streaming), and Graph Analytics (GraphX) The Spark Application Every Spark Application consists of a and a set of distributed worker processes ( ). Driver Executors Spark Driver The Driver runs the main() method of our application and is where the SparkContext is created. The Spark Driver has the following duties: Runs on a node in our cluster, or on a client, and schedules the job execution with a cluster manager Responds to user’s program or input Analyzes, schedules, and distributes work across the exectors Stores metadata about the running application and conveniently exposes it in a webUI Spark Executors An executor is a distributed process responsible for the execution of tasks. Each Spark Application has its own set of executors, which stay alive for the life cycle of a single Spark application. Executors perform all data processing of a Spark job Stores results in memory, only persisting to disk when specifically instructed by the driver program Returns results to the driver once they have been completed Each node can have anywhere from 1 executor per node to 1 executor per core Spark’s Application Workflow When you submit a job to Spark for processing, there is a lot that goes on behind the scenes. Our Standalone Application is kicked off, and initializes its SparkContext. Only after having a SparkContext can an app be referred to as a Driver Our Driver program asks the Cluster Manager for resources to launch its executors The Cluster Manager launches the executors Our Driver runs our actual Spark code Executors run tasks and send their results back to the driver SparkContext is stopped and all executors are shut down, returning resources back to the cluster MaxTemperature, Revisited Lets take a deeper look at the Spark Job we wrote in to find max temperature by country. This abstraction hid a lot of set-up code, including the initialization of our SparkContext, lets fill in the gaps: Part I MaxTemperature Spark Setup Remember that Spark is a framework, in this case implemented in Java. It isn’t until line 16 that Spark needs to do any work at all. Sure, we initialized our SparkContext, however loading data into an RDD is the first bit of code that requires work be sent to our executors. By now you may have seen the term “RDD” appear multiple times, it’s about time we define it. Spark Architecture Overview Spark has a well-defined layered architecture with loosely coupled components based on two primary abstractions: Resilient Distributed Datasets (RDDs) Directed Acyclic Graph (DAG) Resilient Distributed Datasets RDDs are essentially the building blocks of Spark: everything is comprised of them. Even Sparks higher-level APIs (DataFrames, Datasets) are composed of RDDs under the hood. What does it mean to be a Resilient Distributed Dataset? Resilient: Since Spark runs on a cluster of machines, data-loss from hardware failure is a very real concern, so RDDs are fault tolerant and can rebuild themselves in the event of failure Distributed: A single RDD is stored on a series of different nodes in the cluster, belonging to no single source (and no single point of failure). This way our cluster can operate on our RDD in parallel Dataset: A collection of values — this one you should probably have known already All data we work with in Spark will be stored inside some form of RDD — it is therefore imperative to fully understand them. Spark offers a slew of “Higher Level” APIs built on top of RDDs designed to abstract away complexity, namely the DataFrame and Dataset. With a strong focus on Read-Evaluate-Print-Loops (REPLs), Spark-Submit and the Spark-Shell in Scala and Python are targeted toward Data Scientists, who often desire repeat analysis on a dataset. The RDD is still imperative to understand, as it’s the underlying structure of all data in Spark. An RDD is colloquially equivalent to: “Distributed Data Structure”. A JavaRDD<String> is essentially just a List<String> dispersed amongst each node in our cluster, with each node getting several different chunks of our List. With Spark, we need to think in a distributed context, always. RDDs work by splitting up their data into a series of partitions to be stored on each executor node. Each node will then perform its work only on its own partitions. This is what makes Spark so powerful: If an executor dies or a task fails Spark can rebuild just the partitions it needs from the original source and re-submit the task for completion. Spark RDD partitioned amongst executors RDD Operations RDDs are Immutable, meaning that once they are created, they cannot be altered in any way, they can only be transformed. The notion of transforming RDDs is at the core of Spark, and Spark Jobs can be thought of as nothing more than any combination of these steps: Loading data into an RDD Transforming an RDD Performing an Action on an RDD In fact, every Spark job I’ve written is comprised of exclusively those types of tasks, with vanilla Java for flavour. Spark defines a set of APIs for working with RDDs that can be broken down into two large groups: and . Transformations Actions Transformations create a new RDD from an existing one. Actions return a value, or values, to the Driver program after running a computation on its RDD. For example, the function weatherData.map() is a transformation that passes each element of an RDD through a function. map is an RDD action that aggregates all the elements of an RDD using some function and returns the final result to the driver program. Reduce Lazy Evaluation “I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it. — Bill Gates” All transformations in Spark are lazy. This means that when we tell Spark to create an RDD via transformations of an existing RDD, it won’t generate that dataset until a specific action is performed on it or one of it's children. Spark will then perform the transformation and the action that triggered it. This allows Spark to run much more efficiently. Let’s re-examine the function declarations from our earlier Spark example to identify which functions are actions and which are transformations: 16: JavaRDD<String> weatherData = sc.textFile(inputPath); Line 16 is neither an action or a transformation; it’s a function of , our . sc JavaSparkContext 17: JavaPairRDD<String, Integer> tempsByCountry = weatherData.mapToPair(new Func..... Line 17 is a of the RDD, in it we map each line of to a pair comprised of (City, Temperature) transformation weatherData weatherData 26: JavaPairRDD<String, Integer> maxTempByCountry = tempsByCountry.reduce(new Func.... Line 26 is also a because we are iterating over key-value pairs. its a transformation of in which we reduce each city to its highest recorded temperature. transformation tempsByCountry 31: maxTempByCountry.saveAsHadoopFile(destPath, String.class, Integer.class, TextOutputFormat.class); Finally on line 31 we trigger a Spark : saving our RDD to our file system. Since Spark subscribes to the lazy execution model, it isn’t until this line that Spark generates , , and before finally saving our result. action weatherData tempsByCountry maxTempsByCountry Directed Acyclic Graph Whenever an action is performed on an RDD, Spark creates a DAG, a finite direct graph with no directed cycles (otherwise our job would run forever). Remember that a graph is nothing more than a series of connected vertices and edges, and this graph is no different. Each vertex in the DAG is a Spark function, some operation performed on an RDD (map, mapToPair, reduceByKey, etc). In MapReduce, the DAG consists of two vertices: Map → Reduce. In our above example of MaxTemperatureByCountry, the DAG is a little more involved: parallelize → map → mapToPair → reduce → saveAsHadoopFile The DAG allows Spark to optimize its execution plan and minimize shuffling. We’ll discuss the DAG in greater depth in later posts, as it’s outside the scope of this Spark overview. Evaluation Loops With our new vocabulary, let us re-examine the problem with MapReduce as I defined in , quoted below: Part I MapReduce excels at batch data processing, however it lags behind when it comes to repeat analysis and small feedback loops. The only way to reuse data between computations is to write it to an external storage system (a la HDFS)” ‘Re-use data between computations’? Sounds like an RDD that can have multiple actions performed on it! Lets suppose we have a file “data.txt” and want to accomplish two computations: Total length of all lines in the file Length of the longest line in the file In MapReduce, each task would require a separate job or a fancy MulitpleOutputFormat implementation. Spark makes this a breeze in just four simple steps: Load contents of into an RDD data.txt JavaRDD<String> lines = sc.textFile("data.txt"); 2. Map each line of ‘ ’ to its length lines (Lambda functions used for brevity) JavaRDD<Integer> lineLengths = lines.map(s -> s.length()); 3. To solve for total length: reduce to find the total line length sum, in this case the sum of every element in the RDD lineLengths int totalLength = lineLengths.reduce((a, b) -> a + b); 4. To solve for longest length: reduce to find the maximum line length lineLengths int maxLength = lineLengths.reduce((a, b) -> Math.max(a,b)); Note that steps 3 and 4 are RDD actions, so they return a result to our Driver program, in this case a Java int. Also recall that Spark is lazy and refuses to do any work until it sees an action, in this case it will not begin any real work until step 3. Next Steps So far we’ve introduced our data problem and its solution: Apache Spark. We reviewed Spark’s architecture and workflow, it’s flagship internal abstraction (RDD), and its execution model. Next we’ll look into Functions and Syntax in Java, getting progressively more technical as we dive deeper into the framework.

Apache

Initialized

High Level Overview of Apache Spark

Technology products have changed the way we live, eat, interact, and commute. We shop through Amazon instead of going to physical stores. We order Uber or Lyft ride before leaving our home instead of trying our luck to get a taxi on the street. We use GrubHub to order delivery online. We make new friends through Tinder. We use Airbnb to find a place to stay when traveling. We search on Google when we have questions to ask. What changes our lives the most is the way we . People often open their social apps like Facebook, Twitter, or WeChat first thing in the morning before getting out of beds. We often see people on their phones during social events. It is amazing to learn that more than 2.3 billions of people are using Facebook on a daily basis in 2019. Yet, Facebook former vice president of user growth, Chamath Palihapitiya, said that he does not use Facebook nor his children are allowed. Facebook’s LIKE function have been manipulating and controlling people’s psychological weaknesses to seek for quick rewards. interact with one another The short-term dopamine-driven feedback loops that we created are destroying how society works. — Chamath Palihapitiya These technology products can be realized thanks to and many business models exist today relies on . Amazon trains its AI algorithms to make personal recommendations based on purchase and browse history to increase sales. Uber uses AI models to estimate arrival time and determine ride fare base on the current condition of the traffic. Google tunes its AI algorithms to determine the page rankings and earn money through its advertising systems. Without AI models, none of these would even exist. However, these technology giants exist today and they are tweaking the algorithms in favor of them. big data AI algorithms What Tech Giants can do to Their AI Algorithms are way beyond Your Imagination Human relies on others for our own decisions. We often read product reviews or watch video reviews before making a purchase. A psychologist at American Institute for Behavioral Research and Technology in California named Robert Epstein conducted an on 661 Americans who are not familiar with Australian politics to vote for the two candidates in the Australian election based on search engine results. The study found that the ratio of positive to negative suggestions in the autocomplete can change the preference of undecided voters by nearly 80%. experiment The above scenario happened in an experiment, but imagine if it happens in real life, and . In the 2016 US presidential election, Google was of altering the candidate’s autocomplete suggestion for its search engine in favor of Hillary Clinton. When you typed “Hillary Clinton is” in the Google search engine, the phrase “Hillary Clinton is winning” would appear in the autocomplete suggestion. However, in Yahoo and Bing search engines, the results of the autocomplete suggestions were “Hillary Clinton is a liar” and “Hillary Clinton is a criminal”. It is to believe that the tech giant was tinkering with its search suggestion algorithm to cast Clinton in a positive right. it actually did accused Public figures as controversial as Trump also suffered from Google’s autocomplete algorithm. Trump that Google where “96 percent of Google search results for the word “Trump” were articles from left-leaning sites.” believed buried conservative news in search results The Power of Manipulating Human Attention and Sentiment Our moods are also affected by the mood of others just like how we depend on others to make decisions. Have you ever experience this? When you open your social Apps like Facebook or Twitter and read a post from one of your friends, it either brightens up or ruins your day. Facebook’s data scientists believe human sentiments are on social networks. The researchers 689,003 Facebook users and then split them into two groups with one group removing one of the users’ negative emotion posts and the other removing positive emotion posts. The study showed that when the user sees less emotionally active post from friends, the number of positive posts by the user decreases and the number of negative posts increases; vice versa. contagious selected It is unbelievable to imagine that with a minor tweak of an algorithm can make such a huge difference to our sentiments. Yet, what is more striking is that the experiment conducted in the paper tampered the News Feed meaning but you just did not know. without user’s knowledge or consent you may be one of the guinea pigs Whether We should trust Centralized Entities with the Creation of AIs that will shape Our Future Big companies remain threats to . Facebook is only showing users the posts they agree with while Google is altering search results which increase polarization and harm democracy. A fundamental change to business models can reduce the risk of centralized AIs. AI democratization The democratization of Artificial Intelligence is crucial to the future development of mankind. The enormous potential of AI means that it can either increase the inequalities of society or free us from suffering. The current AI research tends to lead to concentrations of power and money, which means that the future development of AI will only be more centralized and may even harm humanity. In order to avoid such an event to happen, AI needs to be much easier to use, more reliable, and more intuitive than it is today. Blockchain may be the answer. Blockchain’s decentralized nature supports transactions directly between disparate parties without third parties (i.e. Facebook and Google). It is ultimately a means for individuals to interact directly and to govern themselves in a more secure and decentralized manner. Blockchain technology provides a unique token economy, also known as tokenomics model. A token economy is a form of behavior modification designed to increase desirable behavior and decrease undesirable behavior with the use of tokens. Individuals receive tokens immediately after displaying desirable behavior. The tokens are collected and later exchanged for a meaningful object or privilege. Infographic by blockchainhub Each blockchain platform has its own unique tokenomics model. was designed so that a steady stream of tokens can enter the network through block rewards. After a block has been successfully validated by a miner, tokens are distributed as block reward. The number of tokens rewarded for each validated block halves overtime. Bitcoin has a maximum supply of 21,000,000 BTC. tokens, similar to Bitcoin, are continuously distributed as block rewards but with no max supply. Bitcoin Ethereum There are several projects working on democratizing AI to foster the world of open source AIs. aims to create a democratically governed network of AI agents where users can access AI algorithms through SDK and API using SingularityNet token, AGI. is leveraging the blockchain technology to achieve Artificial Intelligence democratization where everyone can easily gain access to the AI model and use in smart contracts and DApps. Cortex uses an incentive mechanism to reward AI model contributors where the contributor gets paid when the model is called by a smart contract (similar to Amazon API Gateway where you get paid per the amount of calls). With tokenomics model design, we can communicate with one another without a central entity but also gets the goodies and benefits of AI that bring to our lives. SingularityNet Cortex Conclusions Like many technologies, AI algorithm is a double-edged sword. The question is, Let us not forget that Facebook allowed its data scientists to manipulate user experiences and collect data without users’ explicit consents. Blockchain technology provides a mean for users to have sayings on what is right and what is wrong. The decentralized nature of blockchain provides a powerful solution to democratize AI algorithms and actually serve the user’s interest. This is just the beginning of an era, it still requires an enormous amount of hard work to realize AI democracy and only time will tell where blockchain will lead us to the future development of AI. is it really enough for technology giants to have the right to make rules by relying on the suspicion of not doing evil?

AirBnB

Lyft

Uber

2022 - HackerNoon Contributor of the Year - Innovation

Nominated for 2022 - HackerNoon Contributor of the Year - Innovation

How blockchain could prevent companies like Facebook from manipulating users

Python is everywhere. According to the 2019’s , it is the 2nd most loved programming language in the world. StackOverflow’s Developer Survey Over the past 4 years, Python has constantly been the among GitHub contributors. 3rd most popular language Even here at , we use Python as a core language for our resume-processing engine. CV Compiler Our love for Python made us go even further — with the use of and , we have created own text analytics tool. (Without it, you won’t be reading this data-proven list of skills a modern Pythonista should have.) NLTK spaCy Actually, here is the result of our efforts — the comprehensive set of skills you as a Python dev need to stay marketable in 2019. What skills do I need to succeed as a Python dev in 2019? Jointly with our team, we took 300 job specs for Python developers, scrapped from StackOverflow, AngelList, LinkedIn, and some fast-growing tech companies worldwide. From all these descriptions, we extracted the skills which were mentioned the most frequently, and here they are. (The numbers refer to the number of mentions.) Among the other frequently-mentioned skills not highlighted in this chart were (32), (30), (30), and (30). If talking about the Machine Learning skills, the most commonly-mentioned ones were (29) (24) (15) and (11). Unit Testing Continuous Deployment MongoDB OOP Pandas , NumPy , SciPy , Scikit-Learn Surely, the necessity of these skills depends on how you use Python. If you’re a Machine Learning Engineer, you don’t need to know Django perfectly, and so on. This research shows the overall tendencies in the employment market, not the preferences of Python developers themselves. This rating may differ from the list of technologies most devs prefer to use in their work. For some Python insights, turn to by Python Software Foundation, or by GitHub. Please note: Python Developers Survey Octoverse What is the experts’ opinion about current Python trends? Nowadays, it’s absolutely standard for developers to be familiar with Git, GitHub, and Travis for CI. Testing, preferably with pytest and (judicious) use of unittest.mock, is also vital for a well-run project. Container and virtualization technologies, from docker to lxc to AWS to Azure to Kubernetes and OpenStack, are very “normal” for both development and deployment. Being able to create build artifacts, then deploy a staging version of your infrastructure, then run tests against it, then deploy to production and if necessary rollback to a previous version, should all be push-button operations. Django is still as popular as ever and a useful skill to have. Flask and SQLAlchemy is just as potent a combination, but not quite as highly in demand. As for Data Science, it is a rising star of the Python world. Pandas, Numpy and SciPy are all tools that are highly in demand, along with Jupyter notebooks. Database knowledge is also essential here. _Creator of “mock” (now a unittest.mock standard library)_ Michael Foord, Python core developer. Twitter | Personal website This research shows pretty clearly the direction where the development world is evolving. AWS, Docker, Clouds, and Kubernetes are pretty new, but they have already become a part of the programmers’ everyday routine. It doesn’t mean that you as a Python dev need to know all the nuances of these technologies, but you will be facing them on a daily basis. Machine Learning requires Python in 9 cases out of 10, but it’s a completely different field. If you’re a Python dev, you don’t need ML skills. If you choose Machine Learning or AI, you will definitely need Python. The word ‘microservices’ is very close to Go, REST, and API on the graph. However, this tiny word silently changes the whole market. I recommend starting to invest time in understanding what it brings to the surface. Mikhail Kashkin,Systems Architect and Python Expert. Ex-Googler. YouTube | GitHub For Python devs in general, I don’t think there is one thing they need to learn/know to excel. Data science, DevOps, Web Development, and Automation are totally different fields but all fall under “Python devs”. However, I get asked about Data Science tools more than I used to from my clients. Mostly it’s Pandas, followed by Jupyter notebooks. _former director of the Python Software Foundation._ Trey Hunner,Python and Django team trainer, Twitter | Website I have done a few interviews over the past year or two and there are definitely a lot of employers that want AWS or cloud experience. In addition to that, I see a lot of growth in Data Science / Machine Learning. And much like this research is already showing, if you can do Web Development, that would be good as well. . Mike Driscoll, author of Python books, blogger. Part of the tutorial team at Real Python Twitter | Blog To polish my skills, what resources should I use? To help you out, I’ve gathered some resources respected by both our team and experienced devs from Python communities worldwide. Python-related resources (web development included): is a comprehensive set of Python tutorials. (They also have an inspiring , created by .) RealPython YouTube channel Dan Bader — a curated weekly newsletter featuring articles, releases, and jobs. Python Weekly is full of Python resources for experienced devs, engineers, and scientists. (Created by .) This website Kenneth Reitz has huge playlists about Django, ML, Python&Finance, and so on. (Put together by .) Sentdex YouTube channel Harrison Kinsley , offers about 80 articles, dedicated to Flask. In his blog Miguel Grinberg Data Science, Machine Learning & AI: on implementing Data Science & AI. The topics covered also include AWS and Linux. (Made by .) Technical notes Chris Albon While Keras is one of the most popular neural network libraries for Python, it also has a . useful thematical blog for those who want to master scientific Python. A set of Scikit-learn tutorials Computer Vision and Deep Learning , by . resource guide Adrian Rosebrock Who should I follow online to keep a finger on the pulse? Here is a list of Pythonistas I gladly follow myself on Twitter: — creator of Pipenv & Requests. @kennethreitz — author of Python Cookbook. @dabeaz — Python core developer. @raymondh — Python core developer, founder of EdgeDB. @1st1 — co-creator of Django. @adrianholovaty — co-creator of Django. @jacobian — co-author of ‘Two Scoops of Django’ book, open source programmer. @pydanny — Cloud Developer Advocate at Microsoft. @ixek Practice makes perfect, so, create a list of skills you want to learn or improve, and start your way to the new heights. I hope this material will help you in boosting your IT career! This article was prepared jointly with the team at , a Machine Learning-powered resume enhancement tool for tech professionals, (Python developers in particular). If you need a flawless resume, . You can check your resume for free! CV Compiler tap here

The Graph

Routine

YouTube

Get your IT resume improved

What do companies expect from Python devs in 2019?

Blockchain Smart Cities Rating — From Melaka Straits City to Crypto Project in Nevada Desert The agiotage around blockchain technology has reduced dramatically. But now it’s time for technology adoption in real life. Imagine a city, where payments for goods and services, data exchange between municipal institutions, work of transport and infrastructure projects — almost everything in based on blockchain technology. You think such kind of mass adoption looks like a fairy tale? But this is actually a reality or something that will be a reality in the very near future. I decided to create a short list of the most promising smart city projects based on blockchain, which are already running or are about to be launched. The list includes those projects that are not just developing a technological solution that will be applicable in a particular area of ​​work of the smart city, but projects that are already developing or will be developed in concrete countries and cities. Melaka Straits City — Melaka, Malaysia Melaka Straits City Project Melaka is one of the popular tourists’ destinations, that are visiting Malaysia. Here 835 acres of undeveloped Malacca will turn into a real blockchain city called Melaka Straits City. The project was launched by SWT International and China Wu Yi and is supported by the Chinese government. The architecture of the future city was designed by CPG Corporation, the company known for the development of the Parliament House in Singapore and Changi Airport. The project is planned to become the first blockchain-city in Asia. The , that blockchain technology innovation will be used to improve fundamental infrastructure condition, integrate with city statistical data, highly safe and secured management of a model city to stimulate traditional tourism development. project officials state The basis of the blockchain technology in the Melaka Straits City will be DMI platform with DMI coin. It will mainly be used in payment of government-based services in the city. At the very beginning, it is planned to invest almost $120 mln. One of the project’s partner is China Wu Yi, subsidiary of Fujian Construction Engineering, one of the largest construction holding in the world. Blockchains — Nevada, U.S. Blockchain Smart City project in Nevada Blockchain LLC is the company owned by famous cryptocurrency millionaire Jeffrey Berns that have bought 67,000 acres of land in the Nevada desert. The aim is to build here absolutely new smart city. According to the project officials, there has to be 1,000-acre tech campus, the world’s first esports stadium, a nuclear-hardened bunker to safeguard digital assets, and fully functioning smart city, powered by blockchain technology. It is said that Berns, the author of the project, has already spent between $250 million and $300 million. Still, after the , the project was criticized by several media outlets and crypto influencers, because there was no proof that investment was made, plus the project is too ambitious. announcement Dongdaemun Fashion Town — Seoul, South Korea Dongdaemun Fashion Town Last year blockchain startup Waltonchain and Dongdaemun Fashion Town Tourism Zone a technical agreement regarding the application of the blockchain technology in the fields of retail distribution, logistics, store management, and customer service. reached Dongdaemun gathers enterprises in industries including fabrics, garment accessories, production and processing, fashion design, retail distribution. Products range from small fashion accessories to ginseng, utensils, clothing, toys, jewelry, etc. There are more than 35,000 stores in the fashion town. Authors of the project believe, that blockchain technology will greatly optimize the logistics and retail system of the whole town. Smart Dubai — Dubai, UAE Dubai, UAE This would be incorrect not to mention Dubai in the short-list. If in the previous cases we were talking about projects, launched by a business or rich people, who wants to create blockchain city, in this case, the initiator is the government. Dubai officials have launched , the program is a public-private partnership to create a smart city with a seamless experience for its citizens and an efficient city government powered by blockchain technology by 2020. Smart Dubai To do so, they have launched the Global Blockchain Challenge, which is a blockchain-related competition geared to finding the most innovative blockchain startups from across the world and to entice them to operate in Dubai. The problem is that, apart from loud statements from the authorities, we still don’t see cases of using blockchain technologies at least in some areas of city management. You must admit that this is strange, taking into the account that already in the next year Dubai fully wants to switch to the blockchain. City Coin — Liberstad, Norway Liberstad Smart City It is difficult to imagine this, but already now, in April 2019, there is a place in the world, a whole city that has completely switched to its own cryptocurrency. The name of this city is Liberstad, Norway. For sure, you’ve may heart that some town in the world started to adopt some cryptocurrency. But Liberstad to be the first smart city to officially adopt a cryptocurrency as its sole medium of exchange for the payment of city services and workers’ wages. claims The town official currency is named City Coin. This crypto is based on City Chain, a smart city platform that enables the design, implementation, and use of next-generation services for smart cities and their inhabitants. The issuer of City Coin, City Chain Labs, sais that the platform gives existing and emerging cities like Liberstad the opportunity to build and offer services conducted on a private, internal and voluntary basis. City Coin is available for trading on exchange and Liberstad’s Block Exchange and more than 100 land plots have already been sold within the city. <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Chain

Blockchain Smart Cities Rating — From Melaka Straits City to Crypto Project in Nevada Desert

There’s something I’ve been saying since was : employee surveys are dumb. They’re really, frustratingly dumb. They’re not the solution to the . They’re part of the problem. SoapBox first born employee engagement crisis So stop filing them out — and if anyone gives you grief, send them this article. Here are five reasons why employee engagement surveys need to go extinct. 1. On average, people who share harsh feedback quit before management acts on it ⏳ You’ve sent out an employee engagement survey before, right? Then you know the drill. HR sends a survey to employees, who send their answers back to HR, who sends the responses to managers, who take action on the feedback… 21 weeks later (or longer). By then, most of the feedback will be completely out-of-date — and any of the employees who had major issues with the way you’re running the company are already long gone — after all, it takes most employees 17 weeks to quit. Feedback needs to be immediate and ongoing — without that urgency, employees can’t see how their feedback leads to action, and so they’ll have no motivation to give the feedback in the first place. 2. HR is responsible for the feedback, but the problems are rarely HR issues ↬ Even if you’re doing weekly pulse checks, you’re still running valuable feedback through a needless cycle of HR and senior leadership before it gets back to the person that should actually care about it: the manager. Not only are you running a serious risk of broken telephone. It’s also super inefficient and a waste of time for a number of people. Yes, it’s sometimes more awkward for the employee to deliver feedback directly to their manager, but that awkwardness is part of the vulnerability that makes the relationship strong. Imagine you’re walking into a meeting with . Can you imagine how insane it would be if your employee stood up, walked out of the room and rushed to HR to let them know that you had food stuck between your teeth? Is that seriously less awkward?! spinach in your teeth But most importantly, it sends a horrible cultural message: that feedback SHOULD come through HR — and managers aren’t responsible for pulling feedback from their team. Really, surveys are a cover for bad managers who don’t care to take the time to talk to their team. Instead of doing surveys, you should be coaching your bad managers on asking the right questions and creating the right environment. Or for that matter, spending the time fixing the fact that you’ve hired bad managers in the first place (after all, companies fail to choose the right manager for the job ). 82% of the time 3. They train people to care about the averages, not the individual 🤷‍♂️ Engagement surveys lack context. In other words, it’s really easy to ask a great question — at the wrong time. Let’s say you did a company-wide kickoff where you went over the strategy and goals for the year. And you have a pulse question about whether employees feel connected to the company’s vision. If you send out that pulse question a day or two after the kickoff, you’re going to get a very different answer than if you asked in seven months. Whether you’re doing regular pulsing or annual surveys, it’s really easy for employees to only get the really meaningful questions once a year — and so the insights you’re collecting aren’t really going to improve their experience or performance. And even if you do ask the right question at the right time, you still risk a lack of context. The question “How accountable are your co-workers? 1–10” can mean so many things to so many people. It’s relative to personality types, departments…let alone each individual person’s . approach to ranking Or, the fact that an average score of, say, 7.21 contains “ .” far greater a distortion than if it simply surveyed one person Plus, . So the aggregate data is skewed anyway… 25% of people make up random answers On the other hand, if you’re asking a similar question to a direct report in a , you can get the additional context and can ask follow up questions to make it a much more meaningful discussion. one-on-one meeting 4. They’re annoying, 20-minute distractions from real work 🙄 There’s a reason why HR always worries that surveys are annoying the team. It’s because surveys ARE annoying the team. No one likes surveys. And if you add in the time it takes to switch tasks, even the simplest survey takes an employee away from their work for upwards of 20 minutes. It’s basic . maker vs. manager stuff . And the other 13% who welcome a distraction are the actively disengaged people. So… you’re annoying the people you want more of, and you’re providing an outlet for people you want less of. 87% of people don’t want the distraction Don’t get me wrong: I think it’s super important to know what will make employees happy and satisfied at work. BUT, survey or no survey, . Doesn’t it make more sense to double down on that 70% than to waste everyone’s time addressing that other 30% with another survey that will hinder more than help. 70% of the variance is proven to come directly from their manager Everyone hates filling out surveys. Everyone. Even HR. But we love sending them because they are a cheap way to have someone else do your work. It’s their work. Let them do it. 5. We’ve been doing this for 40 years and the numbers have NEVER CHANGED 📊 I can tell you right now: if you do an employee engagement survey, you’re going to learn that 30% of your employees are engaged, 18% are disengaged. Or you’ll have an eNPS score of like 15. If it’s not that, it’s because: Your company has less than 10 employees, and engagement is staggeringly high because the CEO is your manager ( ). then you’d have 41% engaged employees Everyone at the company has started in the past six months ( ). then you’d have 52% Your company is that rare unicorn where every manager is amazing (if you work at this company, never leave your job — and tell your boss you appreciate them). Your HR or management team has gamed the system by sending out the survey immediately after announcing that you’ll be offering free lunches three times a week. But really, at a certain size you’ll have 30% engaged people. And this number hasn’t changed in 40 years. Why? It’s not because people are born engaged or disengaged. They can switch teams in the same company and go from engaged to disengaged. ? Because 70% of the variance is managers… and we’re still trying to FIX the problem by sending our surveys. OK, so then why Let’s face it, it doesn’t work. And going from once-a-year surveys to weekly pulses won’t fix it. Why haven’t we fixed the problem? Because it’s not HR’s job to keep employees engaged. It’s bad culture, bad awareness, and lazy thinking to put that task onto HR. It’s the manager’s job to keep employees engaged. It’s the CEO’s job. It’s the CTO’s job. Everyone that manages someone else has a responsibility to help that person thrive and grow on an ongoing basis — and that’s not something you can delegate out to HR, or anyone for that matter. Stop with the employee engagement surveys 🙏. Take that budget, that energy, that time and put it toward empowering your managers to be amazing. Because when managers are amazing, your employees will be, too. Thanks for reading! Brennan p.s. If your startup has hired a few employees (or you’re a manager), you should really check out our product now. SoapBox Brennan is the CEO & Co-founder of SoapBox — the manager’s sidekick app, bot, and platform. Build better agendas, have better one-on-ones, team meetings, AMAs, town-halls and more, with your team.

Maker

Soapbox

Startups: Pulse Surveys are Dumb.

The Game Changer Deep Learning has become the main driver of many new applications and it’s time to really look at why this is the case. With so many other options that we’ve been using for so long, why Deep Learning? Deep Learning is popular right now because it’s easy and it works. OK you maybe thinking deep learning is easy and works, but what is it? I’m glad you asked let’s jump right to it. 😄 Deep learning is a subset of machine learning where artificial neural networks, algorithms inspired by the human brain, learn from large amounts of data. Similarly to how we learn from experience, the deep learning algorithm would perform a task repeatedly, each time tweaking it a little to improve the outcome. We refer to ‘deep learning’ because the neural networks have various (deep) layers that enable learning. Just about any problem that requires “thought” to figure out is a problem deep learning can learn to solve. Deep learning differs from traditional machine learning techniques in that they can automatically learn representations from data such as images, video or text, without introducing hand-coded rules or human domain knowledge. Their highly flexible architectures can learn directly from raw data and can increase their predictive accuracy when provided with more data. For example in facial recognition, how pixels in an image create lines and shapes, how those lines and shapes create facial features and how these facial features are arranged into a face. Deep learning changes how you think about representing the problem that you’re solving with analytics. It moves from telling the computer how to solve a problem to training the computer to solve the problem itself For example, a deep learning model known as a convolutional neural network can be trained using large numbers (as in millions) of images, such as those containing cats. This type of neural network typically learns from the pixels contained in the images it acquires. It can classify groups of pixels that are representative of a cat’s features, with groups of features such as claws, ears, and eyes indicating the presence of a cat in an image. Deep learning is fundamentally different from conventional machine learning. In this example, a domain expert would need to spend considerable time engineering a conventional machine learning system to detect the features that represent a cat. With deep learning, all that is needed is to supply the system with a very large number of cat images, and the system can autonomously learn the features that represent a cat. Credit For The Image Goes To: TechTarget The amount of data we generate every day is staggering — currently estimated at 2.6 quintillion bytes and it’s the resource that makes deep learning possible. Since deep-learning algorithms require a ton of data to learn from, this increase in data creation is one reason that deep learning capabilities have grown in recent years. In addition to more data creation, deep learning algorithms benefit from the stronger computing power that’s available today. Deep learning has been also instrumental in the discovery of exoplanets and novel drugs and the detection of diseases and subatomic particles. It is fundamentally augmenting our understanding of biology, including genomics, proteomics, metabolomics, the immunome, and more. What kind of problems does deep learning solve? As mentioned, deep neural networks excel at making predictions based on largely unstructured data. That means they deliver best in class performance in areas such as speech and image recognition, where they work with messy data such as recorded speech and photographs. Should you use always deep learning instead of machine learning? No, because deep learning can be very expensive from a computational point of view. For non-trivial tasks, training a deep-neural network will often require processing large amounts of data using clusters of high-end GPUs for many, many hours. Given top-of-the-range GPUs can cost thousands of dollars to buy, or up to $5 per hour to rent in the cloud, it’s unwise to jump straight to deep learning. If the problem can be solved using a simpler machine-learning algorithm such as Bayesian inference or linear regression, one that doesn’t require the system to grapple with a complex combination of hierarchical features in the data, then these far less computational demanding options will be the better choice. Deep learning may also not be the best choice for making a prediction based on data. For example, if the dataset is small then sometimes simple linear machine-learning models may yield more accurate results — although some machine-learning specialists argue a properly trained deep-learning neural network can still perform well with small amounts of data. What are the drawbacks of deep learning? One of the big drawbacks is the amount of data they require to train, with Facebook recently announcing it had used one billion images to achieve record-breaking performance by an image-recognition system. When the datasets are this large, training systems also require access to vast amounts of distributed computing power. This is another issue of deep learning, the cost of training. Due to the size of datasets and number of training cycles that have to be run, training often requires access to high-powered and expensive computer hardware, typically high-end GPUs or GPU arrays. Whether you’re building your own system or renting hardware from a cloud platform, neither option is likely to be cheap. Deep-neural networks are also difficult to train, due to what is called the vanishing gradient problem, which can worsen the more layers there are in a neural network. As more layers are added the vanishing gradient problem can result in it taking an unfeasibly long time to train a neural network to a good level of accuracy. Why are deep neural networks hard to train? As mentioned deep neural networks are hard to train because of the number of layers in the neural network. The number of layers and links between neurons in the network is such that it can become difficult to calculate the adjustments that need to be made at each step in the training process — a problem referred to as the vanishing gradient problem. Credit For The Image Goes To: makeameme.org Another big issue is the vast quantities of data that are necessary to train deep learning neural networks, with training corpuses often measuring petabytes in size. How much does it cost to invest in deep learning? It depends on your approach, but it will typically cost you hundreds of dollars upwards, depending on the complexity of the machine-learning task and your chosen method. What deep learning techniques exist? There are various types of deep neural network, with structures suited to different types of tasks. For example, Convolutional Neural Networks (CNNs) are typically used for computer vision tasks, while Recurrent Neural Networks (RNNs) are commonly used for processing language. Each has its own specializations, in CNNs the initial layers are specialized for extracting distinct features from the image, which are then fed into a more conventional neural network to allow the image to be classified. Meanwhile, RNNs differ from a traditional feed-forward neural network in that they don’t just feed data from one neural layer to the next but also have built-in feedback loops, where data output from one layer is passed back to the layer preceding it — lending the network a form of memory. There is a more specialized form of RNN that includes what is called a memory cell and that is tailored to processing data with lags between inputs. The most basic type of neural network is a multi-layer perceptron network, the type discussed above in the handwritten figures example, where data is fed forward between layers of neurons. Each neuron will typically transform the values they are fed using an activation function, which changes those values into a form that, at the end of the training cycle, will allow the network to calculate how far off it is from making an accurate prediction. There are a large number of different types of deep neural networks. No one network is inherently better than the other, they just are better suited to learning particular types of tasks. More recently, generative adversarial networks (GANS) are extending what is possible using neural networks. In this architecture two neural networks do battle, the generator network tries to create convincing “fake” data and the discriminator attempts to tell the difference between fake and real data. With each training cycle, the generator gets better at producing fake data and the discriminator gains a sharper eye for spotting those fakes. By pitting the two networks against each other during training, both can achieve better performance. GANs have been used to carry out some remarkable tasks, such as turning these dashcam videos from day to night or from winter to summer, as shown in the video below, and have applications ranging from turning low-resolution photos into high-resolution alternatives and generating images from written text. GANs have their own limitations, however, that can make them challenging to work with, although these are being tackled by developing more robust GAN variants. Which deep learning software frameworks are available? There are a wide range of deep learning software frameworks, which allow users to design, train and validate deep neural networks, using a range of different programming languages. A popular choice is Google’s TensorFlow software library, which allows users to write in Python, Java, C++, and Swift, and that can be used for a wide range of deep learning tasks such as image and speech recognition, and which executes on a wide range of CPUs, GPUs, and other processors. It is well-documented, and has many tutorials and implemented models that are available. Another popular choice, especially for beginners, is PyTorch, a framework that offers the imperative programming model familiar to developers and allows developers to use standard Python statements. It works with deep neural networks ranging from CNNs to RNNs and runs efficiently on GPUs. Among the wide range of other options are Microsoft’s Cognitive Toolkit, MATLAB, MXNet, Chainer, and Keras. Deep Learning Use Case Examples Robotics Many of the recent developments in robotics have been driven by advances in AI and deep learning. For example, AI enables robots to sense and respond to their environment. This capability increases the range of functions they can perform, from navigating their way around warehouse floors to sorting and handling objects that are uneven, fragile, or jumbled together. Something as simple as picking up a strawberry is an easy task for humans, but it has been remarkably difficult for robots to perform. As AI progresses, that progress will enhance the capabilities of robots. Developments in AI mean we can expect the robots of the future to increasingly be used as human assistants. They will not only be used to understand and answer questions, as some are used today. They will also be able to act on voice commands and gestures, even anticipate a worker’s next move. Today, collaborative robots already work alongside humans, with humans and robots each performing separate tasks that are best suited to their strengths. Agriculture AI has the potential to revolutionize farming. Today, deep learning enables farmers to deploy equipment that can see and differentiate between crop plants and weeds. This capability allows weeding machines to selectively spray herbicides on weeds and leave other plants untouched. Farming machines that use deep learning–enabled computer vision can even optimize individual plants in a field by selectively spraying herbicides, fertilizers, fungicides, insecticides, and biologicals. In addition to reducing herbicide use and improving farm output, deep learning can be further extended to other farming operations such as applying fertilizer, performing irrigation, and harvesting. Medical Imaging and Healthcare Deep learning has been particularly effective in medical imaging, due to the availability of high-quality data and the ability of convolutional neural networks to classify images. For example, deep learning can be as effective as a dermatologist in classifying skin cancers, if not more so. Several vendors have already received FDA approval for deep learning algorithms for diagnostic purposes, including image analysis for oncology and retina diseases. Deep learning is also making significant inroads into improving healthcare quality by predicting medical events from Virtual assistants Whether it’s Alexa or Siri or Cortana, the virtual assistants of online service providers use deep learning to help understand your speech and the language humans use when they interact with them. Translations In a similar way, deep learning algorithms can automatically translate between languages. This can be powerful for travelers, business people and those in government. By the way check out this funny fail by Google translate (For all you triggered Bieber fans out there right now it’s not me it’s Google’s fault OK?! :] ) PVision for driverless delivery trucks, drones and autonomous cars The way an autonomous vehicle understands the realities of the road and how to respond to them whether it’s a stop sign, a ball in the street or another vehicle is through deep learning algorithms. The more data the algorithms receive, the better they are able to act human-like in their information processing — knowing a stop sign covered with snow is still a stop sign. Chatbots and service bots Chatbots and service bots that provide customer service for a lot of companies are able to respond in an intelligent and helpful way to an increasing amount of auditory and text questions thanks to deep learning. Image colorization Transforming black-and-white images into color was formerly a task done meticulously by human hand. Today, deep learning algorithms are able to use the context and objects in the images to color them to basically recreate the black-and-white image in color. The results are impressive and accurate. Facial recognition Credit For The Image Goes To: Yann LeCun Deep learning is being used for facial recognition not only for security purposes but for tagging people on Facebook posts and we might be able to pay for items in a store just by using our faces in the near future. The challenges for deep-learning algorithms for facial recognition is knowing it’s the same person even when they have changed hairstyles, grown or shaved off a beard or if the image taken is poor due to bad lighting or an obstruction. The Future of Deep Learning Today, there are various neural network architectures optimized for certain types of inputs and tasks. Convolution neural networks are very good at classifying images. Another form of deep learning architecture uses recurrent neural networks to process sequential data. Both convolution and recurrent neural network models perform what is known as supervised learning, which means they need to be supplied with large amounts of data to learn. In the future, more sophisticated types of AI will use unsupervised learning. A significant amount of research is being devoted to unsupervised and semisupervised learning technology. Reinforcement learning is a slightly different paradigm to deep learning in which an agent learns by trial and error in a simulated environment solely from rewards and punishments. Deep learning extensions into this domain are referred to as deep reinforcement learning (DRL). There has been considerable progress in this field, as demonstrated by DRL programs beating humans in the ancient game of GO. Designing neural network architectures to solve problems is incredibly hard, made even more complex with many hyperparameters to tune and many loss functions to choose from to optimize. There has been a lot of research activity to learn good neural network architectures autonomously. Learning to learn, also known as metalearning or AutoML, is making steady progress. Current artificial neural networks were based on 1950s understanding of how human brains process information. Neuroscience has made considerable progress since then, and deep learning architectures have become so sophisticated that they seem to exhibit structures such as grid cells, which are present in biological neural brains used for navigation. Both neuroscience and deep learning can benefit each other from cross-pollination of ideas, and it’s highly likely that these fields will begin to merge at some point. Conclusion Now with deep learning and GPUs we can achieve higher accuracy at a practical speed! Deep learning is also much more accessible in terms of the learning curve. Much of the open source software is very easy to use and getting a simple language translator, chat bot, or image recognizer isn’t too challenging. Less complex math and coding, more making cool stuff! Thanks for reading!😄 🙌

Ball

Discovery

Travelers

Introduction to Deep Learning

HackerNoon Tech Stories Archive April 22nd, 2019

Bitcoin Bear Market Diaries Volume 19 Max Keiser

@piratebeachbum
Pirate Beachbum

Fixed point math in Solidity

@albertocuestacanada
Alberto Cuesta Cañada

Fixed point math in Solidity

@albertocuestacanada
Alberto Cuesta Cañada

Why Does the Global Freelance Economy Need a Mutiny?

@anthony-bayss
Anthony Bayss

Formjacking: Hackers’ New Favorite Way to Steal Credit Card Information (And How to Avoid it)

@sandeep15r
Sandeep Babu

Gandcrab — the Most Prevalent Ransomware in 2019

@david.w.balaban
David Balaban

Choosing the right platform for chatbot development: UX/UI perspective

@ciklum
Ciklum

High Level Overview of Apache Spark

@ericmgirouard
Eric Girouard

How blockchain could prevent companies like Facebook from manipulating users

@mywaymywei
Oscar W

What do companies expect from Python devs in 2019?

@andrew-ste
Andrew Ste

Blockchain Smart Cities Rating — From Melaka Straits City to Crypto Project in Nevada Desert

@bezverhi
Kirill Bezverhi

Startups: Pulse Surveys are Dumb.

@i_am_brennan
Brennan McEachran

Introduction to Deep Learning

@ilijamihajlovic
Ilija Mihajlovic

HackerNoon Tech Stories Archive April 22nd, 2019

@piratebeachbumPirate Beachbum

@albertocuestacanadaAlberto Cuesta Cañada

@albertocuestacanadaAlberto Cuesta Cañada

@anthony-bayssAnthony Bayss

@sandeep15rSandeep Babu

@david.w.balaban David Balaban

@ciklumCiklum

@ericmgirouardEric Girouard

@mywaymyweiOscar W

@andrew-steAndrew Ste

@bezverhiKirill Bezverhi

@i_am_brennanBrennan McEachran

@ilijamihajlovicIlija Mihajlovic

@piratebeachbum
Pirate Beachbum

@albertocuestacanada
Alberto Cuesta Cañada

@albertocuestacanada
Alberto Cuesta Cañada

@anthony-bayss
Anthony Bayss

@sandeep15r
Sandeep Babu

@david.w.balaban
David Balaban

@ciklum
Ciklum

@ericmgirouard
Eric Girouard

@mywaymywei
Oscar W

@andrew-ste
Andrew Ste

@bezverhi
Kirill Bezverhi

@i_am_brennan
Brennan McEachran

@ilijamihajlovic
Ilija Mihajlovic