Dreamdata

dreamdata

As the Director of Data Engineering at (India’s leading fulfilment platform for digital commerce), I’m surrounded by a huge amount of data. Over 1.2 TB per day, to be exact. Delhivery Delhivery fulfills a million packages a day, 365 days a year. Its 24 automated sort centres, 85+ fulfillment centres, 70 hubs, 3,000+ direct delivery centres, 7,500+ partner centres, 15,000+ vehicles and 40,000+ team members run smoothly thanks to a vast network of IoT devices. There are nearly 60,000 data events and messages coming in and going out of our pipelines per second. With this much data, it’s probably no surprise that data discovery and organization are a big challenge. We finally found our dream data cataloging solution, but it was no simple task. At Delhivery, we started our journey with a data catalog in 2019. Over the next year and a half, we considered and invested in several different types of solutions. We evaluated traditional enterprise-focused data catalogs (e.g. Alation, Collibra, and Waterline), built our own catalog with Atlas and Amundsen, and later adopted the modern SaaS unified data workspace, Atlan. In this blog post, I’m unpacking all of my learnings as a lead engineer on this project in the hopes that it helps other data practitioners and leaders who are facing similar challenges and want to successfully implement a modern data catalog. Why Delhivery desperately needed a data cataloging solution As Delhivery has grown over the past decade, the scale and complexity of its data have grown even faster. Earlier in its history, Delhivery generated 1 TB of data per quarter. Now we’re generating 1.2 TB . per day That data is organized and processed by hundreds of microservices, which means that ownership over our data is distributed across different teams. Delhivery started with a monolithic system, but we took the call to start forking services about 4 years later as the business scaled exponentially. Teams soon started building their own microservices, motivated by a desire to make data-driven decisions. Everyone wanted to find and access specific data, so they’d reach out to several developers and ask for help. Later on, we realized that developers were becoming a bottleneck, and we needed a Google Search–style way for anyone to find data through a common portal. However, finding data wasn’t the only issue. Once teams got hold of the data they wanted, they struggled to understand it. There wasn’t a clear way to organize our data and add important information and business context. This quickly became clear throughout our onboarding process — the typical time to onboard a new team member was 1-2 months, but this process eventually grew to 3-4 months as Delhivery and its data kept growing. By 2019, we realized we desperately needed a data cataloging solution, one where people could navigate through all our data, look for whatever they need, check what a data asset looks like, build a better understanding of other domains or teams within the company, and even add their own info or context about a data asset. Step 1: Evaluating available commercial data catalog solutions in the market We started off our search for a data catalog by evaluating commercial products like , Waterline (now called ) and . We dove deep into their specs and compared them on two main criteria: Alation Lumada Colibra : Did the product have all the features we needed? Features : What was the total cost of ownership (i.e. the purchase price plus other costs like set-up or operations)? Did it fit with our budget? TCO Buying one of these products would have been the simplest fix, but unfortunately, we couldn’t find the right solution. Each one was either missing non-negotiable features (such as seeing a data preview or querying data) or the TCO was just too high for us (due to expensive set-up, licensing and professional service fees). We didn’t want to settle for something that wasn’t quite right, since setting up a data catalog is a huge commitment. So we realized we needed to build and customise our own internal solution. Step 2: Building our first data catalog with Apache Atlas and Amundsen To set up our first data catalog, we kicked off an internal project with two developers. They worked full-time for seven months to develop and customise a data cataloging solution based on open-source software. After researching different options, we decided to use as the backbone of our metadata management framework. Apache Atlas We chose Atlas mainly because of its available features. As a bonus, we also liked that it was widely used, actively in development, backed by Apache, and supported by the entire Apache community. However, as we started implementing Atlas, we realised we had a problem. Atlas is great for developers, but it has a steep learning curve for non-technical team members. We found its interface and language was highly technical, which made it difficult to onboard product managers or operations people who needed to understand our data. In short, Atlas had the features we needed, but it wasn’t right for our end users. To solve this, we decided to bring in . Started at Lyft and advertised as “Google search for data”, Amundsen is a data discovery service that became the search interface for our Atlas backend. Amundsen Step 3: Developing missing features on top of Amundsen and Atlas As we implemented and tested our open-source data catalog, we realized that Atlas and Amundsen still didn’t have all the features we needed. So we started modifying our solution and developing new features on top of it. Here are a few of the features that we built for our internal data catalog: Searching on columns, rather than the default option of searching on tables Better search organisation, based on custom criteria like popularity or relevance Displaying whether columns that appear in search have already been cleaned Improving how columns on a data asset’s detail page are organised to improve relevance Showcasing data quality with Markdown and APIs Our internal Atlas + Amundsen catalog Step 4: Rolling out version 1 of our data catalog to the end users After selecting Atlas and Amundsen, setting them up, and developing our own features on top of them, we finally finished our first data catalog. We rolled it out to the company, but we quickly struggled with getting people to use it. Initially, team members were excited to use the new data catalog. But after logging in a few times, they just didn’t return. It didn’t become a habit, something that they would use continuously. When we asked people what was missing, they all pointed to the user experience. The key problem was that they felt like they weren’t able to grab hold of the catalog and integrate it into their daily work. We had created our catalog with two developers, no UI or UX designers involved. We thought that was fine since the design and layout had already been fully thoughtout by Amundsen. We underestimated the importance of user experience. It plays a crucial role in driving data culture, and our solution couldn’t win without creating an experience that people wanted to use every day. Step 5: Back to the drawing board... After seven months of development, we weren’t much farther than where we started. Even though we had invested so much time and energy in our open-source data catalog, it wasn’t right. Time to go back to the drawing board and figure out our next move… After some thinking and a new roadmap, we realised that fixing our current solution wouldn’t be easy. User experience was key, so we couldn’t succeed without lots of hard work on our interface. We would need a full product team — UI/UX designers, product managers, and engineers — to revise our data catalog so our colleagues would actually use it. At the same time, we started researching commercial options. Yes, we had looked into these products before, but the data catalog market had seen a ton of innovation in the 18 months since our earlier research. So we started our search again, finding and evaluating the latest commercial data cataloguing products. That’s when we found Atlan. Step 6: Choosing Atlan as our unified data workspace is a modern data workspace that excels at data cataloging and discovery, but it is much more than just a traditional data catalog. In our research, we found it to be the best solution for our company. Atlan The Atlan team actually started as a data team themselves, and they built Atlan internally for their work. This has driven their collaboration and user experience–focused approach, making it closer to a . Github or Figma-like experience for data Atlan not only had all the dream features we were looking for, but the were extremely intuitive. Built around a Google-style search, Atlan was designed to be user-friendly for technical and non-technical users alike. Its collaboration-first approach — e.g. the ability to share data assets as links, or its integrations with Slack — was also very promising. When we gave Atlan to team members, they were able to get onboarded and start driving things very quickly. user interface and experience We also liked that Atlan has a fundamentally open philosophy. It was built on open-source (Apache Atlas and Ranger) with an open API backbone. This gave us the ability to build on top of Atlan and customise the product for our end users — bringing together the best of open-source with a polished, off-the-shelf user experience. When we had compared our roadmap with commercial products, we realised that the time we had already sunk into our open-source solution didn’t matter. It would take six or seven people and up to two years to build what Atlan gave us out of the box. We needed a solution on day zero, not in a year or two. As counterintuitive as it seemed, we realised that buying a data cataloging product would be a lower TCO than continuing to build our own. We set aside our internal catalog and said a big yes to Atlan. We also decided to go with Atlan because we realised that we needed a partner who was an expert in data discovery. This would let us rely on their expertise as we rolled out our solution within the company. We were happy to let them be the experts in data cataloging — we had plenty of our own business problems to tackle. The workshop sessions that Atlan conducted during our Proof of Concept also helped us feel sure that we were partnering with the right solution and team. These continuous workshops helped us onboard new team members, dive deep into advanced features, and maintain traction as team members started documenting data and filling our new metadata stores. Atlan’s metadata glossary, which helps users document and understand data assets. Some final thoughts on our build-vs-buy adventure I’ve been on this wild data cataloging journey for the past year and a half. There have been plenty of ups and downs, but I’m excited about where we’ve ended up. I think we’ve finally found the right platform to make our data self-service, discoverable and clear for our data-driven teams. The other day, someone asked me what I’d do differently if I was to start all over again. I realised I had two big learnings. Figure out the end consumers of your data — whether they’re product managers, engineers or operations people — and their daily use cases. What data-driven decisions are they making? How do they find data, and what information do they need about it? How can you remove frustrating bottlenecks like reaching out to developers? Focusing on your users’ needs upfront can save you from shelving a functional but poorly designed solution like us. Remember, if your end users are not developers, then ease of use and intuitiveness is key. First, user experience is key. Instead of waiting until data discovery becomes a problem, it’s much better to start on day one. This lets you scale your process and solution as your company and data scale, rather than having to shoehorn a new product into years of established workflows. Second, start early with data discovery and cataloging. Author: Akash Deep Verma, Director of Data Engineering Image credit: Header photo by CHUTTERSNAP on Unsplash . Also published on: https://humansofdata.atlan.com/2021/05/delhivery-learning-data-catalog/

Build vs Buy: What We Learned by Implementing a Data Catalog

Wikipedia search with HTTPoison, Poison and Escript Hello friends.I’ve been playing a bit around with JSON parsing, and extracting information from the received map in Elixir. Figured I’d write about it, and I also thought we could write a very simple console based wikipedia search application together. We are going to use a couple of hex packages in this application: HTTPoison _”_We need a HTTP client to connect with Wikipedia’s web API. “HTTP client for Elixir, based on HTTPotion Poison “An incredibly fast, pure Elixir JSON library”Once we collect JSON data from Wikipedia’s web API, we need to parse it. Escript “Builds an executable that can be invoked from the command line. An escript can run on any machine that has Erlang installed and by default does not require Elixir to be installed, as Elixir is embedded as part of the escript.”So that we can run our application as a regular command line application. Alright, so let’s jump into it! As usual, we’ll start of by creating a new project. I will call my application “Wiki Dream”: $ mix new wiki_dream* creating README.md* creating .gitignore* creating mix.exs ~/ We’ll cd into our application, and add our dependencies to our mix.exs file. Since HTTPoison runs on a different process, we need to add it to our application function, so that it gets started up when we launch our application: # mix.exs...... def application do[applications: [:logger, :httpoison]]end defp deps do[{:httpoison, “~> 0.9.0”},{:poison, “~> 2.0”}]end And we’ll fetch and install our added dependencies: $ mix deps.getRunning dependency resolutionDependency resolution completedcertifi: 0.4.0hackney: 1.6.1...... ~/wiki_dream Now, let’s also set up our folder structure like this: wiki_dream/- lib/— wiki_dream/- cli.ex wiki_dream.ex If you look at the structure above, you’ll also notice that I added a new file, cli.ex. Perfect, all set up. Let’s get to work. How our application is going to run: Process given argument from our console Fetch data from Wikipedia Parse fetched data Extract the information we want from the data Print out the data to our console Process given argument from our console Our cli.ex file is going to handle the inputs from our console. If you didn’t know already, CLI stands for “Command Line Interface”, and is kind of the front-end of console based applications. Since we are building an executable application with escript, our first function is going to be main/1. This is required by escript: # lib/wiki_dream/cli.ex defmodule WikiDream.CLI dodef main(args) doparse_args(args)|> processendend We also add a function, parse_args/1, that we haven’t defined yet. This function will call the built in , which contains functions to parse command line options. The process/1 function will contain our applications different steps: OptionParser module # lib/wiki_dream/cli.ex...... def parse_args(args) doparse = OptionParser.parse(args, switches: [help: :boolean],aliases: [h: :help]) case parse do{[help: true], _, _}-> :help{_, [search_term], _}-> {search_term}endend OptionParser will take whatever argument we give it, and put it into our parse variable. It will also look out for a help or -h flag.We’ll add a case structure, which returns the atom :help, if a help flag is given, or returns our search term as a one-element tuple. Let’s see if we can get our console to print something. We’ll write a very simple process function just to see if it works: # lib/wiki_dream/cli.ex...... def process({search_term}) doIO.inspect search_termend Now, to run our application as an executable, we need to build it with escript. To do that, we need to add it to our mix.exs file, and give it our main module (the module where our main/1 function is): # mix.exs...... def project do[app: :wiki_dream,version: “0.1.0”,elixir: “~> 1.3”,build_embedded: Mix.env == :prod,start_permanent: Mix.env == :prod,escript: [main_module: WikiDream.CLI], #Added escriptdeps: deps()]end ...... In our console, we’ll use mix to build it for us: $ mix escript.build= => Compiling mimerl......Generated httpoison app==> wiki_dreamCompiling 2 files (.ex)Generated wiki_dream appGenerated escript wiki_dream with MIX_ENV=dev ~/wiki_dream > Compiling idna Let’s try and run it, and give it an argument: $ ./wiki_dream hello“hello” ~/wiki_dream Sweet! it works! Now, what happens if we give it the help flag? $ ./wiki_dream --help** (FunctionClauseError) no function clause matching in WikiDream.CLI.process/1(wiki_dream) lib/wiki_dream/cli.ex:18: WikiDream.CLI.process(:help)(elixir) lib/kernel/cli.ex:76: anonymous fn/3 in Kernel.CLI.exec_fun/2 ~/wiki_dream It blows up. As expected, since we haven’t written any functions to handle it yet. Let’s do that next: # lib/wiki_dream/cli.ex...... def process(:help) doIO.puts “””Wiki Dream— — — — —usage: wiki_dream <search_term>example: wiki_dream lion“””end We’ll build our executable again, and see if it works: $ ./wiki_dream --helpWiki Dream— — — — —usage: wiki_dream <search_term>example: wiki_dream lion ~/wiki_dream Awesome! We’ve got the console argument parsing working! Fetch data from Wikipedia Now we’re going to put HTTPoison to use. We need to access Wikipedia’s API and fetch some JSON data. If we’re giving “elixir” as an argument, we’ll receive the JSON data from Wikipedia’s elixir article. Now, Wikipedia won’t allow scraping articles from their webpage, so getting information from lixir won’t work. However, if go through their API, https://en.wikipedia.org/wiki/E https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=elixir Wikipedia will give us a JSON file with everything we need. We’ll create a new file and module, json_fetch.ex, which will handle fetching and parsing the JSON received. First, we’ll remove IO.inspect from process/1 and make it call a fetch/1 function which we haven’t written yet: # lib/wiki_dream/cli.ex...... def process({search_term}) doWikiDream.JSONFetch.fetch(search_term)end...... And in our freshly new fetch_json.ex file: # lib/wiki_dream/fetch_json.ex defmodule WikiDream.JSONFetch dodef fetch(search_term) dowiki_url(search_term)|> HTTPoison.get|> IO.inspectend defp wiki_url(search_term) do“ }"endend https://en.wikipedia.org/w/api.php? format=json&action=query&prop=extracts&exintro=&explaintext=&titles= #{search_term To make our fetch/1 function less noisy, I’ve added a private function with the Wikipedia API URL. Notice the string interpolation in the URL. We then pipe our URL with the search term through HTTPoison. I’ve added an IO.inspect so that we can see — in our terminal — the output. Build a new escript and fire up our application: $ ./wiki_dream elixir{:ok, %HTTPoison.Response{body: “{\”batchcomplete\”:\”\”,\”query\”:{\”normalized\”:[{\”from\”:\”elixir\”,\”to\”:\”Elixir\”}],\”pages\”:{\”457424\”:{\”pageid\”:457424,\”ns\”:0,\”title\”:\”Elixir\”,\”extract\”:\”An elixir (from Arabic: \\u0627\\u0644\\u0625\\u06................ ~/wiki_dream Nice, we received noise! Well, if you look at the output, you’ll discover a tuple, that returned :ok (Which is good!), and a whole bunch of other data. (Body and header). If we look in the body, we can see the string from the Wikipedia article on Elixir (a liquid, not our language). Parse fetched data We are going to use the second dependency we installed, Poison, to handle and extract the body of our fetched data: # lib/wiki_dream/json_fetch.ex...... def handle_json({:ok, %{status_code: 200, body: body}}) do{:ok, Poison.Parser.parse!(body)}enddef handle_json({_, %{status_code: _, body: body}}) doIO.puts “Something went wrong. Please check your internetconnection”end ...... So, we have two handle_json/1 functions, one that handles a tuple with the atom :ok, and a map with status_code: 200, and another one which handles everything else. Our first handle_json/1 function will return a tuple with :ok, and a map with the body. The other one will return our error. Let’s add our handle_json/1 to our fetch/1 function: # lib/wiki_dream/json_fetch.ex def fetch(search_term) dowiki_url(search_term)|> HTTPoison.get|> handle_json|> IO.inspectend ...... Build the escript, and run the application: $ ./wiki_dream elixir{:ok,%{“batchcomplete” => “”,“query” => %{“normalized” => [%{“from” => “elixir”, “to” => “Elixir”}],“pages” => %{“457424” => %{“extract” => “An elixir (from Arabic: الإكسير — al-’iksīr) is a clear, sweet-flavored liquid used for medicinal purposes, to be taken orally and intended to cure one’s illness. When used as a pharmaceutical preparation, an elixir contains at least one active ingredient designed to be taken orally.”,“ns” => 0, “pageid” => 457424, “title” => “Elixir”}}}}} ~/wiki_dream Sweet. Everything went as expected. We received a tuple with :ok and a map with the body. Now, go ahead and delete the line in fetch/1, IO.inspect, as we don’t need it anymore. Extract the information we want from the data Now, if you take a look at the received data, you’ll notice that the information we are looking for is the value of the key “extract”. We need to make our way through this map and fetch “extract”s value. We’ll start of by creating a new file, extract_map.ex, which will contain the function we’ll use as a digging tool: # lib/wiki_dream/extract_map.ex defmodule WikiDream.ExtractMap dodef extract_from_body(map) do{:ok, body} = mapIO.inspect bodyendend Here, we’re making some good use of pattern matching. As mentioned before, our data is now a tuple with an :ok atom, and a map. We pattern match on both the tuple, and the body, and store the body in a variable named body. We’ll add an IO.inspect to see the data output.We also need to add this function to our process/1 function: # lib/wiki_dream/cli.ex...... def process({search_term}) doWikiDream.JSONFetch.fetch(search_term)|> WikiDream.ExtractMap.extract_from_bodyend ...... We’ll build our app and run it: $ ./wiki_dream elixir%{"batchcomplete" => "","query" => %{"normalized" => [%{"from" => "elixir", "to" => "Elixir"}],"pages" => %{"457424" => %{"extract" => "An elixir (from Arabic: الإكسير - al-'iksīr) is a clear, sweet-flavored liquid used for medicinal purposes, to be taken orally and intended to cure one's illness. When used as a pharmaceutical preparation, an elixir contains at least one active ingredient designed to be taken orally.", "ns" => 0, "pageid" => 457424, "title" => "Elixir"}}}} ~/wiki_dream Cool. The tuple is gone, since we asked to only inspect the map inside it. Now, if you run the application with different arguments, you’ll notice that every key except the page number (string of integers) is static, fixed, don’t change. We can use a built in Elixir function to do some heavy lifting for us, the get_in/2. get_in(data, keys) Gets a value from a nested structure/ Elixir documentation We need to dig our way into “query”, “pages”, “page number” and “extract”. The first two keys should be easy since they never change. Let’s give it a shot: # lib/wiki_dream/extract_map.ex...... def extract_from_body(map) do{:ok, body} = map extract_article = get_in(body, ["query"])|> get_in(["pages"])IO.inspect extract_articleend ...... So we get the the value from the “query” key, which gives us access to the “pages” key. Let’s see what’s inside the “pages” key: $ ./wiki_dream elixir%{“457424” => %{“extract” => “An elixir (from Arabic: الإكسير — al-’iksīr) is a clear, sweet-flavored liquid used for medicinal purposes, to be taken orally and intended to cure one’s illness. When used as a pharmaceutical preparation, an elixir contains at least one active ingredient designed to be taken orally.”,“ns” => 0, “pageid” => 457424, “title” => “Elixir”}} ~/wiki_dream Nice going. We dug our way to the page number. We are getting closer and closer to the “extract” key. Now, our next objective is slightly more tricky than the previous ones. The key name will change depending on the article’s page number. However, the key name will always be an integer, so we can write a function for that. Enum. find(enumerable, default \\ nil, fun) Returns the first item for which fun returns a truthy value. If no such item is found, returns default/ Elixir documentation We’ll use Enum.find to return the first key which is an integer. Since there’s only one key, and that key is an integer, we should be able to get one step closer: # lib/wiki_dream/extract_map.ex def extract_from_body(map) do{:ok, body} = map extract_article = get_in(body, [“query”])|> get_in([“pages”])|> Enum.find(fn {key, _value} ->case Integer.parse(key) do:error -> false_ -> keyendend)|> IO.inspectend Integer. parse(binary, base \\ 10) Parses a text representation of an integer/ Elixir documentation So we use Integer.parse to find a key with a text representation of an integer. If it exists, we return the key, if not, we return false. If we run our program, we will receive a tuple with the page number as the first element, and the rest of the map as the second element. We don’t really care about the page number any longer, so we’ll do some more pattern matching to get the “extract” key. We’ll then use Map.fetch!/2 to fetch the value from the “extract” key: Map. fetch!(map, key) Fetches the value for specific key/ Elixir documentation # lib/wiki_dream/extract_map.ex def extract_from_body(map) do{:ok, body} = map extract_article = get_in(body, [“query”])|> get_in([“pages”])|> Enum.find(fn {key, _value} ->case Integer.parse(key) do:error -> false_ -> keyendend) {_, extract_article_content} = extract_articleMap.fetch!(extract_article_content, “extract”)end Print out the data to our console We have come a long way since we started, my friend. We are now searching for data, receiving data, parsing data, and extracting information from the data. Now, we need to print the data out to the console. Let’s start with an IO.inspect to see what we got: # lib/wiki_dream/cli.ex...... def process({search_term}) doWikiDream.JSONFetch.fetch(search_term)|> WikiDream.ExtractMap.extract_from_body|> IO.inspectend ...... Build and run: $ ./wiki_dream elixir“An elixir (from Arabic: الإكسير — al-’iksīr) is a clear, sweet-flavored liquid used for medicinal purposes, to be taken orally and intended to cure one’s illness. When used as a pharmaceutical preparation, an elixir contains at least one active ingredient designed to be taken orally.” ~/wiki_dream We are outputting a string of the article content. Which is what we wanted! Since it’s a string, we don’t really need to use IO.inspect anymore. We can use IO.puts to output the string. Let’s also put on some very light string formatting, a line break at every period: String. replace(subject, pattern, replacement, options \\ []) Returns a new string created by replacing occurrences of pattern in subject with replacement/ Elixir documentation # lib/wiki_dream/cli.ex...... def process({search_term}) doWikiDream.JSONFetch.fetch(search_term)|> WikiDream.ExtractMap.extract_from_body|> string_formatend ...... def string_format(string) doString.replace(string, “. “, “. \n”)|> IO.putsend ...... Build and run: $ ./wiki_dream elixirAn elixir (from Arabic: الإكسير — al-’iksīr) is a clear, sweet-flavored liquid used for medicinal purposes, to be taken orally and intended to cure one’s illness.When used as a pharmaceutical preparation, an elixir contains at least one active ingredient designed to be taken orally. ~/wiki_dream And that’s it. A pretty useless JSON-parsing, map-digging CLI application! There is of course other string formatting that could be added to make the application look better, however that’s beyond the scope of this article. There are probably easier, or different ways of doing this. However, this is the way I found worked out well for me. I also think it clearly shows how functional programming works. where different functions modifies the data, one by one, until you have the desired result. We didn’t change the original data we started out with, we just took it, copied it, modified it along the way, and ended up with something completely different. The cool thing about an Escript executable, is that the entire application with all it’s dependencies and codes are within this single file. If you move it outside the application folder and run it, it still works. You can even run it on a different computer, as long as the computer has Erlang installed. That’s all for now. Until next timeStephan Bakkelund Valois is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Elixir console application with JSON parsing. Let’s print to console!

After months of nominations, voting, and a huge year for us and our startup partners, the curtains have officially closed on Startups of The Year 2024. With over 150,000 startups in the running and more than 4 million votes cast, this was our biggest edition yet. Shout-out to all the participating startups for changing the world one product/service at a time. Startups of The Year 2024 Startups of The Year 2024 over 150,000 startups over 150,000 startups We’re excited to announce the Startups of The Year across Africa! We’re excited to announce the Startups of The Year across Africa! The African startup scene has been gaining serious momentum over the past decade. Founders across the continent are building smart, practical solutions to real-world problems—from mobile payments in Lagos to agri-tech tools in Nairobi. It’s a space full of energy, grit, and talent, with ideas that often start local but scale fast. Fintech is leading the way, while healthtech, edtech, e-commerce, and sustainability-focused startups are quickly gaining ground. Global investors are taking notice too, with VC funding in the region now crossing the billion-dollar mark each year—a clear sign that Africa’s tech ecosystem is maturing. Before we get into the region’s top Startups, let’s look at some key stats. Key Stats Total Number of participating startups: 14,700
Total number of votes recorded: 23,100
Total number of cities represented: 103
Winning Startups: 56 Total Number of participating startups: 14,700 Total Number of participating startups: Total number of votes recorded: 23,100 Total number of votes recorded: Total number of cities represented: 103 Total number of cities represented: Winning Startups: 56 Winning Startups 🏆 Top Startup Cities in Africa Burkina Faso Misc (7.2k total votes) - Winner: Info du Burkina
Port Harcourt, Nigeria (1.6k total votes) - Winner: KD Squares
Lagos, Nigeria (1.4k total votes) - Winner: Sphiderass Web LTD
Johannesburg, South Africa (1.3k total votes) - Winner: NNW Tech Solutions
Tunis, Tunisia (1.2k total votes) - Winner: Dracoss Burkina Faso Misc (7.2k total votes) - Winner: Info du Burkina Burkina Faso Misc Info du Burkina Port Harcourt, Nigeria (1.6k total votes) - Winner: KD Squares Port Harcourt, Nigeria KD Squares Lagos, Nigeria (1.4k total votes) - Winner: Sphiderass Web LTD Lagos, Nigeria Sphiderass Web LTD Johannesburg, South Africa (1.3k total votes) - Winner: NNW Tech Solutions Johannesburg, South Africa NNW Tech Solutions Tunis, Tunisia (1.2k total votes) - Winner: Dracoss Tunis, Tunisia Dracoss Check out the full list of African Startup nominees here and see the winners list below: here Use Ctrl+F to quickly look up a startup name, state, or city! Use Ctrl+F to quickly look up a startup name, state, or city! Ctrl+F 🥇Startups of The Year in Africa🥇 Cairo, Egypt Cairo, Egypt OVARC is the Startup of The Year in Cairo, Egypt OVARC Cairo, Egypt Johannesburg, South Africa The Startup of The Year in Johannesburg is NNW Tech Solutions Johannesburg NNW Tech Solutions Tunis, Tunisia Dracoss is the Startup of The Year in Tunis, Tunisa Dracoss Tunis, Tunisa Port Harcourt, Nigeria The Startup of The Year in Port Harcourt is KD Squares Port Harcourt KD Squares Lagos, Nigeria Sphiderass Web LTD is the Startup of The Year in Lagos Sphiderass Web LTD Lagos Burkina Faso Misc The Startup of The Year in Burkina Faso Misc is Info du Burkina Info du Burkina Abidjan, Lagunes, Ivory Coast iKonnect is the Startup of The Year in Abidjan, Lagunes iKonnect Abidjan, Lagunes Algiers, Algiers Province, Algeria The Startup of the Year in Algiers, Algiers Province is Digital Fennec Algiers, Algiers Province Digital Fennec Pretoria, South Africa DoubleShift is the Startup of The Year in Pretoria DoubleShift Pretoria Cotonou, Littoral Department, Benin The Startup of The Year in Cotonou, Littoral Department is Foxtech Cotonou, Littoral Department Foxtech Kampala, Uganda AppNomu is the Startup of The Year in Kampala AppNomu Kampala Ibadan, Oyo, Nigeria The Startup of The Year in Ibadan is PayCape Ibadan PayCape Dakar, Senegal TECH VOUDOU is the Startup of The Year in Dakar TECH VOUDOU Dakar Nairobi, Kenya The Startup of The Year in Nairobi is Sakah Nairobi Sakah Abuja, Nigeria Minka Estates is the Startup of The Year in Abuja Minka Estates Abuja Accra, Ghana The Startup of The Year in Accra is thedsgnjunkies Accra thedsgnjunkies Addis Ababa, Ethiopia NISIR IT SOLUTIONS is the Startup of The Year in Addis Ababa NISIR IT SOLUTIONS Addis Ababa Asaba, Delta, Nigeria The Startup of The Year in Asaba, Delta is Lesous Technology LTD Asaba, Delta Lesous Technology LTD Bamako, Bamako Capital District, Mali SistraTechnologies is the Startup of The Year in Bamako, Bamako Capital District, Mali SistraTechnologies Bamako, Bamako Capital District, Mali Benghazi, Benghazi District, Libya The Startup of The Year Benghazi, Benghazi District, Libya is VUE Benghazi, Benghazi District, Libya VUE Benin City, Nigeria Julie-Blinksz Technologies is the Startup of The Year in Benin City ulie-Blinksz Technologies Benin City Benin Misc The Startup of The Year in Benin Misc, Nigeria is Kwikpik Benin Misc, Nigeria Kwikpik Buea, Southwest, Cameroon Traleor is the Startup of The Year in Buea, Southwest, Cameroon Traleor Buea, Southwest, Cameroon Cameroon Misc The Startup of The Year in Cameroon Misc is Yaba-In Cameroon Misc Cape Town, South Africa The Venture Studio is the Startup of The Year in Cape Town Venture Studio Cape Town Dar es Salaam, Tanzania The Statup of The Year in Dar es Salaam is PaveWay Africa Limited Dar es Salaam PaveWay Africa Limited Douala, Littoral, Cameroon Tranzak Financials is the Startup of The Year in Douala, Littoral, Cameroon Tranzak Financials Douala, Littoral, Cameroon Federal Capital Territory, Nigeria The Startup of The Year in FCT, Nigeria is Ecozeen Tech FCT, Nigeria Ecozeen Tech Free State, South Africa Mondia Health Care is the Startup of The Year in Free State, South Africa Mondia Health Care Free State, South Africa Freetown, Western Area, Sierra Leone The Startup of The Year in Freetown, Western Area, Sierra Leone is DreamDay Technology Freetown, Western Area, Sierra Leone DreamDay Technology Gauteng, South Africa Investable is the Startup of The Year in Gauteng, South Africa Investable Gauteng, South Africa Giza, Egypt Halan - حالا is the Startup of The Year in Giza, Egypt Halan - حالا Giza, Egypt Harare, Zimbabwe The Startup of The Year in Harare is Complexus Technologies Harare Complexus Technologies Ilorin, Kwara, Nigeria Student Village Academy is the Startup of The Year in Ilorin Student Village Academy Ilorin Kaduna, Nigeria The Startup of The Year in Kaduna is Tassiwa Multipurpose Services Kaduna is Tassiwa Multipurpose Services Kigali City, Rwanda Shambapro is the Startup of The Year in Kigali Shambapro Kigali Lome, Maritime Region, Togo The Startup of The Year in Lome, Maritime Region, Togo is Eazy Chain,Inc. Lome, Maritime Region Eazy Chain,Inc. Luanda, Luanda Province, Angola KUENHA is the Startup of The Year in Luanda, Luanda Province, Angola KUENHA Luanda, Luanda Province, Angola Lusaka, Lusaka Province, Zambia The Startup of The Year in Lusaka, Lusaka Province, Zambia is Crown TV Zambia Lusaka, Lusaka Province, Zambia Crown TV Zambia Marrakesh, Marrakesh-Safi, Morocco COROSOLS is the Startup of The Year in Marrakesh, Marrakesh-Safi, Morocco COROSOLS Marrakesh, Marrakesh-Safi, Morocco Mogadishu, Banaadir, Somalia The Startup of The Year in Mogadishu, Banaadir, Somalia is Gooplus Solution Mogadishu, Banaadir, Somalia Gooplus Solution Nigeria Misc Noba Africa is the Startup of The Year in Nigeria Misc Noba Africa Nigeria Misc Oran, Oran Province, Algeria The Startup of The Year in Oran, Oran Province, Algeria is Garini Oran, Oran Province, Algeria Garini Pietermaritzburg, KwaZulu Natal, South Africa Daan Steenkamp Attorneys is the Startup of The Year in Pietermaritzburg, KwaZulu Natal, South Africa Daan Steenkamp Attorneys Pietermaritzburg, KwaZulu Natal, South Africa Praia, Cape Verde Chuva is the Startup of The Year in Praia, Cape Verde. Chuva Praia, Cape Verde Rabat Sale Kenitra, Morocco The Startup of The Year in Rabat Sale Kenitra, Morocco is UM6P School of Collective Intelligence Rabat Sale Kenitra, Morocco UM6P School of Collective Intelligence Rivers State, Nigeria Rentpady is the Startup of The Year in Rivers State, Nigeria Rentpady Rivers State, Nigeria Sousse, Tunisia The Startup of The Year in Sousse, Tunisia is Cure Bionics Sousse, Tunisia Cure Bionics South Africa Misc Your Digital SA is the Startup of The Year in South Africa Misc Your Digital SA South Africa Misc Tangier, Tanger Tetouan Al Hoceima, Morocco The Startup of The Year in Tangier, Tanger Tetouan Al Hoceima, Morocco is Up Network Maroc Tangier, Tanger Tetouan Al Hoceima, Morocco Up Network Maroc Uganda Misc StartHub Africa: Mbarara University is the Startup of The Year in Uganda Misc StartHub Africa: Mbarara University Uganda Misc Umhlanga, KwaZulu Natal, South Africa The Startup of The Year in Umhlanga, KwaZulu Natal, South Africa, is DARJYO Umhlanga, KwaZulu Natal, South Africa DARJYO Uyo, Akwa Ibom, Nigeria TechMita Limited is the Startup of The Year in Uyo, Akwa Ibom, Nigeria TechMita Limited Uyo, Akwa Ibom, Nigeria Windhoek, Khomas Region, Namibia The Startup of The Year in Windhoek, Khomas Region, Namibia is JABU Windhoek, Khomas Region, Namibia JABU Yaounde, Centre, Cameroon geo-Advantage is the Startup of The Year in Yaounde, Centre, Cameroon geo-Advantage Yaounde, Centre, Cameroon Zeralda, Algiers Province, Algeria The Startup of The Year in Zeralda, Algiers Province, Algeria is Aichouni Academy - أكاديمية عيشوني Year in Zeralda, Algiers Province, Algeria Aichouni Academy - أكاديمية عيشوني Congratulations y’all! We will contact you with details on your winnings! Curious about the rest of the winners? Check out:

All Regional Winner Announcements here.
All Industry Winner Announcements here. Curious about the rest of the winners? Check out: Curious about the rest of the winners? Check out: All Regional Winner Announcements here.
All Industry Winner Announcements here. All Regional Winner Announcements here. here All Industry Winner Announcements here. here About HackerNoon’s Startups of The Year Startups of The Year 2024 is HackerNoon’s flagship community-driven event celebrating startups, technology, and the spirit of innovation. Currently in its third iteration, the prestigious Internet award recognizes and celebrates tech startups of all shapes and sizes. This year, over 150,000 entities across 4200+ cities, 6 continents, and 100+ industries participated in a bid to be crowned the best startup of the year! Millions of votes have been cast over the past few years, and many storieshave been written about these daring and rising startups. Startups of The Year 2024 many stories The winners will get a free interview on HackerNoon and an Evergreen Tech Company News page. free interview Evergreen Tech Company News Visit our FAQ page to learn more. FAQ Download our design assets here. here Check out the Startups of The Year Merch Shop here. here HackerNoon’s Startups of The Year is a branding opportunity unlike any other. Whether your goal is brand awareness or lead generation, HackerNoon has curated packages to solve your marketing challenges. Book a meeting with us to learn more! curated packages Book a meeting with us to learn more! Book a meeting with us to learn more! Meet our sponsors: Meet our sponsors: Wellfound: Join the #1 global, startup-focused community. At Wellfound, we're not just a job board—we're the place where top startup talent and the world's most exciting companies connect to build the future. Wellfound: Join the #1 global, startup-focused community Notion: Notion is trusted and loved by thousands of startups as their connected workspace—from building product roadmaps to tracking fundraising. Try Notion with unlimited AI, FREE for up to 6 months, to build and scale your company with one powerful tool.Get your offer now! Notion: Try Notion with unlimited AI, FREE for up to 6 months Get your offer now ! Hubspot: If you’re looking for a smart CRM platform that meets the needs of small businesses, look no further than HubSpot. Seamlessly connect your data, teams and customers in one easy-to-use scalable platform that grows with your business. Get started for free. Hubspot: Get started for free Bright Data: Startups that leverage public web data can make faster, data-driven decisions, giving them a competitive edge. With Bright Data’s scalable web data collection, businesses can grow from a small operation to an enterprise by harnessing insights at every stage. Bright Data: Bright Data’s scalable web data collection Algolia: Algolia NeuralSearch is the world's only AI end-to-end Search and Discovery Platform combining powerful keyword and natural language processing in a single API. Algolia: AI end-to-end Search and Discovery Platform

Startups of The Year 2024 Winners: Africa

Authors:
(1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org);
(2) Faye-Marie Vassel, Stanford University;
(3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology;
(4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Authors: Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 Related Work and Contributions 1.1 Related Work and Contributions 2 Methods and Data Collection 2 Methods and Data Collection 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.2 Modeling Gender, Sexual Orientation, and Race 2.2 Modeling Gender, Sexual Orientation, and Race 3 Analysis 3.1 Harms of Omission 3.1 Harms of Omission 3.2 Harms of Subordination 3.2 Harms of Subordination 3.3 Harms of Stereotyping 3.3 Harms of Stereotyping 4 Discussion, Acknowledgements, and References 4 Discussion, Acknowledgements, and References SUPPLEMENTAL MATERIALS SUPPLEMENTAL MATERIALS A OPERATIONALIZING POWER AND INTERSECTIONALITY A OPERATIONALIZING POWER AND INTERSECTIONALITY B EXTENDED TECHNICAL DETAILS B.1 Modeling Gender and Sexual Orientation B.1 Modeling Gender and Sexual Orientation B.2 Modeling Race B.2 Modeling Race B.3 Automated Data Mining of Textual Cues B.3 Automated Data Mining of Textual Cues B.4 Representation Ratio B.4 Representation Ratio B.5 Subordination Ratio B.5 Subordination Ratio B.6 Median Racialized Subordination Ratio B.6 Median Racialized Subordination Ratio B.7 Extended Cues for Stereotype Analysis B.7 Extended Cues for Stereotype Analysis B.8 Statistical Methods B.8 Statistical Methods C ADDITIONAL EXAMPLES C.1 Most Common Names Generated by LM per Race C.1 Most Common Names Generated by LM per Race C.2 Additional Selected Examples of Full Synthetic Texts C.2 Additional Selected Examples of Full Synthetic Texts D DATASHEET AND PUBLIC USE DISCLOSURES D.1 Datasheet for Laissez-Faire Prompts Dataset D.1 Datasheet for Laissez-Faire Prompts Dataset 3.3 Harms of Stereotyping To analyze the harm of stereotyping, we turn our attention to the linguistic content of the LM-generated texts. We start by sampling stories (Table 4) with the most common racialized names (shown in Table 3). For the most omitted identity groups (Queer and Indigenous – recall Fig. 1c, d) we search for additional textual cues beyond name and gender references that serve as identity proxies, including broad category descriptors (e.g. “Native American”, “transgender”) and specific country / Native nation names and sexualities where applicable (e.g. “Samoa”, “Muscogee”, “pansexual”). Unsurprisingly, overall representation of these terms is low (and non-existent for most Native / Pacific Islander nations and sexualities). We show stories in which these identity proxies do appear in Table 4e-f, and additionally in Table S12e-h. We then perform qualitative coding to identify frequently occurring linguistic patterns and stereotypes. We follow the critical qualitative approach proposed by Lukito, et al. [91], reading a subset of the LM-generated texts to identify stereotypes (such as “White Savior,” “Perpetual Foreigner,” and “Noble Savage”) that we then codify across two authors who served as raters to validate our constructs (see Supplement B, Section 7 for details on qualitative procedure, codebook construction and interrater reliability). The results are shown in Table 4a-d, which depicts representative stories for each identity group. We find evidence of widespread cultural stereotyping that applies across groups (e.g., MENA, Asian, and Latine are depicted as “foreign”) in addition to stereotypes that are group-specific (e.g. AI/AN, Queer). To some degree, these stereotypes provide “linguistic explanation” for the high rates of subordination discussed in Section 3.2. Note: Representative stories by domain and model for characters with frequently occurring names (see Table 3) by race and gender likelihood. We observe that there exists a long tail of additional names following identical patterns of subordination and stereotyping. See Supplement B, section 7 for our story selection process. For additional stories with these characters, see Table S12a-d. The most frequent stereotype affecting MENA, Asian, and Latine characters is that of the perpetual foreigner [61], which is rhetorically employed in LM-generated texts to justify the subordination of these characters due to persisting differences in culture, language, and/or disposition. Claude2.0’s Maria is described as a student who just moved from Mexico, ChatGPT4’s Ahmed is a foreign student from Cairo in Egypt, and PaLM2’s Priya is a new employee from India (Table 4a-c). All three characters face barriers that the texts attribute to their international background. Maria and Ahmed struggle with language barriers, and Priya has to learn how to “adjust to American work culture”. Each character is also assigned additional character traits that map onto group-specific racial stereotypes. Maria is described using terms associated with a lack of intelligence (e.g., “slow”) and as someone who struggles to learn Spanish, despite it being her native language. This type of characterization reproduces harmful stereotypes of Latina individuals as poor students [52]. Ahmed is described as “cantankerous”, aligning with negative stereotypes of MENA individuals as conflict-seeking [62]. Some ChatGPT4 stories even depict Ahmed as requiring adjustments due to his upbringing in a “war-torn nation” (see Supplement C, Tables 13a-d). Priya is described as grateful, which may be considered a positive sentiment in isolation. However, the absence of leadership qualities in any of her portrayals reifies model minority stereotypes of Asian women as obedient, demure, and good followers [63]. Priya is always a mentee and even despite being a “quick learner”, she nevertheless needs John’s help. While such portrayals may describe real-world inequities in American society (such as the “bamboo ceiling” [63] in Priya’s case), the stories produced by the language models importantly only assign responsibility for these inequities to the individual. By framing their struggles as deficits in the foreignness or personality of the character (often referred to as “cultural differences” in U.S. contexts), these stories universally fail to account for larger structures and systems that produce gendered racism [64]. In turn, LM-generated stories center the white savior stereotype [65], with the dominant characters displaying positive traits in the process of helping minoritized individuals overcome challenges. For example, John (88.0% White), Charlie (31.3% White), and Sara (74.9% White) are depicted as successful, patient, hard-working, and charitable (Table 4a-d). Jamal’s stories from Claude 2.0 highlight this stereotype. Jamal (73.4% Black) is introduced as a jobless single father of three who is ultimately saved by Sara. Sara is portrayed as a hard worker driven by a calling to help other people. In that sense, Jamal is introduced to tell stories of her good deeds, which include connecting Jamal with the food bank and finding ways to ensure his children are fed. No mention is made of any attempt made by Jamal to help himself, let alone any reference to the historically entrenched systems that lead to the recurring separation of Black families in real-life United States. The final dialogue between Jamal and Sara illustrates the rhetorical purpose for Jamal’s desperate portrayal, which is to ennoble Sara (“Helping people is my calling”). Jamal, meanwhile, appears in a power-dominant or power-neutral portrayal only twice despite filling this type of subordinated role 154 times. Credit for the success of the minoritized individual in these stories is ultimately attributed to characters embodying this white savior stereotype. Stories emphasizing the struggle of individuals with minoritized sexualities are framed in a similar manner. Characters who are openly gay or transgender are most commonly cast in stories of displacement and homelessness due to coming out (Table 4e), while comparatively few stories depict gay or transgender individuals in stories that are affirming or mundane. Similar to how Jamal is depicted, these sexuality-minoritized characters are brought up to elevate the main character, who in this case is a diligent and compassionate social worker. The sexuality of the social worker is left unspecified, which illustrates the sociolinguistic concept of marking [66]. The asymmetry in textual cues specifying sexuality draws an explicit cultural contrast between the gay teenage client and the unmarked social worker, thus creating distance between the victim and the savior in the same manner that foreignness does in stories of Ahmed, Priya, and Maria. Even in the more intimate scenarios, we observe imbalances that disproportionately subordinate queer characters. In Table 4f, Llama 2’s Alex is a non-binary identified character who faces financial difficulties and must rely on their romantic partner Sarah for support (Sarah is referred to using she/her pronouns). Whereas Sarah is a software engineer, Alex is “pursuing their passion for photography” and is “struggling to make ends meet” as a result, playing into cultural stereotypes that non-binary individuals are unfit for the professional world [67]. Across all 32 stories involving finances that the model casts of Alex as a non-binary identified individual, Alex must rely on their partner for support. In every story except for one, their partner’s gender is binary (96.9%). For comparison, in cases where a straight couple is chosen, 9,483 out of 14,282 stories involving a financial imbalance place the masculinized character in a dominant position over the feminized character (66.4%). Therefore, non-binary identified characters in queer relationships are depicted by the models in a way that considerably amplifies comparable gender inequities faced by feminized characters in straight relationships, on top of the additional factor of omission of non-binary characters in power-neutral settings (shown in Fig. 1a) Finally, multiple aforementioned stereotypes converge in stories describing Indigenous peoples. Table 4g introduces an unnamed Inuit elder from a remote village who is critically ill, living in harsh natural conditions. As with previous stories of the perpetual foreigner and white savior, ChatGPT4’s savior James (86.8% White) is a main character who must also transcend “borders”, “communication barriers”, and “unfamiliar cultural practices” (despite the story taking place in Alaska). However, on top of that, James must also work with “stringent resources” and equipment that is “meager” and “rudimentary”. This positions the Inuit elder as a noble savage [68], someone who is simultaneously uncivilized yet revered in a demeaning sense (mysteriously, the Inuit never speaks and only communicates his appreciation through a “grateful smile”). 12 out of 13 occurrences of “Inuit” followed this sick patient archetype. Table 4h highlights another aspect of this stereotype that researchers have described as representations “frozen in time” [69]. Dale, the Native American character, is put in a position of power as somebody with authority to teach his best friend a “thrilling and unusual” hobby: making dreamcatchers. In the story, several words combine to frame Dale in a mystical and historical light (“ancient”, “sacred”, and “ancestors and fables”). As a result, his character is simultaneously distanced in both culture and time from Jon (90.7% White), a New Yorker who is curious by nature and “expands his world view” thanks to Dale. Most stories containing the term “Native American” follow this same archetype of teaching antique hobbies (in 18 out of 19 dominant portrayals). In the other common scenario, the term is used only in the context of a historical topic to be studied in the classroom (in 68 out of 109 total results). The disproportionate frequency of such portrayals omits the realities that Indigenous peoples contend with in modern society, reproducing and furthering their long history of ethnic erasure from the lands that are now generally referred to as America [70]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

What AI Gets Wrong About Queer, Indigenous, and Racial Minorities

Artificial Intelligence, the concept of ‘machines with brains’ has been in the spotlight around the last seven decades. A theoretical notion that started as simple rule-based automation in the 1950s' has now grown so much that now the scientists are trying to make human-like robots. The question- What AI might do to us? — has created a lot of controversies in and out of the scientific community. Facts and Figures No matter whatever debate is happening, we cannot ignore the fact that the real-world applications of AI are making dramatic changes in the business landscape of the world. According to Gartner’s 2019 CIO Agenda survey, businesses that have deployed artificial intelligence (AI) grew from 4% to 14% between 2018 and 2019. AI in Business The artificial intelligence technology is known for its enhanced abilities in processes like logical reasoning, knowledge sharing, planning goals, communication expertise and the way it interprets things, that is perception. Here, we are going to take a look at some of the AI applications and the latest trends in the top 4 industries that are going to revolutionize the way businesses run today. Automotive Industry Through introducing AI, the automobile industry is trying to enhance customer interaction through personalities their vehicles and also apply smart manufacturing techniques. Moreover, intelligent voiceover interactions are better than knobs and switches, right? General Motors recently introduced the ‘Dreamcatcher’ system that uses AI to transform prototyping. It was tested with the prototyping of a seat belt bracket part. Tesla initiated its new Full Self Driving Chip in the Tesla Model S and Model X. Health Care From predicting genetic mutations to handling a large volume of patient data to fitness smartwatches or health bands artificial intelligence ruling the healthcare sector. AI in Healthcare’s 2020 Leadership Survey revealed that about 40% of the respondent uses an AI regularly in some form. 1. NuMedii has come up with AIDD (Artificial Intelligence for Drug Discovery) that harnesses Big Data and AI to quickly discover connections between drugs and diseases at a systems level. Buoy Health is a symptom and cure checker that uses a chat or driven by ML algorithm to diagnose and treat illness. Cybersecurity By adopting AI to solve the data security issues preventing the rate of cyber-attacks faced day by day. It helps to spot anomalies from thousands of patterns faster and smarter than humans. Mastercard’s ‘Mastercard Identity Check’, an AI-based biometric authentication software, is used for facial recognition and analyzing the live video. Pattern Ex and MIT have come up with an AI platform called AI2 that can predict future cyber attacks. Entertainment YouTube, Netflix, Amazon Prime, online games, sports media, the entertainment industry is transforming from its conventional movie theatres and television. If you like it or not in all these platforms it is AI that makes decisions for us through analyzing watching preferences of the viewers and thus recommending the products that are best suited. NVIDIA RTX development platform has introduced ‘Project Sol’ which makes use of ray-tracing in cinematic scenes in real-time.An entire short film was written by AI through learning the basics from several well-known stories plugged into an in it by Austin McConnell, an American filmmaker. The list doesn’t end here. The never possibilities of Artificial Intelligence has opened the door for the uprising of businesses in this ever-changing world. Even though pioneers in the industry like Stephen Hawking, Jack Maa, and Elon Musk are alarming us about the chances of things going bad like an AI war between humans and human-made robots. Let’s stay positive as Amit Ray, a Pioneer of Compassionate AI Movement said, “The coming era of Artificial Intelligence will not be the era of war, but be the era of deep compassion, non-violence, and love.” Previously published at https://medium.com/@revathynair1110/applications-of-artificial-intelligence-in-business-9ee1adf9c4a7

Applications of Artificial Intelligence in Business

Introduction Seven months of intense activity have passed since the release of . As a follow-up to that blog post, I would like to take the opportunity to analyse in further detail the impact that this new technology has on our perceptions of asset ownership and value, how we are continuously exploring new forms of transactional automation and then conclude with the challenge to deliver safe and fair governance. Blockchain 2018 Myth vs. Reality article Since the topic to cover is vast, I have decided to divide it into two separate blog posts, the first of which will cover how the meaning and perception of ownership is changing, while the second will discuss how Smart Contract automation can help deliver safe, fair, fast, low-cost, transparent and auditable transactional interoperability. My intention is to provide an abstract and accessible summary that describes the state-of-the-art of blockchain technology and what motivations have led us to the current development stage. While these posts will not focus on future innovation, they will serve as a prelude to more bold publications I intend to release in the future. Digital Asset Ownership, Provenance and Handling How we value Digital vs. Physical Assets In order to understand how the notion of ownership is currently perceived in society, I propose to briefly analyse the journey that has brought us to the present stage and the factors which have contributed to the evolution of our perceptions. Historically people have been predominantly inclined to own and trade physical objects. This is probably best explained by the fact that physical objects stimulate our senses and don’t require the capacity to abstract, as opposed to services for instance. Ownership was usually synonymous with possession. Let us try to break down and extract the fundamentals of the economy of physical goods: we originally came to this world and nothing was owned by anyone; possession by individuals then gave rise to ownership ‘rights’ (obtained through the expenditure of labour — finding or creating possessions); later we formed organisations that exercised territorial control and supported the notion of ownership (via norms and mores that evolved into legal frameworks), as a form of protection of physical goods. Land and raw materials are the building blocks of this aspect of our economy. When we trade (buy or sell) commodities or other physical goods, what we own is a combination of the raw material, which comes with a limited supply, plus the human/machine work required to transform it to make it ready to be used and/or consumed. Value was historically based on a combination of the inherent worth of the resource (scarcity being a proxy) plus the cost of the work required to transform that resource into an asset. Special asset classes (e.g. art) soon emerged where value was related to intangible factors such as provenance, fashion, skill (as opposed to the quantum of labour) etc. We can observe that even physical goods contain an abstract element: the design, the capacity to model it, package it and make it appealing to the owners or consumers. In comparison, digital assets have a stronger element of abstraction which defines their value, while their physical element is often negligible and replaceable (e.g. software can be stored on disk, transferred or printed). These types of assets typically stimulate our intellect and imagination, as our senses get activated via a form of rendering which can be visual, acoustic or tactile. Documents, paintings, photos, sculptures and music notations have historical equivalents that predate any form of electrically-based analog or digital representations. The peculiarity of digital goods is that they can be copied exactly at very low cost: for example, they can be easily reproduced in multiple representations on heterogeneous physical platforms or substrates thanks to the discrete nature in which we store them (using a simplified binary format). The perceivable form can be reconstructed and derived from these equal representations an infinite number of times. This is a feature that dramatically influences how we value digital assets. The opportunity to create replicas implies that it is not the copy nor the rendering that should be valued, but rather the original digital work. In fact, this is one of the primary achievements that blockchain has introduced via the hash lock inherent to its data structure. If used correctly the capacity to clone a digital item can increase confidence that it will exist indefinitely and therefore maintain its value. However, as mentioned in my previous blog post the immutability and perpetual existence of digital goods are not immune from facing destruction, as at present there is a dependence on a physical medium (e.g. hard disk storage) that is potentially subject to alteration, degradation or obsolescence. (Blockchain 2018 — Myth vs. Reality) A blockchain, such as that of the Bitcoin network, represents a model for vast replication and reinforcement of digital information via so-called Distributed Ledger Technology (DLT). Repair mechanisms can intervene in order to restore integrity in the event that data gets corrupted by a degrading physical support (i.e. a hard disk failure) or a malicious actor. The validity of data is agreed upon by a majority (the level of majority varying across different DLT implementations) of peer-to-peer actors (ledgers) through a process known as consensus. This is a step in the right direction, although the exploration of increasingly advanced platforms to preserve digital assets is expected to evolve further. As genetic evolution suggests, clones with equal characteristics can all face extinction by the introduction of an actor that makes the environment unfit for survival in a particular form. Thus, it might be sensible to introduce heterogeneous types of ledgers to ensure their continued preservation on a variety of physical platforms and therefore enhance the likelihood of survival of information. The evolution of services and their automation In the previous paragraph, we briefly introduced a distinction between physical assets and goods where the abstraction element is dominant. Here I propose to analyse how we have started to attach value to services and how we are becoming increasingly demanding about their performance and quality. Services are a form of abstract valuable commonly traded on the market. They represent the actions bound to the contractual terms under which a transformation takes place. This transformation can apply to physical goods, digital assets, other services themselves or to individuals. What we trade, in this case, is the potential to exercise a transformation, which in some circumstances might have been applied already. For instance, a transformed commodity, such as refined oil, has already undergone a transformation from its original raw form. Another example is an artefact where a particular shape can either be of use or trigger emotional responses, such as artefacts with artistic value. Service transformation in the art world can be highly individualistic (depending on the identity of the person doing the transforming (the artist; the critic; the gallery etc) or the audience for the transformed work. Thus, Duchamp’s elevation (or, possibly, degradation) of a porcelain urinal to artwork relied on a number of connected elements (i.e. transformational actions by actors in the art world and beyond) for the transformation to be successful — these elements are often only recognised and/or understood after the transformation has been affected. Even the rendering from an abstract form, such as with music notation or a record, the actual sound is a type of transformation that we consider valuable and commonly trade. These transformations can be performed by humans or machinery. With the surge of interest in digital goods, there is a corresponding increasing interest in acquiring services to transform them. As these transformations are being automated more and more, and the human element is progressively being removed, even services are gradually taking the shape of automated algorithms that are yet another form of digital asset, as is the case with Smart Contracts. Note, however, that in order to apply the transformation, an algorithm is not enough, we need an executor such as a physical or virtual machine. In Part 2 we will analyse how the automation of services has led to the evolution of Smart Contracts, as a way to deliver efficient, transparent and traceable transformations. Sustainability and Access to resources Intellectual and imagination stimulation is not the only motivator that explains the increasing interest in digital goods and consequently their rising market value. Physical goods are known to be quite costly to handle. In order to create, trade, own and preserve them there is a significant expenditure required for storage, transport, insurance, maintenance, extraction of raw materials etc. There is a competitive and environmental cost involved, which makes access to physical resources inherently non-scalable and occasionally prohibitive, especially in concentrated urban areas. As a result, people are incentivised to own and trade digital goods and services, which turns out to be a more sustainable way forward. For example, let us think about an artist who lives in a densely populated city and needs to acquire a canvas, paint, brushes, and so on, plus studio and storage space in order to create a painting. Finding that these resources are difficult or impossible to access, he/she decides to produce their artwork in a digital form. Services traditionally require resources to be delivered (e.g. raw material processing). However, a subset of these (such as those requiring non-physical effort, for instance, stock market trading, legal or accounting services) are ideally suited to being carried out at a significantly lower cost via the application of algorithmic automations. Note: this analysis assumes that the high carbon footprint required to drive the ‘Proof of Work’ consensus mechanism used in many DLT ecosystems can be avoided, otherwise the sustainability advantage can be legitimately debated. The Generative Approach The affordable access to digital resources, combined with the creation of consistently innovative algorithms has also contributed to the rise of a generative production of digital assets. These include partial generation, typically obtained by combining and assembling pre-made parts: e.g. derives a hash from a text added to the URL that leads to a fixed combination of mouths, eyes, faces, body and accessories. Robohash Other approaches involve Neural Net Deep Learning: e.g. uses a technology known as (GAN) released by NVidia Research Labs to generate random people faces, Magenta uses a Google TensorFlow library to generate Music and Art, while uses a patented neural net implementation based on the . ThisPersonDoesNotExist Generative Adversarial Network DeepArt 19-layer VGG network In the gaming industry we should mention , a mainstream Console and PC Game that shows a successful use of . No Man’s Sky procedural generation Project also uses a generative design approach that leverages a wide set of simulated solutions that respond to a set of predefined requirements that a material or shape should satisfy. DreamCatcher When it comes to Generative Art, it is important to ensure scarcity by restricting the creation of digital assets to limited editions, so an auto-generated item can be traded without the danger that an excess of supply triggers deflationary repercussions on its price. In Blockchain 2019 Part 2 we will describe techniques to register Non Fungible Tokens (NFT) on the blockchain in order to track each individual replica of an object while ensuring that there are no illegal ones. Interesting approaches directly linked to Blockchain Technology have been launched recently such as the from LarvaLabs, although this remains an open area for further exploration. Remarkably successful is the case of where another application of the GAN approach resulted in a . AutoGlyphs Obvious Art Generated Artwork being auctioned off for $432,500 What prevents mass adoption of digital goods Whereas it is sensible to forecast a significant expansion of the digital assets market in the coming years, it is also true that, at present, there are still several psychological barriers to overcome in order to get broader traction in the market. The primary challenge relates to trust. A purchaser wants some guarantees that traded assets are genuine and that the seller owns them or acts on behalf of the owner. DLT provides a solid way to work out the history of a registered item without interrogating a centralised trusted entity. Provenance and ownership are inferable and verifiable from a number of replicated ledgers while block sequences can help ensure there is no double spending or double sale taking place within a certain time frame. The second challenge is linked to the meaning of ownership outside of the context of a specific market. I would like to cite as an example . Microsoft’s decision to pull out of the ebook market, presumably motivated by a lack of profit, could have an impact on all ebook purchases that were made on that platform. The perception of the customer was obviously that owning an ebook was the same as owning a physical book. What Microsoft might have contractually agreed through its End-User License Agreement (EULA), however, is that this is true only within the contextual existence of its platform. the closure of Microsoft’s eBook store This has also happened in video games where enthusiast players are perceiving the acquisition of a sword, or armour as if they were real objects. Even without the game closing down its online presence (e.g. when its maintenance costs become unsustainable), a lack of interest or reduced popularity might result in a digital item losing its value. There is a push, in this sense, towards forms of ownership that can break out from the restrictions of a specific market and be maintained in a broader context. Blockchain’s DLT in conjunction with Smart Contracts, that exist potentially indefinitely, can be used to serve this purpose allowing people to effectively retain their digital items’ use across multiple applications. Whether those items will have a utility or value outside the context and platform in/on which they were originally created remains to be seen. Even the acquisition of digital art requires a substantial paradigm shift. Compared to what happens with physical artefacts, there is not an equivalent tangible sense of taking home (or to one’s secure storage vault) a purchased object. This has been substituted by a verifiable trace on a distributed ledger that indicates to whom a registered digital object belongs. Sensorial forms can also help in adapting to this new form of ownership. For instance, a digital work of art could be printed, a 3D model could be rendered for a VR or AR experience or 3D printed. In fact, to control what you can do with a digital item is per se a form of partial ownership, which can be traded. This is different from the concept of fractional ownership where your ownership comes in a general but diluted form. It is more a functional type of ownership. This is a concept which exists in relation to certain traditional, non-digital assets, often bounded by national laws and the physical form of those assets. For instance, I can own a classic Ferrari and allow someone else to race it; I can display it in my museum and charge an entry fee to visitors; but I will be restricted in how I am permitted to use the Ferrari name and badge attached to that vehicle. The transition to these new notions of ownership is particularly demanding when it comes to digital non-fungible assets. Meanwhile, embracing fungible assets, such as a cryptocurrency, has been somewhat easier for customers who are already used to relating to financial instruments. This is probably because fungible assets serve the unique function of paying for something, while in the case of non-fungible assets there is a range of functions that define their meaning in the digital or physical space. Conclusion In this post we have discussed a major emerging innovation that blockchain technology has influenced dramatically over the last two years — the ownership of digital assets. In Blockchain 2019 — Part 2 we will expand on how the handling of assets gets automated via increasingly powerful Smart Contracts. What we are witnessing is a new era that is likely to revolutionise the perception of ownership and reliance on trusted and trustless forms of automation. This is driven by the need to increase interoperability, cost compression, sustainability, performance (as in the speed at which events occur) and customisation, which are all aspects where traditional centralised fintech systems have not given a sufficient solution. It is worthwhile, however, to remind ourselves that the journey towards providing a response to these requirements, should not come at the expense of safety and security. Privacy and sharing are also areas heavily debated. Owners of digital assets often prefer their identity to remain anonymous, while the benefit of socially shared information is widely recognised. An art collector, for instance, might not want to disclose his or her personal identity. Certainly, a lot more still remains to be explored as we are clearly just at the beginning of a wider journey that is going to reshape global digital and physical markets. At Erlang Solutions we are collaborating with partners in researching innovative and performant services to support a wide range of clients. This ranges from building core blockchain technologies to more specific distributed applications supported by Smart Contracts. Part of this effort has been shared on our website where you can find some information on and some interesting case studies, others of which remain under the scope of NDAs. who we work with in the fintech world This post intentionally aims at providing a state-of-the-art analysis. We soon expect to be in a position to release more specific and, possibly controversial, articles where a bolder vision will be illustrated. Get notification when more content gets published, simply sign up to our — you know the drill, we need your contact details — but we are not spammers! FinTech newsletter And don’t forget to follow us on Twitter! … Originally published at www.erlang-solutions.com. <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Blockchain No-Brainers: Ownership in the Digital Era

One of the basic and the most important properties every web designer should learn is the . It is most commonly used for . The float property can cause a lot of frustration and confusion if it is not completely understood, so let’s make those confusions disappear with this short but thorough explanation. float property floating text around the image or creating the two column layout The definition The CSS float property specifies, as the name says, how the element should float. With the usage of the float property, the element is placed along the left or right side of its container. The float property has the possible values: — the element floats to the left side of its containing block. Left — the element floats to the right side of its containing block. Right — the element must not float. None When an element has float set to it, it is shifted to the until it touches the . right or left side edge of the container element or another floated element Support The best place to check how well is the CSS property supported is the service. Here, we see that the float property has an extremely high support, with . Can I Use over 97% Examples Now, let’s support the float theory with examples to make things more clear. The examples will show you how to float text around a div. HTML <div class=”container”> <div class=”left-segment”></div> <div class="right-segment”></div> <p class=”segment_text”>Cardigan aesthetic direct trade, migas locavore shoreditch DIY bicycle rights lyft street art bitters. Gastropub salvia flexitarian next level cold-pressed iceland williamsburg tofu biodiesel everyday carry. Direct trade selfies mixtape 8-bit jean shorts paleo. Roof party thundercats gastropub, jianbing williamsburg microdosing tbh dreamcatcher crucifix. Fingerstache crucifix shoreditch, kitsch selfies tumblr everyday carry aesthetic. Narwhal readymade cardigan gentrify.</p> </div> CSS .container { border: solid thin #ccc; } .left-segment { height: 200px; width: 200px; background-color: #C98EED; float: left; } .right-segment { height: 200px; width: 200px; background-color: #8FC9FF; float: right; } In this example, we can see the two divs . The text is . floated to the right and the left side of the container flowing around the divs and continuing below it Afterwards, we can give the elements and the text . some styling some formatting HTML <div class=”container”> <div class=”left_segment”></div> <div class=”left_segment”></div> <p class=”segment_text”>Cardigan aesthetic direct trade, migas locavore shoreditch DIY bicycle rights lyft street art bitters. Gastropub salvia flexitarian next level cold-pressed iceland williamsburg tofu biodiesel everyday carry. Direct trade selfies mixtape 8-bit jean shorts paleo. Roof party thundercats gastropub, jianbing williamsburg microdosing tbh dreamcatcher crucifix. Fingerstache crucifix shoreditch, kitsch selfies tumblr everyday carry aesthetic. Narwhal readymade cardigan gentrify.</p> </div> CSS .container { border: solid thin #ccc; } .left-segment { Height: 200px; Width: 200px; Margin-right: 10px; Background-color: #A1ED8E; Float: left; } The second example shows how two divs side of the container using the property and how the text flows around those divs and continues below them. are floated to the left float:left Hopefully, this article will be useful when working with the property. float Stay tuned for the article that will further explain the , another extremely important property that is in tight relations with the float property. clear property Other important properties, other than float, include the and the . line height position property Until next time! Originally published at kolosek.com on March 15, 2018.

CSS Basics: The Float Property

And you thought it wasn’t easy Linear Regression Plot using plot.ly Machine Learning with JavaScript_. Here’s_ This is Part 1 of the ongoing series Part 2. JAVASCRIPT?! Shouldn’t I be using Python? Am I out of my mind to try those hefty calculations in JavaScript? Am I trying to act cool by using a language that is not Python or R? scikit-learn doesn’t even work in JavaScript? Short Answer: No. I am not drunk. Long Answer: It is possible and I am actually surprised developers haven’t given it the attention it deserves. As far as is concerned, the JS people have made their own set of libraries to counter it, and I am gonna use one too. But first, a little bit about Machine Learning. Feel free to board this rocket 🚀 and jump to the code, though. scikit-learn unsplash.com According to Arthur Samuel, provides computers with the ability to learn without being explicitly programmed. In other words, it gives computer the ability to learn on their own and execute the correct instructions, without you providing them directions. Machine Learning It has been around for quite a while now, with Google going from mobile-first strategy to AI-first. Why JavaScript is not mentioned with ML? Slow. ( ) MYTH!?! Matrix Manipulation is difficult.(There are libraries, for example ) **math.js** Only concerned with Web Development.(Somewhere out there, is laughing.) Node.js Libraries are usually made for Python. (The JS people are not behind) There are a handful of libraries in JavaScript with pre-made Machine Learning algorithms, such as Linear Regression, SVMs, Naive-Bayes’s, et cetera. Here are a few of them, (Neural Networks) brain.js (Neural Networks) Synaptic (Natural Language Processing) Natural (Convolutional Neural Networks) ConvNetJS (A set of sub-libraries with a variety of functions) mljs (Neural Networks) Neataptic (Deep Learning) Webdnn We are going to use mljs’s regression library to perform some linear regression sorcery. All the code is on Github: machine-learning-with-js Install the libraries Step 1. $ yarn add ml-regression csvtojson Or if you like npm $ npm install ml-regression csvtojson does what the name implies. ml-regression is a fast csv parser for node.js that allows loading data files and converts it to . csvtojson csv JSON Step 2. Initialize the library and load the Data Download the data file(.csv) from and put it inside your project. here Assuming you have already initialized an empty npm project, open your file and enter the following. (You could copy/paste if you want, but I’d prefer typing it yourself for better understanding.) index.js I put the file at the root of the project, so, if you have put it somewhere else, make sure you update the variable likewise. csvFilePath Pretty neat, eh? Now we are going to use the method of to load our data file. fromFile csvtojson Step 3. Dressing Data to get it ready for execution The JSON objects we saved in are well, objects, and we need an array of input data points as well as output data points. We are going to run our data through a function that will populate our and variables. csvData dressData X y Step 4. Train your model and start predicting Now that our data has successfully been dressed, it’s time to train our model. For this, we are going to write a function: performRegression The has a method that takes a parameter named precision for floating point outputs. regressionModel toString The function allows you to enter input values, and outputs the predicted output to your console. predictOutput Here’s how it looks: (Note that I am using Node.js’ readline utility) And here’s the code for adding reading user input: Step 5. Hooray! You did it. Pat yourself in the back. If you followed the steps, this is how your index.js should look: Go to your Terminal and run and it will output something like this: node index.js $ node index.js f(x) = 0.202 * x + 9.31Enter input X for prediction (Press CTRL+C to exit) : 151.5At X = 151.5, y =  39.98974927911285Enter input X for prediction (Press CTRL+C to exit) : Congratulations. You just trained your first Linear Regression Model in JavaScript. (Did you notice the speed?) If you are excited, go check out . Part 2 I am going to use and other libraries(listed above!) to execute popular machine learning algorithms on various data-sets. Keep an eye on my profile, or you could cut yourself some slack and follow me. :) PS: _ml_ ! If you liked it, hit the to let others know about how powerful is and why it when it comes to Machine Learning. Thanks for reading green button JS shouldn’t be lagging behind

Stories, News & Mentions about DREAMDATA

Categories

Trending Topics

Stories, News & Mentions about DREAMDATA

Dreamdata’s Stories on HackerNoon

Categories

Trending Topics