Sketchnote

sketchnote

🙈 🙉 🙊. Edit: part 2 has also been published I would like to introduce you to streams. In a sketch notes medium. On medium dot com. node.js Sooooooooo, at some point last weekend I was reading through , started writing all those heaps of knowledge down, and voilà this tweet was born: node’s streams documentation As you can surely tell, my 4 year old phone camera is not made for taking any kind of pictures in the year 2016 of our lord (but it was great in 2012, I swear). This post contains a scanned (wow, scanners are still a thing, you say \o/) version of the aforementioned docs. As part 1 of the series, I am noting the basics of readables && , and their respective APIs. Stay tuned for the notes on implementing these lil things! writables You can now also read through . Edit: part 2 of these sketchnotes what are streams even; and a lil’ intro to writables writables writables writables writables; i did say writables, right? wowowoow readables got modes readable API and Events! readables! Continued! As a friendly wave, want to remind you these are , so I advise you to look at the for a more detailed examples of the way a certain method works. sketch notes documentation Thanks for reading through, yo; you’re dope 💯.

A sketch-note guide to node.js streams; part 1

The world’s most polarizing company just put the metaverse on the map. I’m not referring to Facebook creating an uber brand called Meta. Yes, the new name is important because it puts a stake in the ground. But even more significantly, the newly minted Meta is making investments to match its vision for the metaverse. One of those investments focuses on improving how we work -- which companies such as Microsoft are already doing with digital twins in workspaces such as factory floors and supply chains. A Gut Check on Terminology The metaverse is an interconnected virtual world where people live, work, and play through digital twins such as avatars. It’s a mirror of the digital world we are living in already. The metaverse is here already. It’s also where the internet was at the dawn of the 21st Century. We’re treating the metaverse like an exotic creature because someone assigned a name to it. In reality, we’re all building the metaverse whether we know it or not. Whenever any of us uses a digital currency or hangs out on Fortnite, we’re enabling the metaverse. But the metaverse is disconnected. Meta – and many other companies such as Epic Games and Microsoft – want to host that world and monetize it. The new name signifies that the Facebook social network alone won't do that. Facebook with its sister brands (as we know them today and will know them tomorrow) just might. The Company's Commitment The company is making a commitment to the metaverse at an interesting time. The Facebook brand itself has been battered by serious problems of its own making, most notably the emergence of a whistleblower who has laid bare a variety of ethical sins that have been already well documented. Facebook is like a wounded bear. As we know, a bear is most dangerous when it's been wounded or cornered. Facebook is very dangerous right now to its competition. On his best days, CEO Mark Zuckerberg has always been capable of defining a vision that shapes markets and puts the competition on the defensive. In recent quarterly earnings announcements, news media interviews, and at the annual Facebook Connect event, Zuckerberg is defining a compelling vision for the metaverse now as its umbrella company, Meta. “We’ve gone from desktop to web to phones from text to photos to video, but this isn’t the end of the line,” he said at Facebook Connect. “The next platform and medium will be even more immersive and embodied Internet where you’re in the experience, not just looking at it, and we call this the metaverse.” In a , he explained his vision more fully: recent call with investors So what is the metaverse? It’s a virtual environment. We can be present with people in digital spaces. And you can kind of think about this is an embodied Internet that you’re inside of rather than just looking at. And we believe that this is going to be the successor to the mobile Internet. You’re going to be able to access the metaverse from all different devices and different levels of fidelity from apps on phones and PCs to immersive virtual and augmented reality devices. Within the metaverse, you can build a hang out, play games with friends, work, create and more. You’re basically going to be able to do everything that you can on the Internet today, as well as some things that don’t make sense on the Internet today like dancing. The defining quality of the metaverse is presence, which is this feeling that you’re really there with another person or in another place. Creation, avatars and digital objects are going to be central to how we express ourselves and this is going to lead to entirely new experiences and economic opportunities. I think that, overall, this is one of the most exciting projects that we’re going to get to work on in our lifetime. But it could take a lot of work and no one company is going to be able to build this all by themselves. Part of what I’ve learned over the last five years is that we can’t just focus on building great experiences. We also need to make sure that we’re helping to build ecosystem so millions of other people can participate in the upside and opportunity of what we’re all creating. They’re going to need to be new protocols and standards, new devices, new chips, new software from rendering engines to payment systems and everything in between. In his , Venture Capitalist Matthew Ball added something that Zuckerberg did not: the metaverse is a fully functional economy where “individuals and businesses will be able to create, own, invest, sell, and be rewarded for an incredibly wide range of “work” that produces “value” that is recognized by others.” It’s an important point. Meta is not interested in the metaverse for fun. Meta knows there is economic value in the metaverse, and the company intends to benefit. seminal essay on the metaverse The Facebook arm of Meta has invested into the metaverse in 2021. Those investments include: $10 billion Horizon Marketplace: A marketplace where creators and developers can sell their own virtual goods. : where people come together to work in the same virtual room, regardless of physical distance. Horizon Workrooms Horizon Home, for leisure experiences in virtual homes. Better virtual and augmented reality tools needed to create immersive worlds, under development at . Facebook Reality Labs Horizon Workrooms is most intriguing to me. Horizon Workrooms promises to make meeting and collaboration more fun and creative. As such, Horizon Workroom shifts the conversation about the metaverse away from gaming and toward solving business problems such as managing a remote workplace. As Facebook notes, “Workrooms is a mixed reality experience, letting you bring your physical desk and compatible tracked keyboard into the virtual room with you, where you can see them sitting on the virtual meeting table in front of you.” Among other features, Workrooms offers infinite whiteboard space so that your avatars can sketch things out together in real time. Here, we see the metaverse offering an experience that improves on the one we’re getting after slogging our way through months of Zoom and Teams calls, where collaboration means squinting at someone’s shared screen on a tiny window. I think we’re all ready for something better as we work remotely. Improving How We Work Workrooms also suggests an important direction for the metaverse: improving how we work. Facebook is focusing on white-collar environments such as offices. Some businesses are already improving ways of working through the use of digital twins, or virtual representations of real-world places such as factory floors. For example: Anheuser-Busch InBev has used to create a digital model of their breweries and supply chain , “Synchronized with their physical environments, the digital model is live, up to date — reflecting the complex relationships between the natural ingredients and brewing process — and enables AB InBev’s brewmasters to adjust based on active conditions.” Azure Digital Twins . According to Microsoft Unilever also teamed with Microsoft to optimize operations in 300 plants by using building virtual versions of its factories. These digital twins us data streaming from sensor-equipped machines to create digital models that can track physical conditions and enable testing of operational changes. Digital twins may be key to tackling the supply chain crisis, : according to MIT Technology review Digital twins seek to solve breakages in the supply chain by anticipating them before they happen and then using AI to figure out a workaround. The name captures the key idea of simulating a complex system in a computer, creating a kind of twin that mirrors real-world objects—from ports to products—and the processes they are a part of. Simulations have been a part of decision-making in industry for some years, helping people explore different product designs or streamline the layout of a warehouse. But the availability of large amounts of real-time data and computing power means that more complex processes can be simulated for the first time, including the chaos of global supply chains that often rely on numerous vendors and transportation networks. The MIT Technology Review goes on to say that Amazon uses digital twins, and Google is supply-chain digital twins that the car maker Renault announced it had started using in September. Digital twins are significant because they help solve complex problems with scenario planning that cannot be tested effectively on real workplace settings. As these examples show, the metaverse is real. It’s providing value. And it belongs to no one. Meta's most promising opportunity in the near term is to create a better world where we collaborate remotely through our own digital twins. Other companies are already doing just that. Note: I invest in Facebook and Microsoft This article is part of The hosted by HackerNoon in partnership with . Gaming Metaverse Writing Contest The Sandbox Submit your story today for your chance to win up to $2000. #gaming-metaverse

Facebook Just Put the Metaverse on the Map

In the world of big , one of the most common yet challenging tasks is efficiently performing counts, intersections, and union operations on vast datasets. Apache Spark, renowned for its ability to handle large-scale data processing, provides essential tools for tackling these problems. However, the key to unlocking its full potential lies in understanding and leveraging effective data representation strategies. This article dives into how Apache Spark can be optimized for these specific operations. We will explore the role of data sketches, a powerful technique that offers a revolutionary approach to streamlining counts, intersections, and union computations, thereby enhancing query performance in large datasets. data analytics Understanding Data Representation Data representation involves how data is formatted and stored. In Apache Spark, choosing the right data representation is critical for optimizing performance. Spark's diverse data structures, like RDDs, DataFrames, and Datasets, cater to different processing needs, offering flexibility and efficiency. The Role of Data Representation in Query Performance When it comes to processing queries on large datasets, the format in which data is stored plays a critical role in determining query speed. Traditional data storage formats, while effective for a range of applications, often fall short in handling specific operations like distinct counts and intersections, especially when dealing with massive datasets. Traditional formats like CSV or even structured formats like Parquet and ORC are optimized for general query operations. They excel in scenarios involving filtering, aggregation, and joining over large volumes of data. However, when it comes to computing distinct counts or intersections - operations frequently encountered in big data analytics - these formats can be inefficient. This inefficiency arises because distinct count operations often require scanning the entire dataset to ensure accuracy, which becomes increasingly burdensome as data size grows. For intersection operations, traditional methods involve comparing each element of one set with all elements of another, which can lead to exponential increases in computation time as dataset sizes increase. This is particularly problematic in real-time analytics, where speed is of the essence. Data Sketches in Computing Approximations provide an innovative solution to these challenges. Unlike traditional methods that require processing entire datasets for accurate results, sketches use probabilistic algorithms to approximate these counts and intersections. This approach drastically reduces the amount of data that needs to be processed and stored, offering a much more efficient way to handle such queries on large datasets. Data sketches Data sketches are algorithms that allow for approximate computations on large datasets. They are particularly useful in scenarios where exact results are unnecessary or impractical due to resource constraints. By summarizing data into compact, probabilistic representations, data sketches enable fast, scalable, and approximate computations. This makes them ideal for applications in big data environments where the sheer volume of data makes traditional methods impractical.  Sketches trade a small, controlled amount of accuracy for significant gains in efficiency. They are ideal for applications like real-time analytics, where speed and scalability are more critical than exact precision. Implementing Data Sketches in Apache Spark with Scala Apache Spark, combined with libraries like Apache DataSketches, can efficiently implement sketching algorithms. Here's an example using Scala to demonstrate this integration in a real-world scenario. Scenario: Estimating Unique Users in Different Segments Consider a dataset where users belong to multiple segments. We want to estimate the number of unique users in each segment. Step 1: Generating Sketches First, we include the DataSketches library in our project and use it to create sketches for user segments. // Scala code to generate sketches for segments
import org.apache.datasketches.theta.{UpdateSketch, Sketch}

val spark = SparkSession.builder.appName("DataSketchesExample").getOrCreate()
val data = Seq(("user1", "Segment1"), ("user2", "Segment1"), ...)

val dataRDD = spark.sparkContext.parallelize(data)

val segment1RDD = dataRDD.filter(_._2 == "Segment1").map(_._1).mapPartitions { iter =>
  val sketch = UpdateSketch.builder().build()
  iter.foreach(item => sketch.update(item.getBytes))
  Iterator(sketch.compact.toByteArray)
}

// Saving the sketch to a file
val segment1Sketch = segment1RDD.reduce((sketchBytes1, sketchBytes2) => mergeSketches(sketchBytes1, sketchBytes2))
saveSketchToFile(segment1Sketch, "segment1_sketch.obj") Here, and are functions that merge sketches and save them to files, respectively. mergeSketches saveSketchToFile The function takes two byte arrays representing Theta sketches, merges them, and returns the merged sketch as a byte array. This is useful when you have sketches from different partitions of your data that need to be combined into a single sketch. mergeSketches import org.apache.datasketches.theta.{Sketch, SetOperation, UpdateSketch}

def mergeSketches(sketchBytes1: Array[Byte], sketchBytes2: Array[Byte]): Array[Byte] = {
  val sketch1 = Sketch.wrap(sketchBytes1)
  val sketch2 = Sketch.wrap(sketchBytes2)
  val union = SetOperation.builder().buildUnion()

  union.update(sketch1)
  union.update(sketch2)

  union.getResult.compact.toByteArray
} The function saves a given sketch (represented as a byte array) to a file. This is useful for persisting the sketch to disk, so it can be read and used later for queries. saveSketchToFile import java.io.{FileOutputStream, ObjectOutputStream}

def saveSketchToFile(sketch: Array[Byte], fileName: String): Unit = {
  val fos = new FileOutputStream(fileName)
  val oos = new ObjectOutputStream(fos)
  try {
    oos.writeObject(sketch)
  } finally {
    oos.close()
    fos.close()
  }
} Step 2: Querying with Sketches Sketches are initially computed at scale and stored in a distributed storage system such as Amazon S3 or HDFS. This step is typically part of a batch processing job. When a request for an estimate is received, the service first attempts to retrieve the corresponding sketch from a local RocksDB cache. If the sketch is not present in the cache, the service fetches it from the distributed storage system. import org.rocksdb.{RocksDB, Options}
import java.nio.file.Files

// Initializing RocksDB
RocksDB.loadLibrary()
val rocksDBOptions = new Options().setCreateIfMissing(true)
val db = RocksDB.open(rocksDBOptions, "path/to/rocksdb")

def saveSketchToDB(sketch: Array[Byte], key: String): Unit = {
  db.put(key.getBytes, sketch)
}

def getSketchFromDB(key: String): Option[Array[Byte]] = {
  Option(db.get(key.getBytes))
}


def fetchSketchFromDistributedStore(segmentId: String): Array[Byte] = {
  // Placeholder for the logic to fetch sketch from a distributed store (e.g., S3 or HDFS)
  // This would typically involve using APIs or SDKs specific to the storage solution
  Files.readAllBytes(java.nio.file.Paths.get(s"path/to/distributed/store/${segmentId}_sketch.obj"))
} Upon retrieval, the sketch is cached in RocksDB for efficient access in response to future queries. This approach ensures that the initial query for a specific sketch may take slightly longer due to the fetch operation, but subsequent queries are served rapidly using the cached data. This method is particularly beneficial in environments with large datasets, where sketches are computed periodically, and real-time estimates are frequently required. The actual implementation of fetching sketches from the distributed storage will vary based on the storage solution used and should include the necessary API calls or SDK methods for that specific system. Query Service Implementation def queryEstimate(segmentId: String): Double = {
  val key = s"${segmentId}_sketch"
  val sketchBytes = getSketchFromDB(key) match {
    case Some(bytes) => bytes
    case None => 
      val bytes = fetchSketchFromDistributedStore(segmentId)
      saveSketchToDB(bytes, key)
      bytes
  }

  val sketch = Sketch.wrap(sketchBytes)
  sketch.getEstimate
}

// Example usage
val estimateForSegment1 = queryEstimate("Segment1")
println(s"Estimated unique users in Segment 1: $estimateForSegment1") Disadvantages and Limitations of Using Data Sketches While data sketches offer significant advantages in handling large-scale datasets, particularly for distinct counts and intersections, they also come with certain drawbacks that need to be considered: : The most notable limitation of data sketches is that they provide This is a trade-off for their efficiency and scalability. In scenarios where absolute accuracy is paramount, data sketches might not be suitable. Approximation Errors approximate, not exact, results. : Understanding and interpreting the results from data sketches require a certain level of statistical knowledge. The probabilistic nature of the results means that they come with confidence intervals and error bounds, which might not be straightforward for all users. Complexity of Interpretation : Data sketches often involve tuning parameters (like sketch size or number of buckets), which can impact their accuracy and efficiency. Choosing the right parameters requires a good understanding of the underlying algorithms and the specific data characteristics. Tuning Parameters : Integrating data sketches into existing systems can add complexity. It requires selecting appropriate libraries, understanding their APIs, and sometimes significant changes to the data processing pipeline. Implementation Overhead : The error distribution in data sketches is not always uniform across different data ranges and distributions. This non-uniformity can lead to inaccuracies in certain cases and needs to be carefully managed. Non-Uniform Error Distribution Conclusion Efficient data representation, combined with advanced techniques like data sketches, can significantly enhance query performance in Apache Spark. By understanding and applying these methods, data engineers and scientists can optimize their data processing tasks, making them faster and more resource-efficient. Further Reading and Resources Apache Spark Documentation DataSketches Library Algebird Library

Stories, News & Mentions about SKETCHNOTE

Sketchnote’s Stories on HackerNoon