SuperScanis in the Productivity Software Industry

SuperScan

In Swift, using for asynchronous code may seem like pure hell because, under the hood, are considered complete if the compilation of their synchronous code is completed. OperationQueue Operations In other words, compiling the example described below will output a broken execution order since, by the time the asynchronous code is executed, the itself will have already been completed. Operation let operationQueue = OperationQueue()
operationQueue.maxConcurrentOperationCount = 1

operationQueue.addOperation {
    DispatchQueue.main.asyncAfter(deadline: .now() + 1) {
        print("First async operation complete")
    }
    print("First sync operation complete")
}

operationQueue.addOperation {
    DispatchQueue.main.asyncAfter(deadline: .now() + 1.5) {
        print("Second async operation complete")
    }
    print("Second sync operation complete")
} This code will print: First sync operation complete
Second sync operation complete
First async operation complete
Second async operation complete However, there is a way to circumvent these restrictions. To understand how to solve the problem, you need to understand how works under the hood. Operation The itself has four flags by which you can track the life cycle of the operation: Operation — indicates whether the can be performed at this time. isReady Operation —indicates whether an is currently in progress. isExecuting Operation —indicates whether the is currently completed. isFinished Operation —indicates whether the was canceled. isCancelled Operation In theory, the enters the state before the itself is executed asynchronously, so we need to develop a technique by which we will be able to manipulate the life cycle of the . Operation isFinished Operation Operation This possibility can be solved by subclassing the and also by redefining the / methods, as well as all the flags on which the operation's life cycle is built. Operation start cancel Here’s the code: public class AsyncOperation: Operation {
    // MARK: Open

    override open var isAsynchronous: Bool {
        true
    }

    override open var isReady: Bool {
        super.isReady && self.state == .ready
    }

    override open var isExecuting: Bool {
        self.state == .executing
    }

    override open var isFinished: Bool {
        self.state == .finished
    }

    override open func start() {
        if isCancelled {
            state = .finished
            return
        }
        main()
        state = .executing
    }

    override open func cancel() {
        super.cancel()
        state = .finished
    }

    // MARK: Public

    public enum State: String {
        case ready
        case executing
        case finished

        // MARK: Fileprivate

        fileprivate var keyPath: String {
            "is" + rawValue.capitalized
        }
    }

    public var state = State.ready {
        willSet {
            willChangeValue(forKey: newValue.keyPath)
            willChangeValue(forKey: state.keyPath)
        }
        didSet {
            didChangeValue(forKey: oldValue.keyPath)
            didChangeValue(forKey: state.keyPath)
        }
    }
} The subclass we received from the is basic and allows us to forcefully complete it manually. Operation To work with completion blocks, you should create another subclass. However, this will not be a subclass of the , but of . Operation AsyncOperation public typealias VoidClosure = () -> Void
public typealias Closure<T> = (T) -> Void

public class CompletionOperation: AsyncOperation {
    // MARK: Lifecycle

    public init(completeBlock: Closure<VoidClosure?>?) {
        self.completeBlock = completeBlock
    }

    // MARK: Public

    override public func main() {
        DispatchQueue.main.async { [weak self] in
            self?.completeBlock? {
                DispatchQueue.main.async {
                    self?.state = .finished
                }
            }
        }
    }

    // MARK: Private

    private let completeBlock: Closure<VoidClosure?>?
} This subclass will allow us to pass a closure to the , after which the will be completed. Operation Operation Let’s try this type of operation in practice: let operationQueue = OperationQueue()
operationQueue.maxConcurrentOperationCount = 1

operationQueue.addOperation(
    CompletionOperation { completion in
        DispatchQueue.main.asyncAfter(deadline: .now() + 1) {
            print("First async operation complete")
            completion?()
        }
        print("First sync operation complete")
    }
)

operationQueue.addOperation(
    CompletionOperation { completion in
        DispatchQueue.main.asyncAfter(deadline: .now() + 1.5) {
            print("Second async operation complete")
            completion?()
        }
        print("Second sync operation complete")
    }
) As a result, we were able to achieve synchronous execution of : Operations First sync operation complete
First async operation complete
Second sync operation complete
Second async operation complete Don’t hesitate to contact me on if you have any questions. Also, you can always . Twitter buy me a coffee Also published here

OperationQueue + Asynchronous Code: Everything You Need to Know

OperationQueue + অ্যাসিঙ্ক্রোনাস কোড: আপনার যা কিছু জানা দরকার

OperationQueue + Asynchroner Code: Alles, was Sie wissen müssen

OperationQueue + Código asincrónico: todo lo que necesita saber

OperationQueue + Code asynchrone : tout ce que vous devez savoir

ऑपरेशनक्यू + एसिंक्रोनस कोड: वह सब कुछ जो आपको जानना आवश्यक है

OperationQueue + 非同期コード: 知っておくべきことすべて

OperationQueue + 비동기 코드: 알아야 할 모든 것

OperationQueue + código assíncrono: tudo o que você precisa saber

OperationQueue + асинхронный код: все, что вам нужно знать

OperationQueue + Asenkron Kod: Bilmeniz Gereken Her Şey

OperationQueue + Mã không đồng bộ: Mọi thứ bạn cần biết

OperationQueue + 异步代码：您需要了解的一切

In the ever-evolving landscape of computing, the quest for optimal performance has led to the exploration of parallelism — the art of breaking down complex tasks into smaller, manageable ones. At the heart of this pursuit lies the fundamental concept of threading, a powerful technique that enables programs to execute multiple tasks concurrently, unleashing the true potential of modern computing systems. Threading involves breaking a program into smaller, independently executable units called threads, running simultaneously to enhance efficiency and speed. This is one of the fundamental concepts of parallel programming. Before delving into the intricacies of threading, it’s essential to grasp the theoretical framework that underpins the world of parallel computing. Flynn’s Taxonomy, named after computer scientist Michael Flynn, provides a comprehensive classification system for parallel processing architectures. Understanding Flynn’s Taxonomy is crucial because it lays the foundation for comprehending the intricate relationships between threads, processes, and the overall structure of parallelized systems. A clear understanding of these concepts is essential for understanding how to create multi-threaded applications. This article marks the beginning of a blog series on multi-threading, covering all the theoretical and programming aspects of multi-threading. We’ll dig deep into threading APIs and libraries (POSIX Threads, C++ threads) and OpenMP in the upcoming blogs. Without further ado… Flynn’s Taxonomy In 1966 [ ], computer scientist Dr. Michael J. Flynn proposed a classification of computer architectures based on their capability to process the number of instruction streams and data streams concurrently. This classification is still relevant and known as Flynn’s taxonomy. 1 In his initial paper, Flynn proposed four classifications: Single instruction stream, single data stream (SISD) Single instruction stream, multiple data streams (SIMD) Multiple instruction streams, single data stream (MISD) Multiple instruction streams, multiple data streams (MIMD) Flynn published three further classifications of SIMD in a subsequent paper published in 1972[ ]: , , and . 2 Array Processors Pipelined processors Associative Processors “Stream” refers to the sequence of data or instructions as seen by the machine during the execution of a program. [ ] 1 A single stream issues one element in each time unit, resembling a queue. Conversely, multiple streams typically issue numerous elements per time unit, akin to managing multiple queues simultaneously. Consequently, a single instruction stream (SI) issues only one instruction within a time unit, whereas a multiple instruction stream (MI) issues multiple instructions during the same period. Similarly, a single data stream (SD) issues a single data element per time unit, while a multiple data stream (MD) issues multiple data elements within the same time frame. Flynn’s taxonomy can be visually represented as: Before proceeding, it's important to recall how instructions are executed (the so-called instruction cycle). The instruction cycle consists of following steps: Fetch instruction - read the instruction from the memory. Decode instruction - decode to interpret the requested action. Fetch Data (if needed) - get the data for processing the instruction. Execute instruction (process data) - execute the instruction/process the data. Write-back data - save/store the results. Let’s try to understand what these different classifications mean. Single instruction stream, single data stream (SISD) SISD represents a computing architecture where a uniprocessor machine executes a single instruction stream, operating on a single data stream. This architectural design corresponds with the principles of the von Neumann model. The control unit retrieves a sequence of instructions from memory, decodes them, and dispatches them to the processing unit, which handles the data fetched from memory one unit at a time. Consequently, only one instruction is executed at any given time, reflecting a sequential execution of instructions. It is crucial to note that SISD processors may exhibit limited concurrency. For instance, the execution phases of sequential instructions can overlap, facilitated by an additional functional unit. SISD processors achieve concurrent processing through techniques like * *or . pipelined superscalar processors Pipelined processors involve executing multiple instructions in the same execution unit concurrently by segmenting the execution unit into different phases, while superscalar processors execute multiple instructions in parallel using multiple execution units. Before the 2000s, most single-core processors adhered to the SISD architecture, encapsulating the foundational structure of early computing systems. Single instruction stream, multiple data streams (SIMD) In a SIMD system, a single instruction stream is broadcast to all the processing elements (PE), and each element independently processes its assigned portion of the data. This allows for efficient parallelization of tasks and can result in significant speedup for specific computations. SIMD is well-suited for tasks involving performing the same operation on a large data set in parallel, such as graphics processing, scientific simulations, or certain mathematical computations. An instruction is supplied to numerous data elements concurrently, following a synchronized execution pattern. All instructions are queued in this “lockstep” process, and the associated data is distributed across distinct compute units. Each compute unit executes the initial instruction in the queue simultaneously, proceeds to execute the subsequent instruction in sync, and continues this synchronized execution for following instructions in the queue. SIMD architecture finds its most prevalent application in Graphics Processing Units (GPUs). In 1972, Flynn published another research paper classifying SIMD into three further classifications: Array Processors (SIMT) An array (or vector) processor executes an instruction set optimized for efficiently processing large one-dimensional arrays called vectors. In this subtype of SIMD, each parallel processing element has its distinct memory and register file. However, they receive the same instructions. Nvidia’s concept, implemented in GPGPUs, is the most common example of array processors. SIMT combines SIMD with multi-threading and is the basis of most modern GPUs. SIMT (Single Instruction, Multiple Threads) Pipelined Processors (packed SIMD) Just like other SIMD variants, pipelined processors receive the same instructions. However, they read data from a central resource and process fragments of that data independently. Finally, they write back the results to the same central resource. Modern SIMD variations, such as Intel’s AVX or ARM’s NEON, use register files as the shared resource (instead of main memory). This register-based SIMD is commonly called , underscoring its efficient use of register files for parallel computation. packed SIMD Associative Processors (predicated/masked SIMD) Each parallel processing unit receives the same instruction, but the decision to execute or skip the instruction is made independently based on local data within the unit. Modern terminology characterizes this as “ ” or “ ” SIMD, showcasing the ability to conditionally execute operations based on specific criteria within individual processing units. Modern SIMD ISAs such as AVX-512 and GPGPUs use predication to improve cache usage efficiency and ensure effective pipelining. predicated masked Most modern GPUs utilize a combination of these methods, most commonly SIMT with predication. Multiple instruction streams, single data stream (MISD) In MISD systems, multiple instruction streams operate concurrently on a single data set. This type of architecture is less common than other categories in Flynn’s Taxonomy and is often considered more theoretical than practical. In an MISD system, different processors or processing elements receive distinct instructions, but they all operate on the same data stream. Each processor performs a unique computation on the shared data stream based on its assigned set of instructions. The results from each processor can be compared or combined to achieve a final output. One possible use case of MISD systems is creating fault-tolerant systems or specialized applications where redundant computations are performed simultaneously for error checking or reliability. For instance, some earlier MISD systems were designed for fault-tolerant space mission systems. Multiple instruction streams, multiple data streams (MIMD) In MIMD systems, multiple processors or processing elements operate independently, each with its own set of instructions and data. It is well-suited for applications that require parallelism on different tasks or where the workload can be divided into independent subtasks. MIMD systems are highly flexible and represent the most widely used class of computer architecture. Depending on how MIMD processors access the memory, they are classified into two categories: . Shared and distributed memory Examples Let’s try to understand these classifications using a simple example. Suppose we have two 1-D vectors, and , each having an equal number of single-precision floating-point numbers (floats). A B We have to multiply all the elements in A with a scalar and then add them to every corresponding element in B. The result must be stored in a third array called . c result Thus, we have: Vector A: [a1, a2, a3, ...]
Vector B: [b1, b2, b3, ...]
Scalar: c

result: [c * a1 + b1, c * a2 + b2, c * a3 + b3, ...] SISD For the given task, scalar addition and multiplication would involve processing each element of the vectors sequentially. The processor performs the addition of corresponding elements from the two vectors, followed by the multiplication with a scalar, and stores the results in a third vector. The operations are carried out one at a time. For example, iterating through every iteration of this for loop without any optimizations would be the SISD way of getting the vector. result for(int i = 0;  i < n; ++i){
  result[i] = c* A[i] + B[i];
} SIMD Suppose our machine supports Advanced Vector Extensions (AVX) SIMD instructions. Thus, a single SIMD register (YMM) can store 8 single-precision floats. Therefore, each element-wise operation is performed in parallel while using SIMD, enhancing efficiency. The same scalar addition and multiplication are applied to multiple elements concurrently. For example, let us assume we have n elements in A and B. We can perform 8 data operations simultaneously, significantly improving our efficiency. Only n%8 iterations will be done sequentially. In fact, those n%8 elements can also be efficiently performed using mask registers. For example, a C++ code using SIMD intrinsics can really do wonders. // SIMD code (with intrinsics)
int ub = n - (n % 8); // 8 floats per SIMD register
__m256 x, y, z, s, temp; // vectors of 8 float entries
s = _mm256_set1_ps(c); // store the scalar c
for(int i = 0;  i < ub; i += 8){ // notice the increment of 8
  x = _mm256_loadu_ps(&A[i]); // load 8 elements of A in x
  y = _mm256_loadu_ps(&B[i]); // load 8 elements of B in y
  z = _mm256_loadu_ps(&result[i]); // load 8 elements of result in z
  temp = _mm256_mul_ps(s, x); // multiply elements of A with s and store in temp
  z = _mm256_add_ps(temp, y); // add elements of temp with y and store in z
  _mm256_storeu_ps(&result[i], z); // copy the elements to the result vector
} 
 /* Reminder loop for remaining iterations
  * Masked registers can also be used instead of this
*/
for(int i = ub;  i < n; ++i){
  y[i] = a * x[i] + y[i];
} Don’t worry if all the SIMD jargon feels a bit heavy. SIMD intrinsics deserve an article of their own, and I’ll cover them in a future article. offers detailed documentation on how to use different intrinsics. Intel intrinsics guide Modern compilers also enable vectorization if the user gives relevant optimization flags. Feel free to check my article on to learn more about such compiler flags. Compiler Optimizations MIMD In MIMD, multiple processors operate independently, each executing its instructions on its data set. In the given task, different processors might perform various operations on different elements of the vectors simultaneously. Depending on the shared or distributed memory model, several ways exist to solve the given task. For instance, if working with a shared memory programming model like OpenMP, different threads may work together and distribute the loop iterations (See__ __). OpenMP scheduling Similarly, for a distributed memory programming model like MPI, the vectors may be distributed to different processors such that each calculates their respective chunk of result. One of the processors may work as a coordinator, distribute the data, and even collect the final result from all the processors. For example, if p processors exist, the data may be divided into chunks. n/p // Each processor works on n/p data elements
Processor 1: [c * a1 + b1, ....] -> Result1
Processor 2: [c * ae + be, ....] -> Result2 // e = n/p +1
Processor 3: [c * af + bf, ....] -> Result3 // f = 2*n/p + 1 What next? This brings us to the end of the first part of this blog series on multi-threading. In the next part, we’ll dig deep into SIMD and MIMD systems, mainly introducing the programming models used to exploit them. References [1] Very high-speed computing systems | IEEE Journals & Magazine | IEEE Xplore [2] Some Computer Organizations and Their Effectiveness | IEEE Journals & Magazine | IEEE Xplore [3] Intro to Parallelism with Flynn’s Taxonomy (youtube.com) Also published here. Featured Photo by on . JACQUELINE BRANDWAYN Unsplash

Flynn’s Taxonomy and the Concept of Multithreading

When I was little, programming was simple. My friend had a computer and there were Basic and Assembly. You could write your program in Basic, which was easier to do but your program would be slow, or you could write something in Assembly, which was harder, but your program would run significantly faster. The explanation for this was simple too. Basic was an interpreter, in order to run your program, it had to go through your code every time you invoke it, and interpret it line-by-line. If it says "PRINT X", the interpreter had to find a variable named “X”, find a routine that does printing, and call the found routine for the found variable. Assembly was, well, assembly. It did interpret your program in a way too, but only once when you run the assembler. After that, your program would be executable without interpretation. Programs that need interpretation run slower than programs that don't need interpretation. If, of course, they are equivalent programs. And in Basic and Assembly, they usually were. Basic is an imperative language, not even too friendly to structural programming. Even a thing as staple as a "function" is not a built-in language construct there but a pattern: "GOSUB ... RETURN", pretty much like "call ... ret" in Assembly. Now fast forward 30 years. Languages are plenty. Computers are everywhere. Programming is not simple anymore. My department earns its bread by rewriting researcher's code originally written in Python in C++ for performance because common knowledge is that Python is interpreted and slow, and C++ is compiled and fast. But somehow, every year this rewrite gets harder and harder to win any performance from. Something changes and changes fast. The common knowledge however doesn't change so we keep rewriting. But now we're forced to optimize everything like crazy just to justify what we do. An algorithm comes in in Python, we rewrite it in C++ equivalently, and suddenly it becomes 3 times slower. That's... not what we're here for. So we reengineer the algorithm to push and get the performance boost we promised. And most of the time, since researchers don't care about performance at all and algorithmic-wise they do leave some low-hanging fruits behind, this works. Still, this whole business now looks like a scam. We're making code slower by rewriting it in C++ just so we could make it faster by reengineering the code. Why don't we re-engineer it directly in Python then? Ah! The thing is, we don't know Python. We know a little Python, enough to read and understand, but not enough to make ultra-fast programs in it. So what's there to know? Libraries Most of the Python libraries are written in C or Fortran. core is written in C; - in Cython and C; - in Fortran, C, and partially C++. They have no reasons to be slower than whatever was written in C++, Rust, or Julia. They can be faster though. In our company, we cater to both cloud services and desktop applications. And desktop users are getting angry when the new version of their favorite application stops working on their hardware for apparently no reason. So we keep our desktop builds target old. Really old, like pre-Nehalem old. This way, nobody gets angry but nobody gets to enjoy SSE3 either. NumPy Pandas SciPy Of course, a computational library built for a proper target will be generally faster than an equivalent library built for a generic 15-year-old computer with limited superscalar capabilities. The good news is if you're building for a cloud, you can set your target build platform to be exactly the machine you procure, and then your C++ libraries will run at the same speed as Python ones and maybe even a little bit faster. Compilers To be honest, the whole argument about which language is faster is ridiculous. A language is not a compiler or an interpreter, it is what it is – a language: a set of rules that specify how we should tell a computer what we want if to do. A language is just a set of rules, a specification. And nothing else. The very distinction between interpretation and compilation is something from the last century. Nowadays, there are C interpreters like , , or , and there are Python compilers. JIT compilers such as [PyPy] and classic compile-before-you-run compilers such as (that also has JIT capability if you only want part of your code to be compiled). IGCC PicoC CCons Codon Codon is built upon LLVM, the same infrastructure Rust, Julia, or Clang are built upon. The code built with Codon runs, give or take, at the same performance levels as built with any of those. There might be performance disadvantages due to Python’s garbage collection or large native data types but we’re not talking about 100x or 10x anymore. LLVM does its magic. it turns Python code into machine code for you. There are also myths about just-in-time compilation or JIT. Some say that it is superior to the usual compile-before-you-run technique because it always compiles for the architecture users have, and thus exploits it optimally. Some say that there is still compilation overhead that with just-in-time compilation also falls on users. This makes programs run slowly since they have to both run and compile themselves at the same time. The problem with both myths is that they are both true and both unhelpful. Yes, JIT generally compiles to a better machine code unless you build your binaries explicitly for the target machine, which, by the way, happens quite regularly when you deploy in a cloud. And yes, there is a compilation penalty in runtime, but it is negligible if your runtime is measured in months, which, again, when you deploy in a cloud, is not something unheard of. So there are pros and cons. What’s important, Python (Codon specifically) supports both compile-before-you-run and JIT modes so you can choose what will suit your needs the most. Traditional compilers, such as Clang, do not have the JIT option. Numba and its kernel model Speaking of JIT, is probably the most game-changing technology in the world of ultra-fast Python programming. It is a compiler but it only targets selected kernels, not the whole program. You, of course, get to select what should be compiled and for which platform. In this setup, you can run pieces of your code on the CPU, and others - on . Numba GPGPU Technically, one can create backends for other specialized devices such as Google’s or even Lightmatters’ . There is no such backend yet, those guys decided to roll out their own library instead. But, what’s symptomatic, they also choose to provide the interface for the photonic computer in Python so you could interact with Pytorch, Tensorflow, or ONNX seamlessly. TPU photonic accelerator So Lightmatter is not yet there. But NVidia is. They did provide their CUDA backend for Numba and you can now write kernels in Python, and run them on NVidia hardware with maximum efficiency. C++ doesn’t have that. There is, however, a CU dialect, coming from , of course, that extends C++ for this very matter. In Python, you don’t have to extend the language itself. Since Numba works as a JIT kernel compiler, adding a backend is just a matter of patching a library. NVidia So, the kernel model is targeting heterogeneous computing. You can run pieces of code on different devices, which is nice by itself. But there is another dimension of heterogeneity you might have not thought about. With the kernel model, you can target different kernels for different computation contexts and not necessarily hardware devices. This means that if you want one kernel to be fast but not particularly precise, you can build it with a “-fast-math” option. But if, in some other context, you want that kernel to be precise rather than fast, you can rebuild the very same code without the trade-off. This is something that is hard to achieve with traditional compilers where you can’t change compilation options in the middle of a translation unit. Well, with the kernel model, every kernel is its own translation unit. Conclusion Python is not slow. Neither it is fast. It is just a language, a set of rules and keywords. But there are lots of people who got used to these rules and these keywords. They feel comfortable writing in Python and they are interested in making Python better for them. This user base is large enough to attract both new startups with revolutionary technologies such as Lightmatter and their photonic computers, and well-established companies with decades of expertise in high-performance computing such as NVidia. All these people are vastly invested in making Python a better... not language, of course, but environment. The environment in which writing ultra-fast programs is just marginally harder than writing slow throwaways. Altogether they are making huge progress. Python is getting faster every year. At this point, forget about photonic computers if you can, programs written in Python often run on par with the ones written in Julia, C++, or Rust. But Python wouldn’t stop there. It is getting faster than traditional compilers are getting user-friendlier. Be prepared to see that in a few years Python compilers: PyPy, Numba, or something completely new, will borrow techniques from or to generate code so much more effectively that no traditional compiler could possibly come close. After all, writing a new JIT backend in Python is way easier than reimagining the whole LLVM infrastructure. Spiral Herbie

“Python Is Slow” and Other Myths of a Dying Era

“Python es lento” y otros mitos de una era moribunda

"Python est lent" et autres mythes d'une époque mourante

"पायथन धीमा है" और मरने वाले युग के अन्य मिथक

「Python は遅い」など、滅びゆく時代の神話

“Python é lento” e outros mitos de uma era agonizante

“Trăn chậm chạp” và những huyền thoại khác về một kỷ nguyên đang hấp hối

“Python 很慢”和垂死时代的其他神话

📡 Radar for the crypto market Dappradar It is an application that allows you to keep track of such things as viewing statistics and details of different , viewing ratings and statistics of projects, viewing statistics of different collections on different blockchains, tracking your or someone else’s NFT or cryptocurrency in 3 networks (Ethereum, BNB Chain, Polygon), tracking cryptocurrency statistics and many more statistics and ratings in all areas from crypto games to trading platforms. dapps DeFi NFT portfolio So I think this is one of the most useful tools for researching different projects + most of the functionality that you may need is completely free. ⛽ Websites to know the price of gas - are two of my favorite sites that I use to find out the price of gas, these sites will tell you how much it will cost you to approve NFT, wrap ETH -> WETH, create a collection, etc. , gwei blocknative 👛 Transaction tracking - is one of the best trackers for your wallets, you can add all your wallets in one place and track changes in these wallets + they can be divided into groups, for example, create two groups the first will be a group of wallets from different influencers, and in the second group all your wallets and you can not track each wallet separately, and all the wallets together and will see the total balance and more. Another big plus of this tool is that it has an app on your . Zapper phone 📦Spread ETH or tokens between wallets - is a site that will help you distribute eth or tokens quickly to your different wallets. Multisend 🧾 Get data for tax filing - is a website where you can get a record of your transactions for your tax filing. Nftbank 🔓 View when tokens will be opened - is a site where you can see when some project’s tokens will be unlocked, and how many percent of the total supply it will be, and you can use this information to your benefit. Token unlocks 📊 A lot of accurate and interesting statistics - is a site with a very large number of detailed statistics on a variety of projects in the crypto space, a very useful tool if you know how to use it. Dune 🔥 See the hype around the NFT collection - is a site where people vote on which NFT collection they would like to own and this statistic shows if a project you want to mint is desirable and if someone will buy it on the secondary market. Mintyscore Of course, these stats don’t have a 100% chance to come true, but the higher the rating the project has on this site, the faster it usually sells out and the more people are interested in buying it on the secondary market. 👁️ Tracking activity in the NFT market — this site will allow you to keep track of such things as the sharp change in price or in the NFT collections, that are actively buying (for this there is an additional tool ), in which collections buy , what buy with large balances and of course track the wallets you need and their activity. Moby activity mints whatsminting many at once NFT wallets 🔎 Finding upcoming collections with good activity - is a site where you can look at upcoming mints with a large audience, explore these projects, and maybe find a future collection you would like to mint. Nftgo 💎 Find the best offer from the collection - is a tool that will allow you to make a better choice when buying NFTs from any collection, you go to Rarity sniper, find a collection where you want to buy NFTs, and set BUY NOW and now you can see the best deals by rarity. Rarity sniper 🛠 The best extension for working with OpenSea - is, in my opinion, the top 1 free extension that you should download if you use OpenSea. It will give you such features as total inventory cost calculation on OpenSea, and the ability to see all NFTs before anyone else after revealing (this is made by the fact that the normal user has to wait for the OpenSea cache update, but this extension takes pictures for NFTs from the source), warn you if the collection is trying to cheat you, mass-queue items for OpenSea metadata refresh after a reveal, View real-time activity. SuperSea https://twitter.com/subliminoid/status/1439280006945988611 https://twitter.com/subliminoid/status/1450451179377504258 https://twitter.com/subliminoid/status/1439672750730354693 https://twitter.com/subliminoid/status/1450451150050930694 https://twitter.com/SuperSeaNFT/status/1513804908557971459 There are of course also paid features such as Notifications of the price you want, quick buy, mass offer sending, and color highlighting of different rarities. https://twitter.com/SuperSeaNFT/status/1532330609389015041 https://twitter.com/subliminoid/status/1450451116173500425 https://twitter.com/subliminoid/status/1490656856733372417 🧭 Top NFT aggregators - two top aggregators for NFT on Ethereum, they will allow you to see all NFT collections from different trading platforms, this is very useful because usually NFT collections are selling at once on very many platforms and this creates a problem that if you need to find some good price you need to sit in several tabs and update different pages and compare, but here sites have collected everything in one place with comfortable filters and fast buy, It’s a hell of waste not using such tools. , Gem Genie 🔄 Useful site for the NFT flip - is a site that will give you statistics such as the number of NFTs for sale and removed from sale, price changes, changes in the number of owners, percentage of NFTs for sale, and some other statistics that may be useful. flips finance 🪜 Help me grow up Thank you all for your attention and I hope you like this compilation, I am waiting for comments and feedback (most likely this article will be updated as I find new tools). Also published . here

Top 17 Useful Tools for Ethereum

You probably have heard of the idea of out-of-order execution, and that it breaks some common-sense assumptions about writing programs, and software has to insert barriers to make things right. That was a baffling concept to me at least when I heard of it. At the very least, What's the point? why would the hardware want to execute out of order only to have the software correct the behavior back? Is the software taking care of all the quirks for me?  And if not, where's the catch? Well,  all this conundrum comes from the evolution of processor design to create parallelism and the software efforts to harness the power while reining in the unwanted impact. Let's start with a tiny bit of history. Sequential consistency on uniprocessor Going back to the good old days when your CPU was just single-issue (meaning executing one instruction at a time), and there's no multi-threading programming to speak of, the programming model is straight forward -- there's memory and a stream of instructions executing in program order, doing reading/writing to the memory. For as long as the program lives, memory will reliably hold results written to it, and  reproduce the same values when read afterwards. That is, a read after a write to the same memory location will faithfully return the value written. A write after a write to the same memory location will overwrite the old value with the new. To illustrate, look at this simplistic : example1 { i = ; :: << i << ; j = ; :: << i << ; :: << j << ; i = ; :: << i << ; :: << j << ; ; } 1 # include <iostream> 2 3 int main () 4 int 1 5 std cout ' ' 6 int 5 7 std cout ' ' 8 std cout ' ' 9 2 10 std cout ' ' 11 std cout '\n' 12 return 0 13 Easy to predict, the result will be . At first variable assumes the value . Up until , where the value's updated, no matter how many times it's read the location will always return the same value. Right after 2 is written to it, reading that variable will invariably return . 1 1 5 2 5 i 1 line 9 2 There's also the variable . Since it's a different location, what value is written to it doesn't concern the output of whatsoever. j i Now let's look at a 2-threaded example to add a bit of nondeterminism. Keep in mind we're still executing on a single CPU. Imagine we have two functions and executed by two separate threads, in the following : func0 func1 example2 i = ; j = ; output[ ]; {
  i = ;
  output[ ] = j;
} {
  j = ;
  output[ ] = i;
} int 0 int 0 int 2 void func0 () 1 1 void func1 () 1 0 After the program finishes what are the valid outputs? There are roughly four instructions here. Two reads and two writes. Depending on how they are mixed, and are both likely -- if one function happens to execute before another then these will ensue. is possible too, so long as the reads are both performed after the writes are done. 01 10 11 You probably have noticed is not possible. No matter how you execute, the first instruction will be writing a to either of the output elements. 00 1 When you make this conclusion, you are of course intuitively making assumptions that the programming model is . To summarize, this means that even though the instructions from different threads may be interleaved in a nondeterministic way, each thread's execution is in program order. Also the operations are not overlapped. One memory access is done before another begins. sequential consistency Over the years, processor design has improved drastically, and computer architects play all kinds of tricks to extract parallelism from executions. For example, here in our example you may notice that in , the two operations can be performed in parallel, so long as they both execute before . Given a possible instruction sequence, we'll be free to reorder the instructions to maintain the impression that the program still runs with sequential consistency, as long as we maintain the order of memory operations to the same variable! func0 func1 Computer architects actually went ahead and took advantage of this to implement that can execute multiple instructions at a time. They tracked the operation dependencies using algorithms like and . In the history of processor you will see the architects playing nasty and nastier tricks, such as speculative execution, which caused a ton of now in the cloud business. superscalar processors Scoreboarding Tomasulo trouble But I have digressed here. The point is no matter how the system evolves, has been the contract between programmers and system builders. sequential consistency de facto Consistency model on multi-processors In the early to mid 2000's, the idea of SMP (symmetric multiprocessing) really caught on. The reason was that computer architects found it more and more difficult to dissipate the amount of heat generated by a single powerful processor. Using the same die area, they'd rather build multiple smaller "cores". However, like the uniprocessor days, in each core, instructions may still be executed out-of-order. The read-write dependencies are still tracked by the same algorithms so as to maintain the illusion that it's as if the program was still executed in-order. However, this dependency tracking won't be available across cores. Now looking back on our : example2 i = ; j = ; output[ ]; {
  i = ;
  output[ ] = j;
} {
  j = ;
  output[ ] = i;
} int 0 int 0 int 2 void func0 () 1 1 void func1 () 1 0 Suppose the two functions are executed on two separate processors, processor-0 will be free to think is all it's executing. So it's possible for this processor to read the and write the output first. Processor-1 is in the same boat! So under this paradigm, the output is actually possible! func0 j 00 To illustrate this, let's write a full-fledged program that can reproduce this: i; j; output[ ]; ::atomic< > ready; { (!ready.load()) {
  }

  i = ;
  output[ ] = j;
} { (!ready.load()) {
  }

  j = ;
  output[ ] = i;
} {
  i = ;
  j = ;
  ready.store( ); :: ; :: ; ::this_thread::sleep_for( ::chrono::milliseconds( ));
  ready.store( );

  t0.join();
  t1.join(); :: << output[ ] << output[ ] << :: ;
} { (;;)
    run_test();
} # include <atomic> # include <iostream> # include <thread> int int int 2 std bool void func0 () while 1 1 void func1 () while 1 0 void run_test () 0 0 false std thread t0 (func0) std thread t1 (func1) std std 5 true std cout 0 1 std endl int main () for Don't mind the boiler plates for now. The sleep in and the atomic are to make the two threads start roughly at the same time. run_test ready On my x86 processor, is not only possible, it's the dominant output (perhaps because reading is cheaper than writing with cache coherency protocols, which we will cover in later posts). In fact and combined only account for about 25% of the output. 00 01 10 For some reason is really rare, next to nonexistent, but that is just because of the timing of the execution. If you tell both threads to yield between the read and the write, then you will see most of the time. 11 11 Also, to illustrate this won't be a problem on uniprocessors, we can pin the two threads onto the same core by adding the following snippet right after you create the threads. Afterwards you will see output completely disappear. 00 cpuset;
  CPU_ZERO(&cpuset);
  CPU_SET( , &cpuset);
  pthread_setaffinity_np(t0.native_handle(), ( ), &cpuset);
  pthread_setaffinity_np(t1.native_handle(), ( ), &cpuset); cpu_set_t 0 sizeof cpu_set_t sizeof cpu_set_t How do we fix the broken consistency I don't know about you, but when I first saw this result my mind was blown. I knew some RISC processors can go aggressive on relaxing memory models, but I had always thought my old buddy x86 is a dependable fellow. X86 in fact, implements a memory model called TSO, or total store ordering. As the name suggests, the stores (writes) to all locations are globally ordered. More on the relaxed memory model later, but the tl;dr here is that you can imagine TSO places write results in a buffer called LSQ, or load-store-queue, before they go out to the cache. Each read afterwards will in turn check this queue for an early hit, before they themselves go out to the cache. The reason is that cache access takes much longer than register access, and it's valuable to keep a buffering structure on-core. But of course the consequence is that your read may bypass previous writes to different locations, as long as you can find a result in the LSQ, sometimes breaking sequential consistency. The fix is to enforce sequential consistency by removing the reordering. C++ gives you a standard way of doing this. Instead of declaring and as plain integers, we'll wrap them around a type, and the reads and writes will be through the and methods. i j std::atomic load() store() By default these methods will , even when these variables are accessed across different processors. So our new program will look like this: enforce sequential consistency ::atomic< > i; ::atomic< > j; output[ ]; ::atomic< > ready; { (!ready.load()) {
  }

  i.store( );
  output[ ] = j.load();
} { (!ready.load()) {
  }

  j.store( );
  output[ ] = i.load();
} {
  i.store( );
  j.store( );
  ready.store( ); :: ; :: ; ::this_thread::sleep_for( ::chrono::milliseconds( ));
  ready.store( );

  t0.join();
  t1.join(); :: << output[ ] << output[ ] << :: ;
} { (;;)
    run_test();
} # include <atomic> # include <iostream> # include <thread> std int std int int 2 std bool void func0 () while 1 1 void func1 () while 1 0 void run_test () 0 0 false std thread t0 (func0) std thread t1 (func1) std std 5 true std cout 0 1 std endl int main () for As promised before, we have now converted and into atomics. And voilà, the pattern is eliminated even if we're running on 2 cores, fully bringing us to the sequential consistency land! i j 00 Underlying the hood, the compiler inserts memory barriers so that the processors will coordinate to observe a global order on memory operations. Later on we'll get a chance to look at how the barriers are actually inserted at the assembly level. This enforcement is quite slow. If all variables are protected by memory barriers, it kind of defeats the purpose of having multiple cores. But for now, we see it solves the problem of correctness. Recapitulation and what's to come Here we have covered: Sequential consistency is the programming model computer systems strive to deliver. It's always been maintained on uniprocessor programming. When programming on multiple processors, at times programmers need to explicitly enforce sequential consistency on their own. There are language constructs programmers can take advantage of. In the next episode, we'll dig deeper into some of the more relaxed memory models and how they are supposed to be used to maintain the illusion of sequential consistency and at the same time produce as much parallelism between multiple cores as possible. Disclaimer : Compilation in this article is done with clang++ 10, with compiler flags "-O2 -std=c++20 -lpthread". YMMV as to the execution results.

Everything You Need to Know About Multithreading: Sequential Consistency Concept [Part 1]

If you’re new to the world of IoT, chances are you’ve heard a number of terms and acronyms thrown around and wondered about their differences or whether you should purchase a Raspberry Pi or an ODROID. That main reason I became curious about how interchangeably these boards were being used was because some of the main intro to embedded systems labs at MIT (6.08, 6.111, 6.115) teach , PSoCs, and FPGAs. programming microcontrollers I first started hearing about the different types of boards around 2012, around the time Arduino and Raspberry Pi were becoming popular. I’ve mostly used microcontroller development boards due to their ease of integration into smaller projects (usually small IoT devices won’t require high computing power or peripherals like RAM). Another thing that I didn’t realize at first was that the terms referred to the integrated circuits (ICs), and that a development board had to be built around the IC to make them useable or to interface external components. Microcontroller (MCU) Atmel ATMEGA32U4-AU AVR Microcontroller from Tindie A small computer on a single integrated circuit. They are generally less complicated that SoCs (a SoC can have a microcontroller as one of its components). They contain at least one CPU (processor core), memory (RAM, flash, ROM), and programmable I/O peripherals. They are designed for embedded applications, as opposed to microprocessors used in personal computers or other chips used for more general purpose applications. Many IoT devices that are automatically controlled (using some form of a feedback controls system) use microcontrollers. Their main advantage is that they reduce the size and cost of a similar system using separate microprocessors/memory/input/output devices. They can be used for data collection, sensing, actuating physical devices, and low-energy applications, among other uses. Some common microcontrollers at the Atmel AT89, MSP430 from Texas Instrument, and ESP32/ESP8266. You can usually find development boards that include a microcontroller (such as the Arduino Uno, Intel Galileo, TI Launchpad, or BeagleBone Black), thus removing the necessity of designing a PCB for the microcontroller. Microprocessor Intel Core i7 microprocessor A microprocessor is a device that incorporates the functions of a CPU on a single IC (or a few). It is a clock-driven, register-based digital integrated circuit that accepts binary data as input, processes it according to an instruction set stored in its memory, and provides output. Different instrument sets determine the different architectures of microprocessors (AMD x86 and x86–64, PowerPC, ARM). More common architectures today are generally 32-bit or 64-bit, with some reduced instruction set computing (RISC) microprocessors that — as expected — have reduced instruction sets. There are other devices that are multi-core (Intel Xeon processors, AMD Phenom II). Historically, AMD and Intel are known as the companies with the fastest CPUs with their dual-core and quad-core CPUs. Nowadays, CPUs are almost all implemented on microprocessors, causing the two terms to be practically interchangeable. When comparing the microprocessors in laptops, you generally consider the number of cores, the clock speed, the cache size (like RAM, but the memory is stored on the CPU itself so it’s faster), and other aspects like hyperthreading (running multiple threads simultaneously from a single CPU, which is something you can do nowadays) or dynamically increasing clock speed when the need arises. System on a Chip (SoC) AMD Am286ZX/LX, SoC based on Intel 80286 (from Wikipedia) An integrated circuit with all the components of a computer, including a CPU, memory, I/O ports, and secondary storage. They consume less power and take up less space than a multi-chip design, and are common in embedded systems. Compared to the motherboard-based architecture, the CPU, graphics, hard-disk, USB-connectivity, ROM/RAM, and memory interfaces are all integrated (a motherboard would connect all of the components separately or as expansion cards). SoCs are usually built around a microcontroller, built around a microprocessor (found in mobile phones), designed for a specific application, or designed as a programmable SoC with some reprogrammable aspects, similar to FPGAs. Compared to microcontrollers, SoCs usually have far more pins and more systems integration of different peripherals. Note that system-on-a-chip can refer to many things on the market, and it generally refers to a single chip that does everything that used to take up multiple chips (some still require peripheral such as RAM or flash, so the name may be misleading). Some common ones are the Cypress Semiconductor Programmable System on a Chip (PSoC) and Altera System on a Programmable Chip (SOPC). On that topic, PSoC solely refers to the family of microcontroller-integrated SoCs produced by Cypress Semiconductor, although there are other programmable SoCs that exist. has some useful knowledge on current conventions within SoCs. This StackExchange thread Field-programmable gate array (FPGA) Xilinx Spartan FPGA from Wikipedia An FPGA is an integrated circuit that can be configured by a consumer after manufacture (“field-programmable”). Its configuration is specified using a hardware description language (HDL). They contain programmable logic blocks (like AND and XOR) and a hierarchy of reconfigurable interconnects that “wire together” the blocks. These blocks can also contain memory elements (flip-flops, etc.) The biggest companies for delivering FPGAs include Altera and Xilinx. Compared to microcontrollers, they are faster, parallelizable, and have more design flexibility (trading companies often hire FPGA programmers to program their trading strategies in order to beat out competing firms). They are generally more expensive and more difficult to use, but contain the same ICs, logic components, and programmability. Common FPGAs are the Xilinx Spartan and Virtex Series and the Altera Stratix and Cyclone Series. (More info at ) this StackExchange thread Complex Programmable Logic Device (CPLD) An Altera MAX 7000-series CPLD with 2500 gates from Wikipedia A programmable logic device that’s slightly less complex than an FPGA. It primarily consists of a macrocell with logic expressions and operations. It doesn’t require an external configuration ROM (needed by FPGAs) and has less internal state storage and layered logic than FPGAs. CPLDs have thousands to tens of thousands of logic gates, while FPGAs can have up to several millions. They are primarily manufactured by Altera, Atmel, Cypress Semiconductor, Lattice Semiconductor, or Xilinx, and are programmed in VHDL, Verilog HDL, or Standard Test and Programming Language (JAM/STAPL). Programmable Logic Device (PLD) This is a more generic terms that refers to an electronic component used to build reconfigurable digital circuits — they differ from logic gates, which have a fixed function, as PLDs don’t have functions after being manufactured. Different types of PLDs (in approximate order of complexity from least to most) include the programmable logic array (PLA), programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), field-programmable gate array (FPGA), and other variants. Some of the terms used to describe PLDs are trademarked by specific companies (PAL was owned by Monolithic Memories, Inc. (MMI) and currently held by Lattice Semiconductor, for example). Low-complexity devices are programmed in PALASM, ABEL, or CUPL, while higher-complexity devices are programmed in the higher level Verilog and VHDL languages. Digital Signal Processor (DSP) Texas Instruments DSP processor from Analog Devices A specialized microprocessor optimized for digital signal processing. While other microprocessors may have general functionality for executing digital signal processing algorithms, they aren’t usually optimized for continuous processing in real-time. DSPs usually have better power efficiency and are used in mobile phones or devices with power consumption constraints. The algorithms used require a large number of mathematical operations to be quickly performed (FIR filters, Fast Fourier transforms (FFTs), matrix operations). Instruction set architectures commonly used to increase parallelism in DSPs include SIMD, VLIW, and superscalar architecture. Integrated Circuit (IC) Erasable Programmable Read-Only Memory (EPROM) ICs from Wikipedia A set of electronic circuits integrated into one small flat chip of semiconductor material (usually silicon). This results in a chip that is orders of magnitude smaller than a circuit that would have to be built by hand using separate electronic components. These chips are usually mass-produced, and are used in tons of electronics. ICs can be classified into analog, digital, and mixed signal ICs. Digital integrated circuits can contain logic gates, flip-flops, multiplexors, and other circuits, and are usually produced as MCUs, DSPs, or microprocessors. The most advanced ICs are generally microprocessors (“cores”). They can also be classified as logic ICs, memory chips, interface ICs, power management ICs, and programmable devices. Analog ICs can include sensors, power management devices, RF chips, and operational amplifiers (op-amps), which process continuous signals and perform functions such as amplification, active filtering, mixing, and demodulation. They ease the burden on circuit designers by providing an efficiently designed analog circuit that can be difficult to design from scratch. Mixed-signal ICs can create functions such as analog-to-digital converters (ADC) or digital-to-analog converters (DAC), digital potentiometers, as well as clock/timing ICs. Application Specific Integrated Circuit (ASIC) Bitfury 55nm ASIC H-Board from Cryto Store An integrated circuit customized for a particular use (as the name implies). For example, some ASICs are designed for digital voice recording or parallel computing. In Bitcoin (or altcoin) mining, ASICs can be designed to solve specific hashing algorithms. Under some definitions, you can consider a GPU an ASIC, such as NVIDIA GPUs, which are meant for graphics computing. They perform better than CPUs in this specific task due to a compact design (less space and material used), power dissipation (battery life), and speed of computation. Usually the discourse between GPUs vs. ASICs is within the altcoin mining community, which usually discusses ASICs in the context of chips designed to solve a specific complex algorithm. Application-specific standard products (ASSPs) are between ASICs and industry-standard circuits. While ASICs combine a collection of functions designed for a single customer, ASSPs are used as off-the-shelf components. Their uses include USB interface chips, controller chips for a PC, and a chip for a modem. Electronic design automation (EDA) A simple PCB designed in EAGLE A category of software tools for designing electronic systems for integrated circuits (ICs) or printed circuit boards (PCBs). The tools create a design flow for chip designers to create entire semiconductor chips with billions of components. Their simulation software includes transistor, logic, behavior, hardware emulation, and field solvers. For design, they usually inplement schematic capture, layout, and logic synthesis. Analysis includes formal verification to prevent deadlock, equivalence checking, static time analysis for worst-case scenarios, physival verification to prevent physical defects and to check for physical manufacturability, and clock domain crossing verification (like linting) to check for issues in clock domains. Major companies producing EDAs are Altium (Altium Designer), Cadence Design Systems (Verilog, OrCAD), Autodesk (EAGLE), National Instruments (Multisim), and WestDev (Pulsonix). There is also KiCAD, an open-source EDA software, as well as many web-based EDA tools, some of which are web-based versions of the aforementioned EDAs. In terms of some of the differentiations between commonly purchased development boards, knowing the difference in the hardware makes purchasing a board a lot easier. If you’re just starting out, you usually don’t care too much about specifics, and you’ll want a board that’s easy to interface with pin-outs that make sense for whatever project you’re building (you’ll also want systems compatibility with whatever peripherals may be relevant to you). ODROID ODROID C-2 from Wikipedia A series of single-board computers capable of running Android and Linux distributions (the name comes from “open” + “Android”). The hardware features a SoC with a ARM CPU and an on-chip GPU. SD cards are used to store the operating system, and the outputs include USB 2.0, USB 3.0, HDMI, and a 3.5 mm jack. There are also lower-level output pins with general-purpose input/output (GPIO) pins supporting common protocols such as I2C. There are also some models with an 8P8C Ethernet port and an eMMC port. Common models are the ODROID-C1, ODROID-C2, and the ODROID-XU4. Raspberry Pi Raspberry Pi Zero from Wikipedia A series of single-board computers developed for the purpose of teaching basic computer science in schools. The models all featured a Broadcom SoC, with an ARM CPU and an on-chip GPU. SD cards are used to store the operating system, and the outputs include USB ports, HDMI ports, a 3.4 mm jack for audio, and lower-level GPIO pins supporting I2C and other protocols. There are also some models with an 8P8C Ethernet port and an eMMC port. Common models are the Model A, Model B, and Zero. Arduino Arduino Uno from Wikipedia An open-source hardware and software company and user community that produces single-board microcontrollers and microcontroller kits. Their hardware contains an Atmel 8-bit AVR microcontroller with flash memor and a single or double row of pins that can connect to add-on modules (“shields”) and can be addressable via a I2C serial bus. Most models are pre-programmed with a boot loader for uploading programs to the on-chip flash memory. Some of the I/O pins can produce pulse-width modulated signals and receive analog inputs. Common boards are the Arduino Pro, Arduino Mega, and Arduino Yun. As it turns out, there isn’t a ton of standardization in terms of how IoT systems are built and maintained — at least not universally. There are certainly standards that have been set over the past few decades with regards to the protocols and operating systems used that are optimized for smaller devices, but there’s enough variability that how you choose to build your system is dependent on what you value — CPU efficiency, memory, cost, etc. As for the devices themselves, there’s a reason why Arduino is so popular — it takes away all of the mess of having to choose a processor and figuring out layouts for a board and figuring out the programming environment and presents everything neatly packaged such that a 9-year-old child can program an Arduino to turn on an LED. Sticking with the familiar environment that Arduino presents removes a lot of headache, but it doesn’t allow you to explore the greater possibilities of integrating devices that aren’t in the Arduino ecosystem — which is very much a possibility. Sites like Digi-Key (and to varying degrees SparkFun, Adafruit, and Tindie) have a huge array of embedded devices and integrated circuits (ICs) far beyond the scope of Arduino. I’d recommend checking out similar sites if you’re working on a larger scope project. If you liked this post and want to hear more about my thoughts on hardware, computer science, or even life as a university student, feel free to follow me or find me elsewhere online! @sharontlin Twitter @sharonlinnyc Facebook @sharon-lin GitHub

FPGAs, SoCs, Microcontrollers— A Quick Rundown of IoT Devices

As published in VRScout on 4th September 2017 While we were distracted with VR hype cycles and the “when will VR go mainstream?” debate, the understated and overlooked snuck up on us. WebVR If you are wondering why you’ve started seeing more and more discussion of WebVR in the headlines, you’re not alone. This year alone, we’ve seen major announcements from and , as well as the continued growth of independent WebVR communities, such as JanusVR: Google Firefox This is just the beginning; there will be much, much more from here. The roots of WebVR actually started with the conceptualization of Virtual Markup Reality Language (VRML) which ultimately did not take off, but paved the way for the burgeoning VR web as we are coming to know it. 23 years ago What is WebVR? is a JavaScript API for rendering 3D graphics within a browser, meaning that, if you punched in a link in the browser for a 3D website, you would find yourself in a 3D space (whether on your computer browser or within a VR headset). In laymen’s terms, WebVR is a low-barrier entry point to develop VR for the web without having to be familiar with WebGL, which can be frustrating for beginners to grasp. WebVR In its ease of use and lack of friction for consumers, WebVR presents a glimmer of hope for the mainstream adoption of virtual reality (VR) and augmented reality (AR). While there is certainly a place for high-end experiences and hardware, not all VR needs to function as a “high-end” experience. In fact, for many use cases (things like going to basic websites), it’s much more important that these experiences be accessible than it is that they sport cutting-edge graphics. Ultimately, it’s these experiences that will drive mass-adoption of immersive technologies. The ‘Casual VR’ Phase Why and how? While we move through what some call the “trough of disillusionment,” for VR, WebVR — what we can call “Casual VR” — paves a way for consumers without state-of-the-art hardware to experience VR, through the web on multiple platforms. In fact, this could prove to be the best way to democratize VR in the long run. WebVR developers at a WebVR xLab. Copyright: Vanessa Radd. Web developers have been quietly developing and building this new Web for the past few years. The beauty of building without the hype is that developers get to focus. WebVR is a “code once, work on multi-platforms answer” to VR, hence it has proven itself to be a cost-efficient tool for developers to build quick VR or AR prototypes and launch products. Technology Agnostic “The importance of the term ‘ agnostic’ is something I have been stressing for many years, based on the early experiences of the 1990s [developing for VR] when we saw various organizations suffer — fail even — from standardizing on one rendering or modeling toolkit solution,” said Professor Bob Stone, Director of the Human Interface Technologies Team at the University of Birmingham (UK) and a VR pioneer of over over 30 years. technology With the software platforms such as , Division’s dVS/dVise, Superscape’s VRT and Sense8’s WorldToolKit back in those days, VR developers had to go about creatively creating and porting their assets to ensure the longevity and staying power of what they were building. Virtuality “Even today, we still maintain the technology-agnostic policy,” said Stone. “[We archive] our models, textures, code and so on in open or industry-standard formats so that, should one company disappear, our sponsors and collaborators will not be left empty-handed.” A British Nuclear Fuels PLC control room modeled and rendered in the 1990s using Superscape’s VRT platform. The same BNFL control room model hosted under Division’s dVS/dVISE platform, but using the same assets as in the image above. The actual BNFL Control Room, built following early prototyping and human factors trials using VR. Looking Forward Mozilla took the lead in pioneering WebVR, launching its VR content authoring tool in 2015, along with its resource. A-Frame MozVR “What we’re seeing is lots of companies, big and small, using A-Frame as a rapid content authoring tool and publishing it on the web as a low-friction content distribution method,”said Dietrich Ayala, Technical Evangelist for Mozilla. “Our position is that tools like A-Frame lower the barriers to entry for content authors and ease distribution for everyone.” Besides A-Frame, other WebVR frameworks include React VR, Argon.js, PlayCanvas, JanusVR and Primrose. , and also published their WebVR and open-source developer resources. Google Oculus Samsung Internet A WebVR Community Group to decide on WebVR standards was formed by Mozilla, Google, Samsung, Facebook, Intel, Microsoft and other major tech players, with Apple recently joining the community in . July this year “The more web there is in the VR ecosystem, the better the chances that VR will thrive and grow as a place where creativity and innovation can happen,” said Ayala. “With most major browsers implementing WebVR and Apple signaling some interest, the standards side looks really positive.” Browsers On the browser front, browser vendors are developing or enhancing their to make them VR-friendly: there’s Chrome for Android, Firefox Nightly, Samsung Internet, Microsoft Edge, Chromium, Servo and . All you need is a desktop or mobile browser and Internet connection to view a VR-enabled website with a VR headset or just view the VR content casually in 2D, without a headset. browsers Oculus Carmel Mozilla recently announced a new VR enhancement: it’s the first desktop browser for all users. to support WebVR Content and Education In the midst of a dearth of content for VR, WebVR content creators are coming together to create and collaborate. Over a million creators are sharing their 3D models on Sketchfab’s 3D/VR art community platform. Virtuleap also organized the first global WebVR hackathon. “For application domains such as education and heritage, developing VR scenes and experiences for the Web is highly important,” said Stone. “[This] promotes accessibility by many beneficiaries without the need (necessarily) for expensive and sophisticated computing or human interface hardware.” This democratized approach opens up possibilities in education far beyond what we’re seeing today. “I also think that WebVR, as a JavaScript API, also enables a wide range of future students and young developers to ‘dip their toes into the water’ and start to build portfolios demonstrating their capabilities, ultimately to future employers,” said Stone. “I remember the promise of the days of VRML and products such as SGI’s Cosmo and Cortona3D (which still available today, of course). But the ability to be able to open up interactive — and quite impressive — demos of VR experiences that existed in higher quality form on specialised platforms, became an amazing marketing tool in the late 1990s and 2000s.” Creating VR/AR content for WebVR vs native apps So you saw the ‘ ’ portal demos that have taken the internet by storm. Similarly, for WebVR, Mozilla’s A-Frame allows for that albeit with ‘ ’. Here, you can create separate universes to explore, go in and out of it on the mobile or desktop web — no app is needed. The best part is you get to stay within the website itself. This is ‘universe’ (or portal) traversal on the open web. With an app, you have to leave the constraints of the app to open a new one. Made with ARKit link traversal With ARKit, there has been a mindshift as to the possibilities of content creation for VR and AR, making content creation powers accessible to the masses. It has unlocked more creative minds to explore creating VR and AR for the web. It was only a few weeks before we saw people hacking ARKit into WebViews. There are a number of other experimentations around AR on the web: the integration of Argon.js into A-Frame, a Chromium extension that works on Tango (now ARCore) devices, and AR.js. Another exciting development for WebVR was Google’s announcement of ARCore . this week “By the end of the year all key global mobile phone brands would have announced their support for WebVR, and will have devices to support it out by Q4 2017 or Q1 2018,” said Christopher Gomez, XR industry advisor and angel investor. “And with ARKit and ARCore in the mix, we will have a vibrant mix of platforms and ideas. Immersive technologies are here to stay — it will be a reality for everybody. The ‘Metaverse’ — it’s real.” Experimentation is Key Experimenting is needed in this phase of VR — the creative power of communities is huge. And if you are a Unity developer, it is fairly seamless to integrate and embrace developing for WebVR. “I like the workflow of Unity, there are lots of support and tutorials online and if you get stuck searching any question about it will probably give you a useful answer,” said Uve, Samsung Internet developer advocate. “A-Frame, one of the popular WebVR frameworks has a similar object component system to Unity and good documentation so most Unity developers will be able to switch over and get to grips with it relatively fast, if they want to try it out.” Unity as an integrated development environment (IDE) and graphic engine is a versatile tool that enables various types of experiences built quickly. Developers are able to compile or export assets into different native or web platforms. “Nowadays it is possible to use WebVR in Unity with a , and support for WebVR will be integrated into Unity WebGL, making Unity projects work without modifications,” said Diego González-Zúñiga, developer advocate for Samsung Internet_._ “It is a matter of time until the engine will be able to create VR experiences running on the WebVR implementation of a browser.” plugin “Augmented City” #ARCity As WebVR content creators and developers start to converge worldwide, in Singapore, the community assembled at an ‘Augmented City’ experiential xLab in July. This xLab, organized by the ( ) in partnership with Mozilla and the in Asia Society, is the first of its kind globally with the kickoff of this first xLab installment supported by . XR Alliance @xrforce VR AR Women Intel Disclaimer: I’m a founding member of the XR Alliance and founder of VR AR Women in Asia Society. WebVR content creators in Singapore Using A-Frame and the prompt: “How can we augment or virtualize our experience of Singapore in ways that are interesting, engaging and fun?” participants, comprised of technical designers and developers, came together to prototype and build VR/AR content for the web, telling the history and their experience in the city in an engaging way. Creating an AR Tracker The projects, built by the workshop participants, the and the and , like any WebVR builds, allow for anyone on the web to take the code from the and build and mix on top of it for their own creations. AsiaXR group WebVR Asia community group repository “The xLab was developed to encourage creative exploration of tools in the immersive technologies space,” said Gomez. “The lab itself was designed for portability as plans on expanding it is to other cities globally, is underway. In this series, WebVR tools for VR and AR content creation were the focus. Both Mozilla and Intel have been great partners in this, and with their continued support, we know the future for VR and AR is a bright one for all of us in the industry.” Projects churned out from the one-day lab (which included an ideation session followed by a hard coding session) include a VR/AR simulation of a flood in a city (featured image), building of bucket lists, experimenting rotating 360 images in VR and AR, tracking objects in AR, and “We Built This City” — metatags of people’s memories of the city, among others. We are moving towards experiencing everything in 3D: the web, entertainment, apps, global meetings and the like — hence, companies and developers need to get up to speed or be left behind. WebVR allows for exploration of imagination, experimentation and collaboration. Content creators, independent artists and developers are hungry to create and waiting out the VR tech to develop would just not be feasible. WebVR allows for creativity of expression, democratization of VR, and an app-free experience. At least for the foreseeable future, it’s the path of least resistance that allows for the propagation of VR and AR content and a vibrant, democratized ecosystem. — — Vanessa Radd @vanradd is a founding member of the XR Alliance, president of the VR AR Association Singapore, and founder of the VR AR Women in Asia Society. Radd is ranked as the global top three augmented reality influencer.

WebVR: Taking The Path Of Least Resistance To Mainstream VR

A few months ago, I decided that I wanted to pursue a career in data engineering. I was fascinated with data because of all the big data being used, like Spark, but besides that, I knew very little. This is what I learned in the past 4 months. engineering tools Data The first thing you need to grok is what is the point of all the data? The data ultimately helps the people that are making decisions make better decisions. For instance, if you sell T-shirts and you find that most of your customer’s are between 18–25, then you can put Justin Bieber’s face on the T-shirts and all of sudden your sales will go through the roof. Once you have the data, you can do some statistics on it, make fancy visualizations, run some SQL, and as a whole the organization can make better decisions. Big Data Big data, in the most general case refers to a quantity of data, that either can’t be stored on a single server, or computing something over this data using a single server, would take too long (what is too long depends on the application). Historically, the scale of data was small enough to fit on one server. As a result, the tools that developed, e.g. Relational Database Management Systems (RDMS) were largely meant for single node deployments (this isn’t really true since distributed computing models existed since the 1960s and distributed databases like Teradata existed since the 1970s- but these were very niche). Even though our compute and storage capacities were increasing exponentially, the data generated was growing even faster. MapReduce and Hadoop Although, some solutions existed, the modern era of distributed computing and storage started when in 2004 Google published . The key proposition here along with their 2003 paper, , was how to perform distributed and fault-tolerant computing and storage on top of commodity servers. MapReduce: Simplified Data Processing on Large Clusters The Google File System Around the same time Doug Cutting was working on Nutch, an open source web crawler. Doug was facing problems of distributed storage and processing and when Google’s papers were released. He decided to implement them as part of Nutch. Eventually, these would become Hadoop and HDFS. (If you are more interested in the history, see for an excellent writeup on the topic). https://medium.com/@markobonaci/the-history-of-hadoop-68984a11704 Other Parts of Ecosystem I covered two aspects of the big data ecosystem, namely distributed computing and distributed storage. However, the ecosystem is much more diverse and has many more moving parts. Another important component is the ingestion layer. Why do we need an ingestion layer? Let’s say we have logs being generated on many nodes, and we want to store these in HDFS and run a Hadoop job on them. The ingestion layer basically acts as a fault-tolerant, distributed middle layer between the producers of data on one side, and consumers on the other. Players in the Game Now, that we have some sense of what the big data ecosystem looks like, let’s consider the two key players in the game: data scientists and data engineers. Data Scientist The job of the data scientist is to turn the big data into actionable insights. This requires them to apply machine learning, statistics, and analytical thinking, to the data. Some data scientists will have strong programming skills and some will have a deep understanding of the ecosystem of tools. However, their primary job is to run experiments, develop algorithms and get insights. Data Engineer Data engineering is a new enough role that each organization defines it a little differently. However, broadly speaking their job is to manage the data and make sure it can be channeled as required. In some companies, this means data engineers build the underlying system that allows data scientists to efficiently do their job, e.g. at Netflix data engineers may build and maintain the infrastructure that allows data scientists to experiment with recommendation algorithms, and in other companies, the data engineering is the whole shebang, e.g. at Twitter, the biggest challenge is how to make data flow as quickly and efficiently as possible. Typically, a data engineer will have strong programming skills and a deep understanding of the big data ecosystem, and distributed systems in general. A data engineer will perform one of the following: Designing, implementing, and maintaining data warehouses Designing, implementing, and maintaining ETL pipelines DevOps Working on infrastructure tools Running statistical experiments, implementing machine learning, etc. 1. Data Warehouse I mentioned before that sometimes the job of the data engineer is to make data accessible, so for instance the data scientist can run queries on top of this data. However, this leads to the question of how this data should be stored. The answer to this question is very nuanced and this is where the concept of data warehouse comes into play. A data warehouse is a consolidation of all the data in an organization in order to make it easier to analyze. The raw data is often heterogenous in its format and location. For instance, you may have some data coming from log files and other from transactional databases, e.g. a MySQL database that stores the order quantity and order amount. To give you a sense of some of the subtleties in working with data warehouses, consider that for transactional databases, data is stored in normalized form, so that the data does not become anomalous. To perform complex queries on normalized data, requires joins. Joins, in general ,are expensive, and at the big data scale, when the data is located on multiple servers, across the network, joins become prohibitively expensive. A solution to this is to denormalize the data. However, denormalizing the data, typically leads to increased storage space. 2. ETL (Extract, Transform, Load) ETL is related to the concept of Data Warehouses. Essentially, ETL refers to moving the data from some initial source, to a different location. For instance, if we have CSV log files and we want to put them in our data warehouse, we must first read them from the source (extract), then strip the commas, and remove malformed lines (transform), and put them in the final data warehouse (load). 3. DevOps In some cases, data engineers will have to work with the raw compute and storage nodes, e.g. installing the software, etc. This is more typical in smaller organizations where the roles are not clearly delineated. Larger organization will have specialized DevOps roles. 4. Working on Infrastructure Tools In some cases, data engineers will work on infrastructure tools, e.g. adding an extension to Hadoop. Again, this is more typical of smaller organizations, with larger organizations having separate tooling teams. 5. Running statistical experiments, implementing machine learning, etc. This is closer to the role of data scientist, but depending on your background and interests you may be more or less close to the data science side of things. Again, this varies from organization to organization. Data engineering is a very exciting field, and as data driven decision making becomes more central to our businesses so will the challenges and opportunities of data engineering. is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

SuperScan

superscan-app.com

-- employees

Since n.d.

COMPANY RANKING

SUPERSCAN

EVERGREEN INDEX #10840

SuperScan WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SuperScan

superscan-app.com

-- employees

Since n.d.

COMPANY RANKING

SUPERSCAN

EVERGREEN INDEX #10840

RELATED COMPANIESRANK

STORIES ABOUT SUPERSCAN

SuperScan WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!