At SparkLoop, we build the best referral software for growing your email-list

SparkLoop

sparkloop

Data Management and Consolidation in the Integration of Corporate Information Systems

Learn how to integrate a photo booth to IPFS and make your selfie go interplanetary with web3.storage. During Web Summit 2022, the Filecoin events and creative team set up an amazing exhibition stand at the convention that attracted more than 3500 visitors wanting to learn more about Web3, the InterPlanetary File System (IPFS), and the Filecoin network’s distributed storage solutions. Filecoin offers a wide range of solutions for storing and persisting files on Web3 via IPFS Protocol. One of the most popular and easy to use is web3.Storage, a “one API call to and Filecoin” solution for developers. IPFS IPFS is a web protocol that allows sharing files in a peer-to-peer network and sharing data in a distributed computer network. created an open-source protocol that enables a storage market to persist files on IPFS for a long time and with little cost. This solution solves the storage problem in Web3 and upgrades the internet as we know it to handle challenges like interplanetary communication. Filecoin Interplanetary communication to include selfies! Learn how to use a lightweight node.js application to make a photo booth software called Sparkbooth to connect to web3.storage and upload your selfie to IPFS. has How does it work? The main parts of the photo booths are: a digital camera, some lights, a computer with touch screen, some software that controls the camera to take photos and the ability to upload them to IPFS and Filecoin network. A flow of events looks like this: Touch screen receives commands from user to take a photo Photo Booth software (called Sparkbooth) will talk to the camera to take action Sparkbooth stores the photo locally on the computer Sparkbooth asks user if they want to send to a server (via web3.storage) Sparkbooth sends the photo, username, password, and success message in an API call to * *custom nodejs app fil-photo-booth-uploader fil-photobooth-uploader app will: validate the username and password change the format of the photo to what accepts web3.storage retrieves API token/key from system and send to __ __Storage web3. w__ __ sends the photo to the IPFS network and back up on Filecoin eb3.storage w__ __ will return the IPFS Content ID (CID) back to the eb3.storage fil-photo-booth-uploader will construct the http version of the CID url via IPFS gateway fil-photo-booth-uploader web3.storage fil-photo-booth-uploader returns a success message back to Sparkbooth 7 Sparkbooth receives a success message: “Your photo is sent to IPFS via . :-)” along with the URL of the photo web3.storage Sparkbooth displays the QR code generated on the screen The look? It’s just like a normal photo booth: An example of the IPFS CID url: https://bafybeiei7zadrztflc6krunhvqr3umzre7xjxfzvmyjs2ob2w7yykq63ea.ipfs.w3s.link/20221104172648.jpg An example: The entire flow looks like this: To learn more about IPFS and CIDs, please visit . proto.school Step by Step Guide Prerequisites There are few things you might need to run through this tutorial: 📸 Camera - any modern Canon 💻 Computer - any all-in-one desktop with touch screen or a laptop with touch screen 👨🏻‍💻 Photo Booth Software - Sparkbooth 7 🤖 An app server - I am using Heroku for convenience 📦 IPFS Enabled Product - a web3.storage account 🌎 Internet connection - a way to talk to WWW either wired or wireless (wifi)\ Step 1, Web3.Storage Setup Create a account via your account or email web3.storage Github Go to Account > API Token Create a new API Token, name it to anything you like (e.g. Filecoin Web Summit 2022) Copy the API key by using the copy button Save this for later You will need to put this in the fil-photo-booth-uploader application as an environment variable Step 2: nodejs app deployment For Sparkbooth 7 and web3.storage to talk nicely, we will need a little help to translate their languages. The nodejs app will help us achieve this. It will need to be hosted on a server, and we are using Heroku for it (If you know a Web3 native way to host it, please DM me. I would love to try). First go to (sign up for a new account if you don’t have one) github.com Visit the and then click on Fork. This will allow you to copy the source code to your own account or repository and allow you to connect it to Heroku. Filecoin Photo Booth Uploader Sign up with a . Heroku is like AWS, a centralized developer platform that helps run applications or servers almost for free. Heroku account Create a new app and put the event name in it. We are likely going to need a new app for each event. Under “Deployment method,” choose GitHub and then sign into your GitHub account. Authorize Heroku to pull code into itself to build the application and run the server. Under “Connect to GitHub,” search for the repo name you just forked (ex. fil-photo) then connect the code repository. Leave all the settings and  click on “Deploy Branch” to kick off the first build. Heroku is smart enough to be able to recognize the application type. Next:build and deploy it. However, before the server is working,we will need to put the username, password, and the API Token in the server environment. Heroku can access it securely without exposing them to the internet. Go to “Setting” then “Config Vars.” web3.storage Put in the following configurations, swap out the SPARKBOOTH_PASSWORD, SPARKBOOTH_USER, W3S_API_TOKEN with password, username and api token respectively. The name has to be exactly as above. web3.storage The app should be running now! You can go to “Domains” under “Settings” to find the URL to put into sparkbooth 7 later. Step 3: Sparkbooth 7 Setup The final step is to prepare the software on the photo booth laptop to talk to our custom nodejs app to complete the setup flow. \ Head to . Download , pay for or use the trial. Sparkbooth.com Sparkbooth 7 DSLR license fee Open Sparkbooth and confirm your purchase. When you open the app, it will maximize, but you can press “ESC” to exit. Go to the settings on top left: Configure “Send to Account” then enter the username and password you would like to use. The “Service URL” needs to be the Heroku app address + “/upload-w3s”. (e.g. http://fil-photo-booth-uploader.herokuapp.com:443/upload-w3s) You can test the connection by using the test button. It will result in a failure message, but the API call will still go through to the Heroku app and place a test photo in your account. You can check the test image in your web3.storage account. If you can see the “photo booth test” photo, the setup is successful! Click to check. web3.storage here w3up beta coming soon This app was built in 2022 and uses the existing web3.storage APIs. At the time of this writing, there are new beta w3up APIs available from web3.storage, which . This will make it even easier to get started writing apps that use IPFS. include front end web components for multiple frameworks Some improvement suggestions The app is not perfect! There are a few areas that could be improved: Security Even though this is just a gimmick to show ground, the security of the nodejs app is not ideal. Validating the passwords and username in application logic is not preferred. I believe we should put the application behind a firewall or API gateway that validates the authorization in the header before passing the call to our app. Experience The Sparkbooth 7 is great, but the UI does seem quite limiting. It would be good if we could customize this a bit more. Perhaps an open-source photo booth software could help us achieve this? Recent Photo Gallery Potentially, we could use a separate computer to show the latest photos to visitors. They could select the photo they would like to choose to email to themselves, display QR code again, or share it on social media. This will involve building a frontend for the fil-photo-booth-uploader, which could be a very cool project for a front-end developer. Use an decentralised app backend. Maybe someone like or is able to help. Fluence IPVM Conclusion During the three-day Web Summit, the Filecoin photo booth had an uptime of 99% and a1% outage due to the internet connectivity issues at the convention. In total, the photo booth published 934 photos to the Filecoin Network. That’s almost one thousand selfies that are ready to go interplanetary all thanks to the very handy service of web3.storage! The photo booth is also famous. On the third day, a reporter from CNN also interviewed the booth, (yes, the booth is getting famous… not me). CNN reporter interviewing the Filecoin Photo Booth at Web Summit, Lisbon, Portugal Please feel free to use this guide to build your own interplanetary photo booth for your events, gatherings, meetups, parties or just for fun. The Filecoin network now has 16EiB of storage capacity and over 300PiB of data has been stored on it. There are many ways to contribute or use this project. Learn more about getting involved . here

How to Build an Interplanetary Photo Booth

কিভাবে একটি ইন্টারপ্ল্যানেটারি ফটো বুথ তৈরি করবেন

So bauen Sie eine interplanetare Fotokabine

Cómo construir un fotomatón interplanetario

Comment construire un photomaton interplanétaire

इंटरप्लेनेटरी फोटो बूथ कैसे बनाएं

惑星間写真ブースの作り方

행성 간 사진 부스를 만드는 방법

Como construir uma cabine de fotos interplanetária

Gezegenlerarası Fotoğraf Kabini Nasıl İnşa Edilir

Cách xây dựng một gian hàng ảnh liên hành tinh

如何建造星际照相亭

Ever heard of “gas wars”? It is what users experience on the Ethereum network when performing transactions in high network congestion. Due to the popularity of Ethereum which has caused price appreciation in ETH in the past few months, users have had to deal with high gas fees when performing different operations. Refer to Gas and Fees in Ethereum’s Docs to first understand what gas fees are. A particular solution was offered for tweaking Ethereum’s current gas fee model: EIP 1559, which recently has been activated on 5th of August, 2021. So, we're currently undergoing major changes. Gas fees have gone beyond 200 Gwei since the beginning of this year, which is extremely inefficient to users. ETHERSCAN.IO This proposal was first by Ethereum co-founder Vitalik Buterin in 2018 to replace Ethereum’s old price-auction model, which suffers from a lot of pitfalls. brought up The old price-auction model (Pre - EIP 1559): All pending transactions from the Ethereum network land in the Mempool, a place where miners pick them up to verify and include them in the next Ethereum block. Since there is a maximum gas limit per block (number of transactions to be included per block) which is capped to 12.5 million gas, miners are incentivized to verify and include the most profitable ones, which has given the power to the richer users and has allowed for the existence of unfairness: Users who don’t overpay suffer from delays, and most of the time, their transactions could fail. Yet, regardless of whether transactions fail or succeed, users still pay for the computation: miners must still validate and execute your transaction (compute) even if it fails. All of these mentioned cases have no social benefit to users, which creates pure deadweight loss. “front running” in decentralized exchanges. So, how will EIP - 1559 fix these issues? The new transaction pricing mechanism will include a fixed-per-block base fee and “tips'' to incentivize miners. In more depth, the base fee is fully burned – miners will not be rewarded. This fee contracts/expands according to an algorithmic formula which is the function of gas used in the parent block (most recent previous block) and gas target of the parent block. The per-block fixed gas usage (12.5m) is relaxed in this proposal and is rather utilized as a median; the base fee adjusts whenever the size of the latest parent block is bigger/smaller than this average block size. Block size can at most increase by 2x, which according to the proposal, will approximately increase the base fee by 1.125x; this should mitigate the risk of “over-stressing nodes and miners” during periods of high network congestion. Moreover, the creator of the transaction specifies a certain amount for the “tip” to be paid with the base fee, which directly goes to the miner in charge. Just as in the price auction model, the larger the tip, the faster a user’s transaction will be verified. Users also specify the maximum gas fees (tip + base fee) they are willing to pay in their transactions. Accordingly, the proposal should definitely enhance users’ experiences on the Ethereum network. It doesn’t aim to reduce gas fees but rather places caps to mitigate fee spikes. Users' fee estimation will be very straightforward and will not be based on other people’s behaviors since base fee changes are algorithmically set. Moreover, base fee changes are restricted, which makes it even more measurable for users. The likelihood of miners including dummy transactions might eventually decrease since they’ll be obliged to pay the base fees (which they wont be rewarded with and will get burned). Yet, on the other hand, there might be some pitfalls and some people have raised their concerns. First, ETH will be deflationary which should cause price appreciation for ETH holders if more is burned on the base fee than is generated in mining rewards – the tips. Developers aren’t certain whether ETH will end up being inflationary or deflationary since it’s entirely dependent on user demand for block space. By burning the base fee, the supply of ETH will not be constant over time, which could create some economic instability in the long run - the argument that inflation is ‘healthy’ could be brought up. Secondly, of course, miners are the most negatively affected by those new changes. Popular pools like Ethermine, Sparkpool, and Flexpool have shown their discontent. Over 60 percent of the hash power is against the proposal; even though this means that a 51 percent attack is probable, it’s deemed to be unlikely as there are more financial incentives not to attack the network. Overall, it is estimated that miner revenue will be reduced by 20% to 35% at most. Lastly, in Laura Shin’s Unchain Podcast, both Kain Warwick, co-founder of Synthetix, and Kyle Samani, managing partner at Multicoin Capital, pinpointed that there’ll be an undistributed wealth effect: a unit bias that’ll allow people to feel that they don’t own a decent stake of the network, which could disincentivize investors to lock in their ETH. They further iterate that simply, due to the burning of base fees, the price appreciation of ETH will make the Ethereum network more expensive to use, which counterfeits what the proposal aims to do.

      Ethereum: The EIP- 1559 Narrative

Is it finally for ETH transaction fees to go lower with EIP-1559? 🤔 The Time Yes, but juuusst a little bit! 🤏 EIP-1559 introduces several changes that will help you , and settle your transactions . But don’t get your hopes too high; I’ll explain that in a minute. pay less fees quicker than before A brief overview 👨‍💻 EIP-1559 was first proposed by Vitalik Buterin in 2018 and is the most anticipated upgrade in Ethereum’s history. Before we go into the details, here is a TLDR for what EIP-1559 is all about: 👉 There is no ‘single fee’ anymore Instead, there are two fees - One is called , and the second one is called , or a tip for the miners. The base fee is burned, while the inclusion fee (tip) goes to the miners. Base fee Inclusion fee 👉 Elastic block size Block sizes are now dynamic and will increase or decrease depending on the state of the network congestion. The maximum gas limit per block will now be 25 million, which is 2x the current gas limit per block of 12.5 million. However, the target gas limit per block will be kept at 12.5 million, but in the state of transient congestion, bigger blocks will be used to accumulate more transactions. That’s pretty much this improvement proposal is all about! 😀 Now let’s perform a much deeper analysis to understand the mechanics and impact of EIP-1559. First Price Auction model & wrong fees estimation 🚨 Aside from the standard block reward, miners or block producers are also rewarded with transaction fees. Each transaction includes a fee, and miners incentivize themselves by prioritizing transactions with higher fees to maximize their revenue. If you want your transaction to process quicker, you’ll simply bid a higher fee. This is called the first price auction model, where the chances of your transaction included in the nearest block are maximized if you bid higher. First price auction model is designed to prioritize high-value use cases and make sure that the blockchain isn’t filled with lower-value use cases. However, there is one single problem with first price auction mechanism, i.e., there is no way to calculate the optimal price of a single transaction. Let’s say you included a Tx fee of $10 because the previous higher Tx fee was $8, but if everyone else is bidding $5, you would have been better off if you included a Tx fee of $6, saving $4 straightaway! Guess what? The current ethereum wallet you’re using cannot accurately estimate how much Tx fee you should be paying, and almost all the time, people tend to pay higher than they should be for their transaction to go through. One solution to this problem could be a , where the next bid is always higher than the previous bid. In this case, however, there is a serious loophole where the miners will include their own transactions in a block, thereby pumping the Tx fee artificially to maximize their revenue. uniform price auction How does EIP-1559 solve the problem? EIP-1559 introduces a new transaction pricing mechanism to overcome the problems that we saw in the first price auction model. It introduces two different types of fees, i.e., the . base fee and inclusion fee For the geeks out there, this fee will be a function of gas used in the parent block and gas target (formerly known as gas limit) of the parent block. The Base fee will be a fixed-per-block network fee that any transaction has to include. If the blocks are above the gas target limit, this and when they are below the gas target limit, . The gas target limit is set to 12.5 million, but it can go to a maximum gas limit per block of 25 million (2x the gas target). base fee will increase base fee will decrease When the network congestion is higher, the base fee will be higher, and when the network congestion is lower, the base fee will be lower. However, the transactions will be slightly quicker because blocks with higher gas limit would include more transactions to be processed. Because these base fee changes are constrained, the maximum difference in base fee from block to block is . This will allow wallets to auto-set the gas fees for users in a highly reliable fashion. predictable Deflationary pressure on Ethereum 📉 This base fee will be burned, thereby putting deflationary pressure on Ethereum. To put this in perspective, the issuance rate of Ethereum stands at 4.5% per year and according to the estimates, the issuance rate once the proposal is implemented will be 0.5 - 1%. This is HUGE! If EIP-1559 was live yesterday, it would have burned roughly 17,000 ETH. Annually that equates to over 6.2 million ETH! It will have more effect than halving has on Bitcoin. Miners’ perspective & the inclusion fee 💸 EIP-1559 is, without a doubt, bad news for the miners! Last month, Ethereum miners made a whopping $1.3 billion, and almost 50% of it came from transaction fees. Imagine losing a big portion of your business; it sounds painful! Alongside the base fee, there is an inclusion fee that will directly go to the miners. However, people can choose not to add an inclusion fee. Two of the three largest Ethereum mining pools have taken a public stance on EIP-1559. (combined hashrate above 50% at present) is on the opposing side, while (11% of the network's hashrate) is on the supporting side. Sparkpool F2Pool Regardless, EIP-1559 gives miners more control over the block size and makes their revenue more predictable. EIP-1559 will be packaged with the London hard fork in July this year, regardless of the mining industry’s discontent with the proposal. Learn more about EIP-1559 📖💡 If you want to learn more about this improvement proposal, have a look at the following resources: (official source) 👉 EIP-1559 GitHub (official source) 👉 Ethereum Notes (a brief guide) 👉 CoinDesk Over and out! 👋 Haroon Agree or disagree with some of the ideas presented in this article? Let us know your thoughts on Digg. Looking for more blockchain articles? Subscribe to our newsletter in the footerbelow!

What the heck is EIP-1559? 

Introduction In this blog, we have detailed the approach of how to use Spark on Kubernetes and also a brief comparison between various cluster managers available for Spark. Spark Spark is a general-purpose distributed data processing engine designed for fast computation. The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application. It supports workloads such as batch applications, iterative algorithms, interactive queries and streaming. During execution, it creates the following components: Driver Executor To manage these components, there is a cluster manager that takes care of resource allocation. Following are the various options available for cluster manager: Standalone Apache Mesos Hadoop YARN Kubernetes Standalone Cluster Manager To use Spark Standalone Cluster manager and execute code, there is no default high availability mode available, so we need additional components like Zookeeper installed and configured. Hadoop YARN YARN (Yet Another Resource Negotiator) focuses on distributing MapReduce workloads and is majorly used for Spark workloads. Apache Mesos The Mesos kernel runs on every machine and provides applications with APIs for resource management, scheduling across the entire datacenter, and cloud environments. It provides a cluster manager which can execute the Spark code. Kubernetes It uses the kube-api server as a cluster manager and handles execution. Comparison between Hadoop YARN and Kubernetes – as a cluster manager Kubernetes is a container orchestration engine which ensures there is always a high availability of resources. Apart from that it also has below features. Brief introduction Kubernetes and its components Self healing Automatic rolling updates and rollback Resource management Service discovery Load balancing Service discovery Architecture of Kubernetes The architecture of Kubernetes has 2 major components, they are: 1. Master components All the requests from the user using API, kubectl are sent to master component that is the API Server converts json or yaml requests to http call API Server contains the details of the cluster and its components and current state ETCD ensures that the cluster is always in the desired state Controller takes care of object creation based on resource availability Scheduler 2. Node components The following are the components that are come under the nodes: Kubelet is the agent which takes care of the execution of the tasks which have been assigned to it and reports back the status to the api server Kube Proxy is the networking component that takes care of networking related tasks Container Runtime it provides an environment on the nodes for container execution Other components Kubectl: a utility used to communicate with the Kubernetes cluster Spark execution on Kubernetes Below is the pictorial representation of spark-submit to API server. We can use spark-submit directly to submit a Spark application to a Kubernetes cluster. Once submitted, the following events occur: Creation of a Spark driver running as a . Kubernetes pod Creation of executors which are also run within Kubernetes pods, connects to them, and executes the application code Termination and clean up of executor pods occur when the application completes. But the driver pod persists, logs and remains in “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up. There are 2 options available for executing Spark on a EKS cluster: Option 1: Using Kubernetes Master as Scheduler Option 2: Using Spark Operator Option 1: Using Kubernetes master as scheduler Below are the prerequisites for executing spark-submit using: A) Docker image with code for execution B) Service account with access for creation of pods, services, secrets C) Spark-submit binary in local machine A. Creating Docker image for Java and PySpark execution Download Spark binary in the local machine using this link https://archive.apache.org/dist/spark/ In this path spark/kubernetes/dockerfiles/spark there is Dockerfile which can be used to build a docker image for jar execution Ensure that you are in the Spark directory as it needs jars and other binaries to be copied. So it uses all the directories as context. docker build -t latest -f kubernetes spark/Dockerfile . spark: /dockerfiles/ Once this image is built, it can be used as a base image for the other code execution. Docker image creation for PySpark code execution: In this path: there is a ready Docker file which will be used for PySpark execution. spark/kubernetes/dockerfiles/spark/bindings/python Ensure that you are in the Spark directory as it needs jars and other binaries to be copied so it will use all the directories as context. docker build -t latest -f \
kubernetes spark dockerfiles bindings Dockerfile . sparkpy: /dockerfiles/ /kubernetes/ /spark/ /python/ B. Creating Kubernetes service account and cluster-role binding Components that will be used: Service account: an account which will be used for authentication of processes running inside the pods Cluster-Role: defines the access for the service account across the cluster Cluster-Role Binding: binds and creates the role with the service account Create a yaml file with below contents apiVersion: v1 kind: ServiceAccount metadata: name: spark-sa --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: spark-role rules: - apiGroups: [""] resources: ["pods", "services" , "configmaps" ] verbs: ["create", "get" , "watch" , "list" , "post" , "delete" ] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: spark-role-binding subjects: - kind: ServiceAccount name: spark-sa roleRef: kind: ClusterRole name: spark-role apiGroup: rbac.authorization.k8s.io kubectl apply -f rbac.yaml Getting cluster information: cluster- kubectl info This command gives the master url it and it will look as shown below Kubernetes master is running at https: //ABCDZZZZZZZZZZZZZZZ.sk1.region.eks.amazonaws.com Make a note of this url C. Executing Spark-submit: Now go to the directory which has Spark binary and use the below command: ./spark-submit  
--deploy-mode cluster
--master <master_url> 
--conf spark =spark-sa
--name spark-pi
--conf spark = --conf spark =<docker image with code>  
--conf spark =<docker image with code>  
local: .kubernetes .authenticate .driver .serviceAccountName .executor .instances 5 .kubernetes .driver .container .image .kubernetes .executor .container .image ///opt/spark/examples/src/main/python/pi.py The driver and executor will be created with whatever name is specified as app name. Using the driver pod we can view logs, access url below commands can be used for it. To view logs kubectl logs -f spark- - - pi 1581014997671 driver To view spark ui kubectl port-forward (driver-pod-name) : 4040 4040 Post execution, the drivers will be available based on the execution state, until it is deleted manually. Additional useful options that can be used with Spark-submit Environment Variables: When the script requires any environment variable that need to be passed, it can be done using Kubernetes secret and referred to it. Details of achieving this are given below. “ ”: “ ”, spark .kubernetes .executor .secretKeyRef .DB_PASS snowsec :db_pass Create a secret with the name snowsec and in that db_pass is the key and which will be referred to the spark environment using DB_PASS. Adding labels to the pod: When we want to add additional labels to pod we can use below options For driver pod: spark.kubernetes.driver.label.[LabelName] For executor pod: spark.kubernetes.executor.label.[LabelName] Using node affinity: We can control the scheduling of pods on nodes using selector for which options are available in Spark that is . ​ spark .kubernetes .node .selector [labelKey] Option 2: Using Spark operator on Kubernetes Operators Operator is a method of packaging, deploying and managing a Kubernetes application. Kubernetes application is one that is both deployed on Kubernetes, managed using the Kubernetes APIs and kubectl tooling. Using Spark operator on Kubernetes Official link https://operatorhub.io/operator/spark-gcp Installing Kubernetes operator on EKS: Prerequisites: Helm needs to be installed and configured Verify Helm installation using below command helm version –short Adding Helm repo to running EKS Helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator Installing Helm repo on EKS : helm install incubator/sparkoperator 
--name sparkv24 
--namespace -- =default  
-- = default set sparkJobNamespace set enableWebhook true Verifying Installation: helm ls Once applied, the following components will be created: Spark-operator-deployment Spark-serviceaccount Spark-rbac Spark-operator-serviceaccount Spark-operator-rbac Webhook-init-job Webhook-service Here we need the job in yaml. Once we have yaml file, we can submit the job using the below command: kubectl -f apply The Yaml file looks as follows: : : SparkApplication : : spark-pi-jar : default : : Scala : cluster : : Always : org.apache.spark.examples.SparkPi : : : : Never :
    - : : : : Directory : : : : : : . : sparkv24-spark :
      - : : : : : : : : . :
      - : : apiVersion "sparkoperator.k8s.io/v1beta2" kind metadata name namespace spec type mode image "gcr.io/spark-operator/spark:v2.4.4" imagePullPolicy mainClass mainApplicationFile "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar" sparkVersion "2.4.4" restartPolicy type volumes name "test-volume" hostPath path "/tmp" type driver cores 1 coreLimit "1200m" memory "512m" labels version 2.4 4 serviceAccount volumeMounts name "test-volume" mountPath "/tmp" executor cores 1 instances 1 memory "512m" labels version 2.4 4 volumeMounts name "test-volume" mountPath "/tmp" Once we submit the job, it will create 2 pods: Executor Driver We can verify these using kubectl pods get Logs by using below command logs -f pod_name kubectl Details of submitted job using below command kubectl describe sparkapplication Contains details about Web-UI, service and events that occurred during creation. Access Web UI: To access the Web UI for a long running job can be done using port forwarding, using the below mentioned command kubectl port-forward (driver-pod-name) : 4040 4040 Conclusion Key considerations for Production Spark code on Kubernetes 1. Cost-Effective No requirement of up and running infrastructure to use Spark on EKS. 2. Build and Deployments As we deploy the Docker image with the Spark submit, so when we have code changes we need to pass the docker image with Spark submit. These images can be tagged to track the changes. 3. Ideal Use Cases When workload is less (e.g. 8-10 hr job executions per day) and as batch processing. 4. Availability/Fault Tolerance Kubernetes has the scheduler which manages the pods created as driver and executor. This enables the usage of pods based on resource availability. Quotas for a namespace can be assigned for better resource management. 5. Resource Tracking We have used node selectors in Spark submit which allows us to run specific workloads on a specific node. This in turn allows us to track the usage of resources. 6. Monitoring We have integrated Spark workloads monitoring with Prometheus and Grafana, by using Kube-state-metrics and creating a dashboard. We do a Spark submit by assigning pod labels which allow us to create custom dashboards for specific labels. 7. Logging We have used the ELK stack for visualizing the logs. The approach we have detailed is suitable for pipelines which use spark as a containerized service. It also ensures optimal utilization of all the resources as there is no requirement for any component, up and running before doing Spark-submit. Additionally, Spark can utilize features like namespace, quotas along with other features of Kubernetes. About the Author Ajaykumar Baljoshi is a Senior Devops Engineer at , who currently works on Containerization, Kubernetes, DevOps and Infrastructure as Code. Sigmoid Previously published at https://www.sigmoid.com/blogs/containerization-of-pyspark-using-kubernetes/

Containerization of Spark Python Using Kubernetes

Peeling the invisibility cloak to reveal what magic of Grin and the magical world of the MimbleWimble blockchain implementation Expecto Patronum — Grin is conjured Grin has just launched as of 11PM GMT+8. I’m extremely excited for this event and have chosen to put together this content piece to help commemorate this milestone and make an easy to read piece for people who are new to Grin. This should bring you in as first years and equip you with enough resources to venture to the second year and beyond. You can use this post as a guide, read the links and come back later to finish the rest of the post. Onwards~ Read the whitepaper(s) to understand why Mimblewimble or Grin On 19 July, 2016 — A person or a group nicknamed: “Tom Elvis Jedusor” dropped the whitepaper for Mimblewimble on the world in a private IRC channel as an introduction to Grin: https://download.wpsoftware.net/bitcoin/wizardry/mimblewimble.txt “I call my creation Mimblewimble because it is used to prevent the blockchain from talking about all user's information” - Tom Elvis Jedusor Within a span of 3 months, Andrew Poelstra submitted a follow-up whitepaper solving various issues and thus bringing us deeper down the rabbit hole. https://download.wpsoftware.net/bitcoin/wizardry/mimblewimble.pdf If you’re relatively new to cryptocurrencies and blockchain, I’d suggest this explanation of the benefits of Mimblewimble by the Beam team: https://medium.com/beam-mw/mimblewimble-explained-like-youre-12-d779a5bb483d You can then supplement the information with the official documentation and introduction to Grin and Mimblewimble: https://github.com/mimblewimble/grin/blob/master/doc/intro.md Note that Grin and Beam are both implementations of the MimbleWimble blockchain. Now that you understand what’s Elliptic Curve cryptography, Pederson commitment, and how it enables the node to never know what the transaction amounts were, you can proceed. How many Grin coins does Gringotts have? The number of Grincoins can be modeled in the following chart. The total supply at any time can be determined by block height * 60 since Grin is designed to have a block every 60 seconds on average with 60 Grincoins in each block. One can see that this is very similar to that of Bitcoin. CryptoProfG goes in depth and analyses Grincoins’ monetary model in detail with his views as a store of value: https://medium.com/@CryptoProfG/grin-money-explained-4-exploring-grins-monetary-model-e48b1761653 Am I contributing to someone’s pockets again? Since last year, we’ve seen many companies where the team receives outsized rewards and have not delivered on product. Grin is not doing an ICO, there’s no premine and the development of Grin is entirely funded by donations and the community. Needless to say, this gets us very excited to be a part of the community to actively contribute. Supply starts from 0. Let the chanting (mining) begin As Ethereum miners shut down their miners due to profitability issues, we’ve seen significant mining demand for Grin. Most large pools are now supporting Grin: f2pool, SparkPool, MWGrinpool, Grinmint to name the top few. Grin Mainnet will accept Cuckaroo29 and Cuckatoo31+ with the latter comprising 10% of blocks mined. Cuckatoo31+ is meant to be ASIC friendly and will likely be 100% after 2 years, giving manufacturers ample time to plan ahead and develop ASICs. The primary proof-of-work is designed to be ASIC friendly and is not expected to change in the future. ASICs manufacturers are encouraged to design specialized machine to mine it. For existing GPU holders, the estimated GPU mining stats for the various supported cards can be found here: https://github.com/mimblewimble/docs/wiki/GPU-Mining-Stats If you are curious on what the network hashrates and difficulty are, keep close tabs on this resource by the BlockCypher team. _grin mining pool_www.grinmint.com Grinmint - Grin Mining Pool | Network Charts Viewing the Triwizard Grin tournament Not everyone can participate in mining, but if you want to just watch the blocks get created, I’d recommend you build or use a Grin block explorer (Grinscan maybe? I checked and it’s bought by some cunning folk). _GrinExplorer.net is powered by grin-explorer released under the GNU AGPL._grinexplorer.net GrinExplorer _GrinMint/Explorer is powered by grin-explorer released under the GNU AGPL._www.grinmint.com GrinExplorer Our friends at Counter.network built this too: https://grin.merklepath.com/ You can also head over to CoinMarketCap.com and CoinGecko.com to observe prices and other information once they have successfully integrated the APIs. Greed is good As with any amazing cryptocurrency project, we all have to admit speculation is part of the fun. Exchanges, decentralized exchanges, OTC networks and derivatives markets all serve that purpose and more. We also need a healthy market and liquidity to make sure all participants have a healthy playground. We see a growing list of exchanges throwing their hats to the game. bisq.networkcounter.networkbitmesh.comgalleon.exchangegrinpay.orgchainrift.comhb.topnexex.iovinex.networkkkex.vipbitforex These exchanges will be possible places to store your tokens. I’d also recommend holding them in your own wallet with your own keys. You can follow this guide to install and run a Grin node + wallet on your MacOS or Linux device: . https://medium.com/luxor/how-to-install-and-run-a-grin-node-wallet-32f344591478 Wallet713 for Grin by Vault713: https://github.com/vault713/wallet713 I definitely hope to see good hardware wallet solutions being developed. Time to get to work Ledger, KeepKey, Cobo Vault, Trezor and imKey team. Keep the party going Donation is a key part of making sure Grin core developers continue making Grin more awesome. You can buy Grin merchandise, organize events, attend events and contribute to the general fund: https://grin-tech.org/funding Meet other fellow wizards What’s a community without conversations, discussions and debates? You can discuss over telegram chats, meetups, conferences Gitter: Discord: Grinhub Telegram Group: t.me/GrinCoinGrinfans Telegram Group: t.me/grinfansGrincon US: Grin Asia meetup: https://gitter.im/grin_community/Lobby https://discordapp.com/invite/Z3sEfEU https://grincon.us/ https://meetup.com/grinasia/ Spencer Yang can be found lurking on Telegram @spenceryang. He’s a partner at Hashtag Capital. Hashtag Capital is a multi-strategy electronic trading fund. We specifically provide OTC services for Grin. We will donate a certain % of profits to the Grin community fund. We co-organize events like Grin Asia (meetup.com/grinasia) in Singapore on 23 January, 2019. The partners of Hashtag Capital have supported the growth of companies like CryptoAudiences, Counter Network, CryptoKitties, CoinMarketCap, Cobo, Chainrock, Akio Labs, PledgeCamp, EOSIO.SG and many others over the past 2 years.

Expecto patronum — a starter user guide to Grin & Mimblewimble

This Tutorial shows how to generate a billing for AWS Glue ETL Job usage (simplified and assumed problem details), with the goal of learning to: Unittest in PySpark Writing Basic Function Definition and Conversion to UDF Repository: https://gitlab.com/suekto.andreas/gluebilling Business Problem This tutorial shall build a simplified problem of generating billing reports for usage of AWS Glue ETL Job. (Disclaimer: all details here are merely hypothetical and mixed with assumption by author) Let’s say as an input data is the logs records of job id being run, the start time in RFC3339, the end time in RFC3339, and the DPU it used. The price of usage is 0.44USD per DPU-Hour, billed per second, with a 10-minute minimum for each ETL job, while crawler cost 0.20USD per DPU-Hour, billed per second with a 200s minimum for each run (once again these numbers are made up for the purpose of learning.) Now we are going to calculate the daily billing summary for our AWS Glue ETL usage. Prerequisite Install JDK 1.8, and setup the JAVA_HOME into the corresponding location (I personally using Mac, and following this and — and JAVA_HOME would be /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/) link download-link Python 2.7.x Python Coverage, unittest, and PySpark packages (Preferably installed under virtualenv) Actually for this project I am using a bash-script builder to establish the basis of the project, it still in beta state, and this tutorial is being used to see how comfortable I am with that auto script builder. Similar result can be achieved by manually following this articles: https://hackernoon.com/setting-up-python-dev-in-vs-code-e84f01c1f64b https://medium.com/@suekto.andreas/automate-documentation-in-python-925c38eae69f Activities To do the above we would go along with the following pseudocode: Load into PySpark DataFrame Calculate the duration in seconds out of the from and to unix timestamp Calculate the fee per records 001 — Calculate Duration (+ unit test) First let’s build the calculate duration function. The following is the module directory structure that we are going to use : module gluebilling — billing.py All files *.py are started with empty files, the __init__.py is to indicate that we define gluebilling as a python module. let’s start with writing the function skeleton in billing.py. """Process records of ETL Job usage into the billing of fee"""from datetime import datetime as dt def calculate_duration(from_timestamp, to_timestamp):"""Returns duration in second between two timereturn 0 Based on the skeleton, we already have a clearer though of how it should be called and what kind of value shall it returns, now let us create the unit test for this function, to define in more details the behaviour of it. This is written in billing_utest.py """Unit Testing for Glue Billing Project"""import unittest import gluebilling.billing as billing class CalculateDurationTest(unittest.TestCase):"""Test Cases for Glue Billing Unit Testing""" def test\_positive\_duration(self):  
    """Test successfully calculate duration  
    between 2 unix timestamp string"""  
    duration = billing.calculate\_duration(  
        "1535824800", "1535835600")  
    self.assertEqual(duration, 10800)

def test\_negative\_duration(self):  
    """Test successfully generate negative number"""  
    duration = billing.calculate\_duration(  
        "1535835600", "1535824800")  
    self.assertEqual(duration, -10800) In writing a unit testing, we import the unittest module, and the module of which we want to test (gluebilling.billing) module. Next we define the class to host the test cases (CalculateDurationTest) extending the base class (unittest.TestCase) Afterward within the CalculateDurationTest, we define the list of test cases in a object method format prefix with `test_`. We put in the input arguments, execute the function we want to test, and finally check the result using the assertEqual function coming from unittest.TestCase. if we run the unittest now it should be failing $ coverage run --source=gluebilling -m unittest discover -p "*utest*.py" -f -s gluebilling/ coverage with unittest result Now after having the red state, let us start implementing the function """Process records of ETL Job usage into the billing of fee"""from datetime import datetime as dt def calculate_duration(from_timestamp, to_timestamp):"""Returns duration in second between two timeprovided in unix timestamp"""from_dt = dt.fromtimestamp(float(from_timestamp))to_dt = dt.fromtimestamp(float(to_timestamp)) time\_delta = to\_dt - from\_dt  
return time\_delta.total\_seconds() Now we shall have all test cases in green state calculate_duration implementation completed ! Noticed that the screenshot above is a development environment setup with automated run of testing and coverage calculation, based on https://hackernoon.com/setting-up-python-dev-in-vs-code-e84f01c1f64b 002 — Calculate Fee (+ unit test) Following up the same technique above we are adding the calculate fee function as below : in billing.py """Process records of ETL Job usage into the billing of fee"""from datetime import datetime as dtfrom decimal import Decimal ... def calculate_fee(duration_in_second, dpu_num,minimum_duration, fee_dpu_hour):"""Returns a decimal of the fee incurred in USD,quantized into 8 digit behind comma""" charged\_duration = duration\_in\_second  
if charged\_duration < minimum\_duration:  
    charged\_duration = minimum\_duration

fee = charged\_duration \* dpu\_num \* Decimal(fee\_dpu\_hour) / 3600  
return fee.quantize(Decimal('0.00000000')) in billing_utest.py we shall add another class to host test cases for calculate_fee. """Unit Testing for Glue Billing Project"""import unittestfrom decimal import Decimal import gluebilling.billing as billing ... class CalculateFeeTest(unittest.TestCase):"""Unit Test for calculate_fee function""" def test_more_than_10_min(self):"""When usage is more than 10 min"""fee = billing.calculate_fee(800, 3, 600, "0.44")self.assertEqual(fee, Decimal("0.29333333")) Noticed that the dpu_hour value is set as instead of float, as this is the more accurate result of calculation if we further convert it into Decimal type. string “0.44” The reason why I separate the test cases for the 2 functions into different classes because the snake case requires the length of function capped into 30 characters, so to maintain readability we divide it into different classes for each function to test. pylint C0103 The coverage test result after implementing calculate_fee, notice that the coverage can be done better, it is left for the reader to exercise (see the red colour bar) 003 — PySpark Billing Calculation (+ functional test) This article is using similar basic concept from , the differences would be in the details where we focus our setup to be completely standalone (this shall be reflected in how we initialised the Spark Session, and how we prepare the test data) tutorial from David Illes Here is the version that we are going to use, we store it as pyspark_htest.py """PySparkTest is base class to do functional testing on PySpark"""import unittestimport loggingimport osfrom pyspark.sql import SparkSessionfrom pandas.testing import assert_frame_equal class PySparkTest(unittest.TestCase):"""BaseClass which setup local PySpark""" @classmethod  
def suppress\_py4j\_logging(cls):  
    """Supress the logging level into WARN and above"""  
    logger = logging.getLogger('py4j')  
    logger.setLevel(logging.WARN)

@classmethod  
def create\_testing\_pyspark\_session(cls):  
    """Returns SparkSession connecting to local context  
    the extrajava session is to generate   
    the metastore\_db and derby.log into .tmp/ directory"""  
    tmp\_dir = os.path.abspath(".tmp/")  
    return (SparkSession.builder  
            .master('local\[1\]')  
            .appName('local-testing-pyspark-context')  
            .config("spark.driver.extraJavaOptions",  
                    "-Dderby.system.home="+tmp\_dir)  
            .config("spark.sql.warehouse.dir", tmp\_dir)  
            .getOrCreate())

@classmethod  
def setUpClass(cls):  
    """Setup the Spark"""  
    cls.suppress\_py4j\_logging()  
    cls.spark = cls.create\_testing\_pyspark\_session()

@classmethod  
def tearDownClass(cls):  
    """Clean up the Class"""  
    cls.spark.stop()

@classmethod  
def assert\_dataframe\_equal(cls, actual, expected, keycolumns):  
    """Helper function to compare small dataframe"""  
    exp\_pd = expected.toPandas().sort\_values(  
        by=keycolumns  
    ).reset\_index(drop=True)  
      
    act\_pd = actual.toPandas().sort\_values(  
        by=keycolumns  
    ).reset\_index(drop=True)  
    return assert\_frame\_equal(act\_pd, exp\_pd) Noticed that this base class providing many useful class methods with the goal of : suppressing any logging into WARN only (PySpark using py4j for logging) building Spark Session in localhost with 1 core, and setting up the temporary metastore_db to be tidied up stored in .tmp/ directory, alongside with the derby.log (Just so that our workspace is tidy and clean) — create_testing_pyspark_session The setUpClass and tearDownClass is automatically called once. assert_dataframe_equal — receiving PySpark Dataframe, and then converting them all into Pandas, sorting it by the keys (because PySpark results does not maintain the order) then we use Pandas testing to compare the two dataframe. Let’s create the skeleton of the function (billing.py) alongside with the expected Schema definition of our input records and targetted output, the billing records. def get_usage_record_schema():"""Retruns StructType containing the Input UsageData Expected Schema"""return StructType([StructField("job_id", StringType(), False),StructField("type", StringType(), False),StructField("dpu", IntegerType(), False),StructField("from_unix_timestamp", StringType(), False),StructField("to_unix_timestamp", StringType(), False)]) def get_pricing_schema():"""Retruns StructType containing the InputPricing Data Expected Schema"""return StructType([StructField("type", StringType(), False),StructField("dpu_hour", StringType(), False),StructField("minimum_duration", IntegerType(), False)]) def get_billing_schema():"""Retruns StructType containing the Billing Schema"""return StructType([StructField("job_id", StringType(), False),StructField("type", StringType(), False),StructField("dpu", IntegerType(), False),StructField("from_unix_timestamp", StringType(), False),StructField("to_unix_timestamp", StringType(), False),StructField("dpu_hour", StringType(), False),StructField("minimum_duration", IntegerType(), False),StructField("duration", IntegerType(), False),StructField("fee", DecimalType(20, 8), False),]) def generate_billing(usage_df, pricing_df):"""Returns DataFrame of Fee from a DataFrame of Usage Records"""return None Next we go into the billing_ftest.py to prepare for our functional testing. """Functional Testing for Glue Billing Project"""from decimal import Decimalfrom pyspark.sql import Row, SQLContext import gluebilling.billing as billingimport gluebilling.pyspark_htest as pysparktest class GenerateBillingTest(pysparktest.PySparkTest):"""Test Cases for Generate Billing""" def generate\_usage\_data\_001(self):  
    """Generate usage data for testing it is a record of  
    AWS Glue ETL usage"""  
    rdd = self.spark.sparkContext.parallelize(\[  
        Row("JOB001", "etl", 3, "1535824800", "1535835600"),  
        Row("JOB002", "crawler", 3, "1535824800", "1535824850"),  
        Row("JOB003", "crawler", 3, "1535824800", "1535835600")  
    \])  
    schema = billing.get\_usage\_record\_schema()

    sqlctx = SQLContext(self.spark.sparkContext)  
    return sqlctx.createDataFrame(rdd, schema)

def generate\_pricing\_data\_001(self):  
    """Generate pricing data for testing it is a record of  
    AWS Glue ETL usage"""  
    rdd = self.spark.sparkContext.parallelize(\[  
        Row("etl", "0.44", 600),  
        Row("crawler", "0.20", 200)  
    \])  
    schema = billing.get\_pricing\_schema()

    sqlctx = SQLContext(self.spark.sparkContext)  
    return sqlctx.createDataFrame(rdd, schema)

def generate\_expected\_billing\_001(self):  
    """Generate expected billing"""  
    rdd = self.spark.sparkContext.parallelize(\[  
        Row("JOB001", "etl", 3, "1535824800", "1535835600",      
            "0.44", 600, 10800, Decimal("3.96")),  
        Row("JOB002", "crawler", 3, "1535824800", "1535824850",   
            "0.20", 200, 50, Decimal("0.03333333")),  
        Row("JOB003", "crawler", 3, "1535824800", "1535835600",   
            "0.20", 200, 10800, Decimal("1.80"))  
    \])  
    schema = billing.get\_billing\_schema()

    sqlctx = SQLContext(self.spark.sparkContext)  
    return sqlctx.createDataFrame(rdd, schema)

def test\_with\_set\_001(self):  
    """Using all 001 test data set"""  
    usage\_df = self.generate\_usage\_data\_001()  
    pricing\_df = self.generate\_pricing\_data\_001()  
    expected = self.generate\_expected\_billing\_001()

    actual = billing.generate\_billing(usage\_df, pricing\_df)

    self.assert\_dataframe\_equal(actual, expected, \["job\_id"\]) We prepare our input data in two function:1. generate_usage_data_0012. generate_pricing_data_001The technique being used here is by creating rdd using Row and paralellize method of SparkContext, and then combining with the defined Schema from the main script. We also prepare the expected output data inside billing_ftest.py, in function generate_expected_billing_001, with similar technique as we prepare the input data. The last component of billing_ftest.py is test_with_set_001, which is where the test being executed by combining the generation functions of input, and expected dataframe, and then we execute the main script function generate_billing, finally we do asssertion, by leveraging the helper assert method we define in pyspark_htest.py. Finally let’s complete the function implementation in billing.py. def generate_billing(usage_df, pricing_df):"""Returns DataFrame of Fee from a DataFrame of Usage Records""" duration\_udf = udf(calculate\_duration, IntegerType())

join\_data\_df = usage\_df.join(  
    pricing\_df,  
    usage\_df.type == pricing\_df.type  
).select(  
    usage\_df.job\_id, usage\_df.type,  
    usage\_df.dpu, usage\_df.from\_unix\_timestamp,  
    usage\_df.to\_unix\_timestamp,  
    pricing\_df.dpu\_hour, pricing\_df.minimum\_duration,  
    duration\_udf(  
        usage\_df.from\_unix\_timestamp,  
        usage\_df.to\_unix\_timestamp).alias("duration")  
)

fee\_udf = udf(calculate\_fee, DecimalType(20, 8))

billing\_df = join\_data\_df.select(  
    "job\_id", "type", "dpu", "from\_unix\_timestamp",  
    "to\_unix\_timestamp", "dpu\_hour", "minimum\_duration",      
    "duration",  
    fee\_udf(  
        join\_data\_df.duration, join\_data\_df.dpu,  
        join\_data\_df.minimum\_duration, join\_data\_df.dpu\_hour  
    ).alias("fee")  
)

return billing\_df We are wrapping our calculate_fee and calculate_duration function into udf, as this is the the type that can be passed into pyspark. The 1st argument is the function to be wrapped, while the 2nd argument is the expected return type. Then we use this in the SELECT part of the PySpark SQL DataFrame to generate the 2 new columns duration and fee accordingly. Conclusion That’s it now we have implemented the functional test, I differentiate this with the calculate_fee and calculate_duration because of the speed on how this test being executed, it requires several seconds to start up the pyspark hence its worthed to be group differently. This open up options for us to filter out what kind of test do we want to run on save (in case we are automating the development experience)

Stories, News & Mentions about SPARKLOOP

Sparkloop’s Stories on HackerNoon