Hive is an email \u0026 SMS marketing CRM that helps brands sell more while keeping their customers happy and engaged

Hive.co

Looks like it’s part 2 of my series on Data Quality! In a , I explored Airbnb’s strategy for enhancing data quality through incentives. They implemented a single score and clear scoring criteria to establish a common understanding between data producers and consumers, fostering a genuine sense of ownership. previous post https://hackernoon.com/data-quality-score-one-score-to-rule-them-all?embedable=true Now, Lyft is taking a distinct approach, not attempting the same thing differently, but rather focusing on different aspects of data quality. And, Lyft’s strategy complements Airbnb’s efforts. While I regard Airbnb’s DQ score (or any similar score) as an effective means to consolidate various attempts at elevating data quality, Lyft is tackling this challenge from a different angle. The Airbnb’s DQ score serves as a valuable tool for providing a concrete visualization of data quality. In essence, any initiative to enhance data quality should manifest a discernible impact on this score. On the other hand, Lyft presents one possible initiative to proactively enhance quality by testing and validating data against specific quality criteria. https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6?source=post_page-----415baad6c95b--------------------------------&embedable=true Fundamentally, it’s a different point in the data quality lifecycle. Introducing a mechanism to improve quality necessitates the ability to measure it initially. So, while ’s focus lies on and data quality, relying on the producer’s interest to enhance this quality and “look good,” Lyft takes a different approach. Lyft places emphasis on actively and data quality, providing both producers and consumers with the means to effectively improve and control the quality. Airbnb measuring observing testing validating Collectively, these approaches provide a comprehensive strategy to address and enhance throughout its . data quality lifecycle For this reason, I was particularly interested in taking a closer look at Lyft’s approach. Another factor that intrigued me is testing, more specifically, contract testing, which has been used for many years now in basic software engineering with the emergence of microservice architecture. Data contracts, however, are something more recent in the domain of data engineering and are seen as the pinnacle or one of the ultimate steps to put in place on the path to building high-quality data pipelines. This is why I wanted to examine Lyft’s approach in more detail and explore some potential parallels. As mentioned, the approaches taken by Airbnb and Lyft are complementary and aim to achieve the same goal: improving data quality. has developed the DQ score, which focuses on measuring and enhancing 4 distinct : Airbnb aspects of data quality DQ Score has guiding principles, including full coverage, automation, actionability, multi-dimensionality, and evolvability. It has dimensions like Accuracy, Reliability, Stewardship, and Usability. Lyft’s Verity is a platform designed to enhance data quality across 5 dimensions Defines data quality as the measure of how well data can be used as intended, covering aspects like . semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness It’s easy to draw parallels between the 5 data quality aspects improved by Lyft’s Verity and the 4 data quality dimensions measured by the Airbnb’s DQ score. For example, aspects like Timeliness would certainly contribute to the DQ score’s Reliability, while Accuracy would be dependent on semantic correctness, completeness, and uniqueness of the data. The Usability score, on the other hand, is influenced by the consistency of the data among other factors. Lyft’s Verity focuses on defining related to . It follows a test-first and validation approach, whereas Airbnb’s unified DQ score emphasizes evaluating data quality through various dimensions. checks semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness If we wanted to incorporate the DQ score into this last schema, it would be on the side of the Alert/Debug steps. Airbnb’s DQ score uses different signals to assess data quality across the . Accuracy, Reliability, Stewardship, and Usability aspects We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). As discussed earlier, there are likely overlaps between Airbnb’s DQ score and Verity. While Airbnb focuses on pushing data quality to the right, emphasizing measurement and scoring, Lyft’s Verity takes a proactive approach by shifting the check definition configurations, testing, and validation processes to the left, emphasizing proactive improvement of data quality. Now, my primary interest lies to the left, in the check definition configurations, testing, and validation aspects. How Lyft integrates data quality testing into its processes? Let’s first examine the execution paths. Currently, Lyft’s Verity is primarily focused on ensuring the quality of data stored in its Hive data warehouse. However, there are plans to expand its capabilities to support other data sources in the future. Note that while they refer to Hive as a data workhouse, they actually utilize it as a hybrid data storage solution, operating as a data warehouse for structured, processed, and cleansed data (silver layer) while also serving as a data lake for raw event data (bronze layer). Poor data quality in Hive caused tainted experimentation metrics, inaccurate machine learning features, and flawed executive dashboards. Verity’s checks can be integrated into an Airflow DAG to ensure that only high-quality raw data is processed and stored in Hive as derived data. Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed inside or . Airflow Flyte … The VerityAirflowOperator can be used in a blocking fashion to halt a DAG upon a check failure, preventing bad data from ever reaching production. This utilizes the “ ” pattern: we create data in a staged schema, verify the data with a blocking operator, then promote it to production if it passes quality checks. Stage-Check-Exchange Checks can also be performed manually or scheduled automatically to verify both raw and derived data. Verity Scheduled Checks are isolated from any data orchestration engine, so they still run even if Airflow or Flyte are completely down. This remedies a common problem of checks not alerting because the Airflow Task never ran. So, there are essentially 3 primary ways to trigger a check: as part of an Airflow DAG, manually, or scheduled through the Verity platform/UI. I don’t believe that current checks can be integrated into real-time streaming pipelines (such as Flink + Kafka) to validate for example data as it enters Hive (or even earlier). Implementing this type of real-time check would enable prompt detection of discrepancies, leading to reduced storage and processing costs and overall data quality enhancement. Well, to be thorough, Verity checks are managed through an API server, which could be used to trigger checks programmatically through some APIs. Verity API Server — This service handles all the external APIs regarding running checks as well as persisting and retrieving their results. The API Server does not execute any checks, but rather writes a message to our Check Queue, which utilizes (SQS). SimpleQueueService So, maybe, you could potentially trigger these jobs in a more real-time fashion, such as from a streaming job or even, as a long stretch, by integrating with Hive change data capture (CDC). However, when executed outside of Airflow, these jobs would not be able to block the data processing job; instead, they would generate asynchronous alerts pushed to the Check Queue. Some consumers would prefer to have data processing delayed when a check fails, while others would rather proceed and receive an alert. Now, let’s look at these data quality tests. Here’s an example that checks whether is never null. This is accomplished through a combination of query and condition components. rider_events.session_id core rider events session_id is not null: # check name
  metadata:
    id: 90bde4fa-148b-4f06-bd5f-f15b3d2ad759
    ownership_slack: #dispatch-service-dev
    tags: [rides, core-data, high-priority]
  query:
    type: dsl
    data_source_id: hive.core.rider_events
    filters:
      - session_id = null
  condition:
    type: fixed_threshold
    max: 0
  notifier_group:
    pagerduty_policy: dispatch-service
    email: dispatch-service@lyft.pagerduty.com Verity is primarily focused on defining and enforcing data quality checks rather than defining complete data schemas. Schema validation is not a novel concept. Several methods exist for defining event data schemas in event-based systems, such as JSON Schema, Protocol Buffers, Avro, or storage formats like Parquet. The optimal choice depends on your tech stack, usage, and specific requirements. While data schemas are valuable for defining the overall structure of data objects or table rows, they fall short in capturing more sophisticated validation checks specific to consumers, such as data distribution, business rules, SLAs, and thresholds. Data contracts go beyond schema validation, which focuses on identifying . I personally find that JSON Schema offers here a more suitable and readable option, effectively separating these structural and syntactical validation capabilities from serialization or storage concerns. syntactical errors However, to address and enhance data accuracy, having an effective means of defining data checks allows for defining the other facet of data contracts. semantic errors This is where Verity DSL comes into play. Beyond semantical validation, data contracts offer another crucial aspect that deserves attention. From a syntactical standpoint, validation checks remain consistent regardless of the consumer or producer involved. The set of validation rules is not tied to any specific consumer or producer and can be defined once and for all as a single schema. However, the Verity data contract DSL offers a finer granularity defining small independent rules, which is particularly well-suited for this context: the semantic meaning and usage of the data vary depending on the specific consumer. Additionally, not all consumers need to utilize all of the properties of an object. Their expectations differ. This doesn’t imply they are contradictory (which would certainly be an issue), but rather complementary and distinct ones. Therefore, allowing all consumers to establish unique rules that, when combined collaboratively, can provide a comprehensive understanding of the semantic significance of data quality is quite crucial. And it’s this aspect that particularly resonates with me. Bear with me, this may seem like a stretch, but from my perspective, it’s worth mentioning. :) collaborative Data exchange enables different teams (producers and consumers) to collaborate effectively. Establishing a shared understanding of these data exchanges is paramount, just like APIs in traditional software development. In microservice architectures, a known as consumer-driven contracts (CDC) has emerged, where consumers define the expected behavior of APIs provided by producers. Producers are responsible for verifying these contracts before releasing new versions. collaborative testing approach I think data contracts share a similar collaborative spirit. Even though data validation is performed on real data, rather than at release time, and does not block releases, it’s based on cooperation and encourages teamwork between data producers and consumers. I strongly believe that this collaborative approach is key to improving data quality and should be further integrated into the process. Well, I’m a big fan of drawing parallels… Notice actually that this collaborative aspect is something mentioned also as part of Lyft’s verity charter. The VerityUI provides a streamlined data discovery experience via the Verity Homepage. Our full-text search on the Check Definition Metadata lets users see all the checks currently being enforced and their Check Results. This has useful aggregations like owning team, table name, and tags. I’m not entirely clear on how data contract issues are shared between consumers and producers through the Verity platform UI, but I definitely recognize the significance of collaboration through the dashboards: The producer of a data product interface can confidently ensure that they’re not inadvertently causing downstream breakages that they hadn’t anticipated. The consumer of the interface can rest assured that their reliance on the interface is not and will not be compromised. While Verity is a remarkable tool for defining data quality checks, it’s unfortunately not open-source. Fortunately, there’s another open-source data quality framework called Soda Core that provides similar functionality. Soda Core is a free and open-source command-line tool and Python library that enables data engineers to test data quality. It utilizes user-defined input to generate SQL queries that run checks on datasets in a data source to find invalid, missing or unexpected data. When checks fail, they surface the data that you defined as “bad” in the check. https://medium.com/@seckindinc/soda-core-the-simplest-open-source-data-reliability-tool-f35ae5b576d9?source=post_page-----415baad6c95b--------------------------------&embedable=true During a scan of a dataset, Soda Core evaluates the predefined checks to identify invalid, missing, or unexpected data. Here’s the equivalent check using Soda.core syntax for the Verity DSL test that was previously defined. name: rider_events_session_id_check
source: hive
query: SELECT * FROM rider_events WHERE session_id IS NULL;
raise_alert: true
threshold: 10
action: slack
message: "There are more than 10 rows that are null for the 'session_id' property in the 'rider_events' table. Please investigate this issue." soda run --check checks/rider_events_session_id_check.yaml Soda Core is a powerful tool for ensuring the quality of your data. It can help you to identify and fix data problems early, before they can cause issues for your business. It’s worth noting that Soda Core can also facilitate data quality checks for streaming data by seamlessly integrating with Spark DataFrames. While Verity’s data quality checks for Hive are applied to static datasets, the checks for streaming data need to be more lightweight and efficient. Data would typically be processed in small batches of events, with very low latency, making them suitable for real-time checks and specific use cases like anomaly detection. Finally, let’s mention that there are other data validation tools available, such as DeeQu, Great Expectations, … As we wrap up, I hope you have a clearer understanding of the steps you can take to enhance your data quality journey. We’ve seen two distinct approaches to data quality improvement, each with its own strengths and methodologies. One focuses on increasing visibility and observability, motivating data producers to raise the quality bar. The other prioritizes elevating the quality standard through a testing and validation-first approach. Both are complementary. Verity is not merely a domain-specific language (DSL) for defining data checks; it’s a centralized platform that empowers data practitioners to collaborate effectively. This platform helps producers and consumers align on data quality expectations, including format, structure, and accuracy. Verity’s data contract management capabilities could (is?) further enhanced by integrating with a broader set of features, such as metadata management and discovery, to address more sophisticated data quality needs. Similar to Airbnb’s DQ score, Lyft’s Verity fosters a collaborative feedback loop between data producers and consumers. By incentivizing and empowering each team to take ownership of data quality, Verity cultivates a supportive environment where data quality continuously improves. , Found this article useful? Follow me on Linkedin , and ! Please 👏 this article to share it! Hackernoon Medium Also published . here

Advancing Data Quality: Exploring Data Contracts with Lyft

ডেটা গুণমান উন্নত করা: Lyft-এর সাথে ডেটা চুক্তিগুলি অন্বেষণ করা

Verbesserung der Datenqualität: Erkundung von Datenverträgen mit Lyft

Mejora de la calidad de los datos: exploración de los contratos de datos con Lyft

Améliorer la qualité des données : explorer les contrats de données avec Lyft

डेटा गुणवत्ता को आगे बढ़ाना: Lyft के साथ डेटा अनुबंध की खोज करना

データ品質の向上: Lyft とのデータ契約の検討

데이터 품질 향상: Lyft를 통한 데이터 계약 탐색

Avançando na qualidade dos dados: explorando contratos de dados com Lyft

Повышение качества данных: изучение контрактов на передачу данных с Lyft

Veri Kalitesini Geliştirme: Lyft ile Veri Sözleşmelerini Keşfetmek

Nâng cao chất lượng dữ liệu: Khám phá hợp đồng dữ liệu với Lyft

提高数据质量：与 Lyft 探索数据合同

There are circumstances when tasks (Spark action, e.g. save, count, etc) in a PySpark job can be spawned on separate threads. Doing so, optimizes distribution of tasks on executor cores. From , Spark docs If the jobs at the head of the queue don’t need to use the whole cluster, later jobs can start to run right away, but if the jobs at the head of the queue are large, then later jobs may be delayed significantly. By default, Spark’s scheduler runs jobs in FIFO fashion. Each job is divided into “stages” (e.g. map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc. Threading tasks addresses this particular situation: through, if the jobs at the head of the queue are large, then later jobs may be delayed significantly, Spark assigns tasks between jobs in a “round robin” fashion, so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can start receiving resources right away and still get good response times, without waiting for the long job to finish. Simple But Very Common Use Case Saving multiple results to parquet files is a common action. Here’s how to thread three action. write.parquet An important reminder is to set set('spark.scheduler.mode','FAIR') in the sparkContext. threading conf = SparkConf().\
            setAppName(app_name).\
            set( , ).\
            set( , ).\
            set( , ) sc = SparkContext(conf=conf) :
        sc._jvm.org.apache.hadoop.hive.conf.HiveConf()
        sqlCtx = sqlContext = HiveContext(sc) py4j.protocol.Py4JError:
        sqlCtx = sqlContext = SQLContext(sc) sc, sqlContext sc.setLocalProperty( , str(pool_id))
    df.repartition( ).write.parquet(path)
    sc.setLocalProperty( , )
sc, sqlContext = set_spark_context( ) path1 = path2 = path3 = df1_write = threading.Thread(target=saveParquet, args=(df1,path1, ,sc,))
df2_write = threading.Thread(target=saveParquet, args=(df2,path2, ,sc,))
df3_write = threading.Thread(target=saveParquet, args=(df3,path3, ,sc,))
df1.start()
df2.start()
df3.start()
df1.join()
df2.join()
df3.join() import : def set_spark_context (app_name=None) 'spark.hadoop.mapreduce.output.fileoutputformat.compress' 'false' 'spark.sql.parquet.compression.codec' 'uncompressed' 'spark.scheduler.mode' 'FAIR' # need to set this on the sparkContext try except return #function to be called on spawned threads* : def saveParquet (df,path,pool_id,sc) "spark.scheduler.pool" 1 "spark.scheduler.pool" None "Simple TASK" # Do processing here to produce three dataframes to be written as parquets. # The tree results are df1, df2, df3 "/path/1" "/path/2" "/path/3" 1 2 3 When this job is submitted on the cluster, at first it will show 1 scheduler pool. However, at the stage where the actual snippet above is already being interpreted, it will show the 3 pools spawned and the active pools currently being processed. Hopefully, this is something useful for us trying to speed up running jobs.

Threaded Tasks in PySpark Jobs

State of Bitcoin (Fall 2018)

Not too long ago, Howtotoken started the (details about it ) and the core idea was simple: build a moderated list of 10,000 questions for anyone who wants to learn more about the blockchain and cryptocurrency in general, and this challenge is already running into its fourth week: 10KQA Challenge here Week 1 report Week 2 report Week 3 report One of the most interesting questions asked this week was in a discussion about Bitmain starting to sell its , and how this will affect the profits when mining Ethereum. Thanks to the experts, we have received an interesting answer that we will now share with the community in this weekly report. But before we dive into that, let’s go over some of the other interesting questions and answers that were talked about this week. Antminer E3 Why are cryptocurrencies acceptable under Sharia Law? As we all know, last week Islamic countries were allowed to use crypto by Muhammad Abu Bakar, but the most interesting question here is why cryptocurrency could be considered Halal and here are some of the reasons: Cryptocurrencies are not controlled by a centralized system, which is what makes them more secure than other current financial systems out there. Cryptocurrencies are widely accepted; hence, it is money. However, Abu Bakar notes that while cryptocurrencies are halal in most cases, they are not halal in all. For instance, they are not halal in jurisdictions were they are banned. He also stated that it is not advisable to use cryptocurrencies for investment purposes. Regarding the current situation with the high volatility of Bitcoin and other cryptocurrencies, there was an interesting question asked about stable coins. Which USD-backed stable coin is the most reliable and why? , suggests considering if you need a replicative asset for dollars in the world of crypto. Jonathan Ponciano TrueUSD The main reason for it is the legal framework developed by the issuer to protect token holders. It does this by creating a strong link between tokens and underlying dollars. The product is being developed in partnership with banks, two Washington D.C. law firms, a crypto hedge fund, and a New York investment firm. This platform is called TrustToken, where all operations with this token are stored. The next questions are in regards to mining, as more and more news about this field is currently the hot-topic of conversation. Security and hidden miners still plague users, but there are some basic recommendations that you can follow to avoid that. How do I prevent hidden mining on my computer? , a co-founder and an active entrepreneur, describes six ways to guarantee that no one will mine from your computer in an article for his website. Adash Verma Fossbytes He says that most websites are using Coin Hive for mining. Here are the instruments to block it: 1. Use the No Coin Chrome extension “As soon as you visit a website, No Coin will detect and show you if any such activity is going on. You’ll see a red symbol as shown in the screenshot below. While this extension blocks any such activity, it also allows you to whitelist a website for a period of time.” 2. Use the minerBlock Chrome extension The principle of this extension is the same. However, currently this service has less miner domains on their list. 3. Block coin mining domains in hosts file “This is the manual way of blocking particular domains that you find harmful or irritating. Due to this method of blocking, your browser won’t be able to connect to these domains. We can edit the hosts file and redirect it to 0.0.0.0. For Windows, navigate to C:\Windows\System32\drivers\etc and edit the hosts document to add 0.0.0.0 coin-hive.com at the end. Please note that this change blocks the mining scripts which are hosted by coin-hive.com. In case you come across other mining script domains, you can add them further down the line.” 4. Block domains in AdBlocker Go to the “Customize” section, then direct yourself to “Block an Ad by its URL” and insert the following link: https://coin-hive.com/lib/coinhive.min.js 5. Use NoScripts in Firefox NoScripts might be an option if you use Firefox. However, it is pretty aggressive and usually blocks too many websites. 6. The Opera browser “With the release of the Opera 50 web browser, the company brought in-built functionality to block CoinHive scripts and cryptocurrency mining techniques. This will allow users to make sure that their computers doesn’t go crazy whenever any website that is using the digital coin mining technique is encountered. The users can find this option in Settings (Preferences on macOS) > Basic > Block ads, and under the Recommended lists of ad filters click NoCoin.” How do you spot a website that is mining using your computer? , an author of , has the solution for you. Eric Limer popularmechanics.com One of the most prominent symptoms of “cryptojacking” is an immediate and abrupt change in the overall performance of your computer. It will usually slow down your machine significantly. “If you notice either of these conditions and get suspicious, just pull up your Task Manager on Windows (Ctrl+Alt+Delete) or Activity Monitor on a Mac (Command + Space and search for “Activity Monitor”). If your web browser is pulling a particularly high percentage of CPU power (close to all of it), there might be a cryptojacker at play. If you suspect a certain website is to blame, you can test your theory by closing the tab and monitoring the results or, in Google Chrome, go to the Menu, then More Tools, then Task Manager to check CPU usage by tab.” What is the effect of Bitmain’s ASIC-miner E3 on mining process of ETH? The first bulk of miners are already sold out and will be delivered to its users in the second half of July 2018, but now we should take the time to calculate and analyze the characteristics and risks that need to be taken into consideration. The Antminer E3 is the first Ethereum-based ASIC miner. Prior to this, GPUs like the RX570 were the most cost-effective method to mining Ethereum, offering 1MH/s at a . The E3 in comparison offers 1MH/s at $4.40, so it’s around 2.27x more cost effective. Interestingly enough, it isn’t more efficient in power usage; the RX570 and GPUs are more efficient in that regard. cost of around $12 1080 Ti It’s great to see what the ROI is for this miner: Full-size table here: https://www.anythingcrypto.com/bitcoin-mining-calculators The main risks for miners is that the plan for changing to the hybrid proof-of-work/proof-of-stake algorithm called Casper (which could potentially lead to Ethereum becoming fully proof-of-stake based) could make it so that you are no longer able to mine it. The real question that needs to be asked is when this will be implemented, as ASICs are going to become a problem in several months. However, Casper may not be implemented until months or even years after that — so a short-term solution may be needed (although, because the E3 isn’t drastically more efficient than existing GPUs, this may not even be necessary). To join our challenge and get new updates, follow us on: Telegram: TENKQA Twitter: @howtotoken I will be appreciative of all participants, ideas, and discussions. It is my hope that we will all be able to get something positive from the process and that unforgettable feeling of having a flow. We are also looking for moderators to join our challenge who will learn with us and search for new questions and answers regarding the blockchain and cryptocurrency in these categories: Core Blockchain Development Profits (trading, ICO, mining) Real Disruptive Models Laws and Regulations ** **👏 We need your claps to reach more people since this is a new publication. Please spread this story with a tweet, Facebook share, etc. About the author: — Founder of Geekforge.io and Howtotoken.com. Interviewing the top 10,000 worldwide experts who reveal the biggest issues on the way to technological singularity. Join my . Kirill Shilov #10kqachallenge: GeekForge Formula

Will the new Bitmain Antminer E3 make you hit the jackpot on ethereum mining?

In my previous story , I wrote about utilising the CPU power of the user with his/her consent and benefiting the publisher. Monetise With Your User’s CPU Power However, there are web services which are making users mine without their knowledge. Recently , was found doing the same. thepiratebay has made it easy by releasing a javascript library. Just inserting a couple of lines of code will start the mining process. This will cause a spike in cpu usage and increase in temperature of the computer. CoinHive There could be other sites doing the same. If you see increase in your CPU activity even when you are not doing any costly computation like image editing or playing video games, there is a possibility your CPU power is being utilised without your knowledge. This is with respect to browser. But, people can build viruses that can infect the operating system of mobile/laptops to mine without user’s knowledge. This could be the future of viruses. You can go ahead an ban this URL, to avoid being mined using coin-hive’s library. https://coin-hive.com/lib/coinhive.min.js The other alternative is to use a , since the library name and url can be changed. Hopefully antivirus softwares would deploy a patch that would detect such mining viruses soon. script blocker Update: published a browser extension, that can block miners(For now just CoinHive). Rafael Keramidas https://github.com/keraf/NoCoin Follow Hackernoon and me ( ) for more stories. I am also writing a book to raise awareness on the Blue Whale Challenge, which has claimed lives of many teenagers in several countries. It is intended to help parents understand the threat of the dark web and to take actions to ensure safety of their children. The book is available for pre-order on Amazon. The title will be released on 20th of this month. Febin John James Fight The Blue Whale

You could be a victim of illegitimate mining

I always wonder if the internet will be ever free from pop ups, banner ads, redirection ads, etc. One reason I love medium is for building an ad free platform. But the problem is for small to medium sized publishers monetisation is difficult and are totally dependent on such ads. While going through HackerNews, I found a service that could help the same. helps website owners monetise through their CPU power. You get paid in Monero( A cryptocurrency). CoinHive is experiencing unusually high traffic now, it might be a bit slow. You can monetise in the following ways. CoinHive Captcha You can easily implement the captcha like service where your user’s CPU has to solve a number of hashes in order to be verified. This comes with the benefit of earning you money. Short links When you want to forward the user to another page, you can use a short link. When the user clicks on the short link, it waits until the mining is complete. You can get a demonstration by clicking . here Custom Functions You can make a custom function , for example Medium can implement this in their clapping feature. When the user claps, the user’s CPU will solve a number of hashes and the resulting money can be paid to the user directly. Start the mining process by loading the CoinHive library and using calling the start function. <script src="https://coin-hive.com/lib/coinhive.min.js"></script><script>var miner = new CoinHive.User('<site-key>', 'john-doe');miner.start();</script> Later, check the balance by call the following API which takes name and secret key as arguments. url "https://api.coin-hive.com/user/balance?name=john-doe&secret=<secret-key>"# {success: true, name: "john-doe" balance: 4096} This can be even used in apps and games. Because most apps on the app store has hopeless number of advertisements. However, I feel the money earned compared to ads might be less. May be in the long term if the cryptocurrencies price rises you will hit a jackpot. Follow and me ( ) for more stories. Hacker Noon Febin John James I am also writing a book to raise awareness on the Blue Whale Challenge, which has claimed lives of many teenagers in several countries. It is intended to help parents understand the threat of the dark web and to take actions to ensure safety of their children. The book Fight The Blue Whale is available for pre-order on Amazon. The title will be released on 20th of this month.

Monetise With Your User’s CPU Power

Discard Manual Scheduling With DolphinScheduler 3.1.x Cluster Deployment

Pursuing data-driven management at a consumer financing company, we aim to serve four needs in our data platform development: monitoring and alerting, query and analysis, dashboarding, and data modeling. For these purposes, we built our data processing architecture based on Greenplum and CDH. The most essential part of it is the risk data mart. Risk Data Mart:  Apache Hive I will walk you through how the risk data mart works following the data flow: Our is imported into for real-time analysis to generate BI reports. Part of this data also goes into for queries and modeling analysis. business data Greenplum Apache Hive Our are updated into in real-time via message queues, while Elasticsearch ingests data into Hive for analysis, too. risk control variables Elasticsearch The is passed from to Hive for risk control analysis and modeling. risk management decision data MongoDB So these are the three data sources of our risk data mart. This whole architecture is built with CDH 6.0. The workflows in it can be divided into real-time data streaming and offline risk analysis. : Real-time data from Apache Kafka will be cleaned by Apache Flink, and then written into Elasticsearch. Elasticsearch will aggregate part of the data it receives and send it for reference in risk management. Real-time data streaming : Based on the CDH solution and utilizing Sqoop, we ingest data from Greenplum in an offline manner. Then we put this data together with the third-party data from MongoDB. Then, after data cleaning, we pour all this data into Hive for daily batch processing and data queries. Offline risk analysis To give a brief overview, these are the components that support the four features of our data processing platform: As you see, Apache Hive is central to this architecture. But in practice, it takes minutes for Apache Hive to execute analysis, so our next step is to increase the query speed. What is Slowing Down Our Queries? Huge data volume in external tables Our Hive-based data mart is now carrying more than 300 terabytes of data. That's about 20,000 tables and 5 million fields. Putting them all in external tables is maintenance-intensive. Plus, data ingestion can be a big headache. Big flat tables Due to the complexity of the rule engine in risk management, our company invests a lot in the derivation of variables. In some dimensions, we have thousands of variables or even more. As a result, a few of the frequently used flat tables in Hive have over 3000 fields. So you can imagine how time-consuming these queries can be. Unstable interface Results produced by daily offline will be regularly sent to our Elasticsearch clusters. (The data volume in these updates is huge, and the call of the interface can get expired.) This process might cause high I/O and introduce garbage collection jitter, and further leads to unstable interface services. batch processing In addition, since our risk control analysts and modeling engineers are using Hive with Spark, the expanding data architecture is also dragging down query performance. A Unified Query Gateway We wanted a unified gateway to manage our heterogeneous data sources. That's why we introduced . Apache Doris But doesn't that make things even more complicated? Actually, no. We can connect various data sources to Apache Doris and simply conduct queries on it. This is made possible by the feature of Apache Doris: It can interface with various data sources, including data lakes like Apache Hive, Apache Iceberg, and Apache Hudi, and databases like MySQL, Elasticsearch, and Greenplum. That happens to cover our toolkit. Multi-Catalog We create Elasticsearch Catalog and Hive Catalog in Apache Doris. These catalogs map to the external data in Elasticsearch and Hive, so we can conduct federated queries across these data sources using Apache Doris as a unified gateway. Also, we use the to allow data communication between Spark and Doris. So basically, we replace Apache Hive with Apache Doris as the central hub of our data architecture. Spark-Doris-Connector How does that affect our data processing efficiency? : This is about real-time data querying. We access our real-time data in Elasticsearch clusters using Elasticsearch Catalog in Apache Doris. Then we perform queries directly in Apache Doris. It is able to return results within seconds, as opposed to the minute-level response time when we used Hive. Monitoring & Alerting : As I said, we have 20,000 tables in Hive so it wouldn't make sense to map all of them to external tables in Hive. That would mean a hell of maintenance. Instead, we utilize the Multi Catalog feature of Apache Doris 1.2. It enables data mapping at the catalog level, so we can simply create one Hive Catalog in Doris before we can conduct queries. This separates query operations from the daily batching processing workload in Hive, so there will be less resource conflict. Query & Analysis : We use Tableau and Doris to provide dashboard services. This reduces the query response time to seconds and milliseconds, compared with several minutes back in the "Tableau + Hive" days. Dashboarding : We use Spark and Doris for aggregation modeling. The Spark-Doris-Connector allows mutual synchronization of data, so data from Doris can also be used in modeling for more accurate analysis. Modeling Cluster Monitoring in the Production Environment We tested this new architecture in our production environment. We built two clusters. : Configuration Production cluster: 4 frontends + 8 backends, m5d.16xlarge Backup cluster: 4 frontends + 4 backends, m5d.16xlarge This is the monitoring board: As is shown, the queries are fast. We expected that it would take at least 10 nodes but in real cases, we mainly conduct queries via Catalogs, so we can handle this with relatively small cluster size. The compatibility is good, too. It doesn't rock the rest of our existing system. Guide to Faster Data Integration To accelerate the regular data ingestion from Hive to Apache Doris 1.2.2, we have a solution that goes as follows: Main components: DolphinScheduler 3.1.4 SeaTunnel 2.1.3 With our current hardware configuration, we use the Shell script mode of DolphinScheduler and call the SeaTunnel script on a regular basis. This is the configuration file of the data synchronization tasks: env{
  spark.app.name = "hive2doris-template"
  spark.executor.instances = 10
  spark.executor.cores = 5
  spark.executor.memory = "20g"
}
spark {
  spark.sql.catalogImplementation = "hive"
}
source {
  hive {
    pre_sql = "select * from ods.demo_tbl where dt='2023-03-09'"
    result_table_name = "ods_demo_tbl"
  }
}
 
transform {
}
 
sink {
  doris {
      fenodes = "192.168.0.10:8030,192.168.0.11:8030,192.168.0.12:8030,192.168.0.13:8030"
      user = root
      password = "XXX"
      database = ods
      table = ods_demo_tbl
      batch_size = 500000
      max_retries = 1
      interval = 10000
      doris.column_separator = "\t"
    }
} This solution consumes fewer resources and memory but brings higher performance in queries and data ingestion. Less storage costs : The original table in Hive had 500 fields. It was divided into partitions by day, with 150 million pieces of data per partition. It took to store in HDFS. Before 810G : For data synchronization, we call Spark on YARN using SeaTunnel. It can be finished within 40 minutes, and the ingested data only takes up of storage space. After 270G Less memory usage & higher performance in queries : For a GROUP BY query on the foregoing table in Hive, it occupied 720 Cores and 1.44T in YARN, and took a response time of . Before 162 seconds : We perform an aggregate query using Hive Catalog in Doris, , and receive the result after . We also try and put the table in Doris and conduct the same query in Doris itself, that only takes . After set exec_mem_limit=16G 58.531 seconds 0.828 seconds The corresponding statements are as follows: Query in Hive, response time: 162 seconds select count(*),product_no   FROM ods.demo_tbl where dt='2023-03-09'
group by product_no; Query in Doris using Hive Catalog, response time: 58.531 seconds set exec_mem_limit=16G；
select count(*),product_no   FROM hive.ods.demo_tbl where dt='2023-03-09'
group by product_no; Query in Doris directly, response time: 0.828 seconds select count(*),product_no   FROM ods.demo_tbl where dt='2023-03-09'
group by product_no; Faster data ingestion : The original table in Hive had 40 fields. It was divided into partitions by day, with 1.1 billion pieces of data per partition. It took to store in HDFS. Before 806G : For data synchronization, we call Spark on YARN using SeaTunnel. It can be finished within 11 minutes (100 million pieces per minute ), and the ingested data only takes up of storage space. After 378G Summary The key step to building a high-performing risk data mart is to leverage the Multi Catalog feature of Apache Doris to unify the heterogeneous data sources. This not only increases our query speed but also solves a lot of the problems coming with our previous data architecture. Deploying Apache Doris allows us to decouple daily batch processing workloads with ad-hoc queries, so they don't have to compete for resources. This reduces the query response time from minutes to seconds. We used to build our data ingestion interface based on Elasticsearch clusters, which could lead to a garbage collection jitter when transferring large batches of offline data. When we stored the interface service dataset on Doris, no jitter was found during data writing and we were able to transfer 10 million rows within 10 minutes. Apache Doris has been optimizing itself in many scenarios including flat tables. As far as we know, compared with ClickHouse, Apache Doris 1.2 is twice as fast in the SSB-Flat-table benchmark and dozens of times faster in the TPC-H benchmark. In terms of cluster scaling and updating, we used to suffer from a big window of restoration time after configuration revision. But Doris supports hot swap and easy scaling out, so we can reboot nodes within a few seconds and minimize interruption to users caused by cluster scaling. (One last piece of advice for you: If you encounter any problems with deploying Apache Doris, don't hesitate to contact the Doris community for help, they and a bunch of SelectDB engineers will be more than happy to make your adaption journey quick and easy.)

HIVE.CO

hive.co

-- employees

Since n.d.

COMPANY RANKING

HIVE.CO

EVERGREEN INDEX #4540

Representing this company? Click here to claim and customize this page!

Hive.co WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

HIVE.CO

hive.co

-- employees

Since n.d.

COMPANY RANKING

HIVE.CO

EVERGREEN INDEX #4540

RELATED COMPANIESRANK

STORIES ABOUT HIVE.CO

Representing this company? Click here to claim and customize this page!

Hive.co WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!