Bazaarvoice

bazaarvoice

Let's learn about via these 49 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Data Storage /Learn Repo 1. Data Lakehouses: The New Data Storage Model Data lakehouses are quickly replacing old storage options like data lakes and warehouses. Read on for the history and benefits of data lakehouses. 2. Win $10k AKT in the Filebase + Akash Hackathon Join the Filebase + Akash Hackathon that is from 1 September to 30 October and compete to win up to $10,000 AKT and up to $250 in free object storage. 3. Kafka Storage Design  -  Making File Systems Cool Again! What makes Kafka so Fast? A Deep Dive into Kafka Storage Internals. 4. How To Fix: The File is Too Large for the Destination File System Error Have you ever encountered this error “the file is too large for the destination file system” while moving files from your windows PC to USB flash drive or external hard drive? 5. Top 10 Javascript File Managers to Use in 2022 A brief intro to 10 file managers for software developers 6. Mounting Web3 Storage as a Folder on Your Desktop Learn in this post how to mount Web3 Decentralized Storage as a folder on your desktop. Drag and Drop onto Web3 in seconds using Filebase and Mountain Duck. 7. Hadoop Across Multiple Data Centers Hadoop cluster across multiple data centers 8. Optimized Metadata Loading Process on ShardingSphere: A Technical Deep-Dive The powerful database middleware ShardingSphere core functions such as data sharding, encryption and decryption are all based on the database metadata. 9. What is a Distributed Storage Network on a Blockchain System? A Distributed Storage Network (DSN) is a peer-to-peer network based on blockchain. It is a decentralized and distributed network that provides storage. 10. MongoDB vs. DynamoDB: Choosing the Best Database for Your Business All about MongoDB vs DynamoDB. Explore benefits, and in-depth comparison to find out the best choice for your business app. 11. Storing data with Vinyl This article describes how the developers of the in-memory computing platform Tarantool implemented disk storage. 12. Books vs. Servers: Where Should We Store All of the Human Knowledge? We are used to storing our information in the cloud, which has clear benefits over paper databases and books. What is the best place to keep human knowledge? 13. Distributed Ledgers: The Next Logical Step How modern blockchain approaches the problem of data storage in decentralized systems and how a distributed ledger can be organized. 14. How To Restore Your Database From a SQL Backup Backing up SQL and databases using the manual restore method, the full database backup, the incremental restore, or the manageauditing command. 15. A Brief Introduction to Ethereum Swarm The idea for Swarm came from Gavin Wood, one of the founders of Ethereum. 16. Sustainable Computing beyond the Cloud Extreme increases in data streams are expanding the cloud's carbon footprint; a sustainable alternative to Cloud dependence has been developed. 17. What Is Redis and How Can It Make Your Website 30-40% Faster? Redis is a type of database that can be used to significantly improve your website's loading speed thanks to its design and its versatile selection of modules. 18. The Benefits of Amazon S3 Explained Through a Comic AWS S3 is one of the most fundamental services of AWS Cloud. 19. Beyond Container Orchestration - Kublr's Approach to Kubernetes Infrastructure Abstraction [This post is inspired by an Interview with Kublr CTO, Oleg Chunikhin] 20. What Web3 & Decentralization Mean for Data Storage I think of web3 and decentralization on a spectrum - it’s not just one or the other, but you can take incremental steps on the path towards your end goal. 21. Decentralized Storage Networks — An Explainer A comprehensive analysis of decentralized storage networks, technologies behind them, benefits, use cases, current issues, and an overview of DSN offerings 22. How Does Cloud Storage Work? Cloud storage is the technology that allows users and companies to store, maintain and access data on highly available servers via the internet. 23. How to Increase Space in C Drive on Windows 10 Without Losing Data If you receive a low disk space warning on your C: drive, you may use the Disk Cleanup utility to remove temporary and unwanted downloaded files. 24. Why Are Removed Posts Still Visible on Reddit? Even if moderators delete a post that is breaking the rules of Reddit, it is still very easy to find. 25. Decentralizing Data Storage on the Blockchain: An Exclusive Interview With Vincent Irlweck Interview with Vincent Irlweck, CMO at Inery Blockchain to discuss their  decentralized data storage solution and democratizing the data industry. 26. 9 Data Trends You’ll See in 2023 2022 saw the data space grow by leaps and bounds. Here are the top 9 things our team of data experts expects to see in 2023. 27. How Does Economic Recession Affect Colocation? Some tout data storage as recession-proof, but with little precedence to learn from, what will a recession look like for colocation? 28. The Shortcomings of Computer-controlled Robots Computer-controlled robots are monotonous. They are mostly able to perform a sequence of processing operations that is fixed by the equipment configuration and 29. Benefits of Corporate Data Backup and Best Practices to Keep in Place Nowadays, companies are increasingly relying on corporate data backup solutions to guarantee the safety and recoverability of their data. Read on to learn more 30. Transfer Big Data Across Cloud Platforms With Ease With Big Data and the rise of more affordable object storage such as; Wasabi Cloud Storage and Backblaze B2, the need to move large amounts of data from the big three cloud providers is trending. But how do you do it? Often it is is near impossible, or extremely hard to break up with someone like Jeff Bezos and go somewhere with cheaper rates. [31. How Velocity Can Change Data Storage And Propagation In Blockchain Nodes?](https://hackernoon.com/how-velocity-can-change-data-storage-and-propagation-in-blockchain-nodes-07x3z8b) Blockchain technologies have been disruptive and propagate the democratization of the data. This has led to its wider usage and popularity. The blockchain paradigm has shifted from financial usage towards application development usages. 32. How to Make Your Own and Free Backup Application In our age of rapidly developing technologies, data loss can be a disaster not only for large corporations, but also for the average user, showcasing the immense importance of backup and data recovery in today’s data driven world. 33. 5 Ways to Store Market Data: CSV, SQLite, Postgres, Mongo, Arctic What's the most efficient way to store market data? SQL or NoSQL? Let's compare 5 most common options and find out what is best. 34. How to Protect your Business from Any Type of Criminals: The Complete Guide Today’s modern world is, undoubtedly, not a safe haven for any business. Make no mistake, even running a small-time operation or setting up a niche venture can become challenging. But surprisingly enough, the major brands like Sony or top cryptocurrency exchanges such as Binance aren’t under the greatest threat - criminals and hackers mostly see their prey under a much lower grade. Network security is one of the stumbling stones for businesses of any scale nowadays. Digital age fraudsters rarely engage in assaulting the top corporations - the largest chunk of their bounty comes from the companies that would never make it to the Forbes list. 35. Top Players in the Hyperconverged Infrastructure Software Market for 2022 This blog examines the top players in the Hyperconverged Infrastructure software market for 2022, key trends in the market, and investigates their impact. 36. How to Perform a Cyber Security Risk Assessment: A Step-by-Step Guide Companies are increasingly spending money on cyber security. However, attackers are launching more sophisticated cyber attacks that are hard to detect, and businesses often suffer severe consequences from them. 37. Decentralized Cloud Storage is changing the face of the internet (1/2) (Read Part 2 here) 38. What are the Best Options to Store Data and Keep it Safe Forever? This article will go over the most effective, economical and long-lasting methods for storing our data. 39. Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution In this article, Thai Bui describes how Bazaarvoice leverages Alluxio as a caching tier on top of  AWS S3 to maximize performance and minimize operating costs on running Big Data analytics on AWS EC2. The original article can be found on Alluxio's engineering blog. 40. The Evolution Of Hacking Data Storage [Infographic] When the first computers were made, the information needed to run them was on punch cards. The computing device would decode the patterns on the punch cards and translate it to an action. It wasn’t until 1956 that IBM came up with the first magnetic hard drive, and floppy discs didn’t enter the scene until the 1960s. Early computer storage was rudimentary, which is why there was no real viable threat of hackers in those early decades of computing. Once data storage became more sophisticated, hackers became a real threat. Subsequently, the need for cyber protection was born. 41. Data Storage Security: 5 Best Practices to Secure Your Data Data is undoubtedly one of the most valuable assets of an organization. With easy-to-use and affordable options such as cloud-based storage environments, storing huge amounts of data in one place has become almost hassle-free. However, space is not the only concern for businesses any more. 42. The Best Options to Store Data and Keep it Safe Forever As technology evolved the options for storing the large amount of data have also changed. In this article we've discussed the terms Archiving and Backups. 43. An Analysis of Key Players in the Software Defined Storage (SDS) Market for 2022 This blog examines the top Software-Defined-Storage (SDS) market players for 2022 and highlights the importance of each one as an enterprise solution. 44. NFTs - Exploring Infrastructure, Usability, Role in DeFi, And Questions About Ownership Walk through NFT Standard, NFT characteristic traits and explanations, NFT utilisation in DeFi. 45. Making Sense of Unbounded Data & Real-Time Processing Systems A real-time processing architecture should have these logical components to address event ingestion & processing challenges, such as a stream processing system. 46. "We plan to establish the first IPFS node on Mars" - says Filecoin Miner Neo Ge As Filecoin gears up for launch, miners across the globe have been participating in Space Race, competing to onboard as much storage as possible to the testnet. 47. Digging into Postgres's Lesser Known Features Postgres Handles More than You Think 48. 10 Ways to Reduce Data Loss and Potential Downtime Of Your Database In this article, you can find ten actionable methods to protect your mission-critical database. 49. "We want functional decentralization" Q&A with Wildland Creators Wildland is a new, open data management protocol with improved users' privacy, security, and multi-categorization. A Q&A with J. Zawistowski and A. Regulski. Thank you for checking out the 49 most read stories about Data Storage on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

49 Stories To Learn About Data Storage

89 Stories To Learn About Big Data Analytics

Many of our friends in the valley feel completely unempowered to negotiate compensation plans since COVID began making waves in the US in March. The common stories we hear from top talent are either: (A) not even considering recruiting due to fear of the current economic climate (B) a feeling of “no leverage  to have any compensation conversations internally and (C) fear of on-boarding to a new role while everyone is remote. While that fear may have some validity it does not outweigh the potential lost opportunity of a better career advancement. Many in the country are hurting, and plenty of people  are without jobs. However, many software companies are seeing phenomenal growth, and the career options and opportunities for those in tech career paths become better with almost every passing day. With the current climate of uncertainty, even high performers seem to be more risk averse while negotiating their value in roles. But companies are more worried than ever about losing top talent in this uncertain time, and so right now might be the best time for you to renegotiate. This is the time to align your value with your company’s goals and your own. A few of our friends have recently tested this theory, and one even got a raise of $45K from a base pay under $200K. In March of 2020, recruiting and hiring in many companies froze, or was delayed due to COVID uncertainty. It felt like every other day there was a new article talking about layoffs around the country. And while the job market in the tech industry has bounced back, the rhetoric of nationwide unemployment is still rampant. In fact, by May that tide had turned and many companies gained ability to forecast their core business, as well as forecast hiring needs. Two examples include: Truework itself, with not even 100 employees and over as of 11 October 2020 13 openings Bazaar Voice with around 1K employees and as of 11 October 2020. 78 job openings If you sit in a competitive and growing part of the software and tech world, the poor job market is a myth.  This is no time to feel disempowered. Instead, the first half of 2021 is both an important time to consider your and a key time to negotiate for plans either internally or externally. As a high-performer, now may be the best time of your career to renegotiate and create long-term alignment with your company, or seek a new company that is better aligned with either your own values or the massive economic shifts underway. Know what you are worth and push for the right plan for your current contributions to the business. career path compensation We have seen a few trends in how to negotiate compensation / recruit during COVID. to avoid and tips to reframe your thinking. The following are the most common mistakes Focusing only on what you have done and what you have been paid in the past. All come down to leverage in some form or the other. By only focusing on the past you significantly weaken your position. Ultimately a founder's goal is to strategically spend capital in a way that maximizes future return on investment. No company in their right mind would offer an outside hire more money because they felt their compensation was too low at their old job. negotiations A common response when an employee uses this tactic is that your past performance is covered by your previous contract with the company, and that is a valid point. Focusing only on your title Some companies simply lack the ability to make material changes to title without mountains of political buy-in and support. For other companies, they may be able to change the title to your preference, but leave the rest of your compensation the same. Fighting for title, if you choose that path, may win a small battle  but lose the war in terms of achieving your overall goals around compensation. Making it personal / emotional Your negotiation has nothing to do with your self worth or how much your managers respect you. Particularly in this climate, company needs may have changed with scale and markets - what it needed a year ago may no longer be relevant. Preface the conversation by telling your manager this is not personal and you want to find long term alignment with the company. By mentioning this in the beginning you have a framework to reference as negotiations become more intense. Making it obvious that you are not willing to walk away Know your market value and that there are always options out there, and back to earlier points, negotiations come down to leverage. When you make it clear you are staying no matter what happens in the negotiation, your leverage is decreased, perhaps down to zero. Failing to get organized All too often employees simply state they want a higher base pay or bonus. The complete lack of research, lack of preparation, and lack of real-time comparables harms the negotiating process before it has even truly begun. Leverage your internal network, mentors, or even outside advisors on how to prepare for this negotiation including who you should talk to within the company first. Instead of falling into the above quagmire of mistakes, we recommend these solutions to ensure you are putting your best foot forward, and giving yourself the maximum chance of success in your endeavor to obtain a better or more fair compensation package. Focus on what the business needs in the future and why you are a key part of that solution Make clear in words, and ideally in quantitative details why you are the best person who can help the company reach the next stage in their growth trajectory. Clearly communicate how you are a proven asset and team player. Any new hire is risky and expensive,, not to mention the opportunity cost of ramping them up. Know your BATNA (Best Alternative To a Negotiated Agreement) What is most important to you and what is most important to the company, both now and in the future?  Are you willing to sacrifice cash now for equity later, and if so what is your preference, and what is your preference versus your understanding of company preference. For example, Amazon is widely known to have high equity compensation, and yet rarely if ever pays more than around $170K in cash base pay. Knowing your company’s preferences can save you a great deal of time and effort. If you have the time and interest, another offer is often powerful leverage in these sort of conversations. It can take significant time and energy to get this outside offer. However, if you know your company matches outside offers, and rarely works on a negotiation without it, then this may be the right path for you. Patty McCord, ex Chief People Officer at Netflix, is famous for encouraging employees to interview at other companies. Research, and predict what levers your boss/company cares about This research should help inform you on both who to talk with first ( likely your first line manager), and will also probably help shed light on how other negotiations have occurred for either existing employees or current recruits. If you have any relationship with HR or recruiting that can prove helpful to understanding where the market is currently for other hires, and how to think about positioning yourself. Negotiating by Email We rarely hear of anyone who emails these sorts of issues to managers with great success. Even if you are a star player, not using the video or phone call to convey important information like a compensation concern can put a bad taste in the mouth of leadership and harm your overall potential for the ideal outcome. Additionally, you should be ready to adapt your strategy based on the response and verbal/visual clues your boss gives you during the negotiation. You would never call an all-in bet in poker without first trying to read your opponent. Every negotiation is different, and your own compensation makes it difficult, if not impossible, to be fully objective. However, if you follow these steps, our experience is that you create the possibility of a very healthy engagement with your leadership, and probably showcase that you can and will stand up for yourself at the least. In a best case scenario there is an open attitude and mindset, your ask is logical and makes sense, and your leadership grants all or most of what you ask for enabling you to be an even more productive employee. These conversations are almost never easy, and rarely is there a “ ” during COVID to surface these issues. However, if you avoid solving an underlying point of friction for you as an employee, it will likely surface eventually as a point of conflict in either your departure to another job, or decreased performance. Neither of these outcomes are ideal for either party, so get out there and see what is possible. perfect time Hopefully, after you follow these best practices, you achieve your goal in compensation, promotion, and other career advancements. Congrats. The next step is to clearly define goals and objectives required to succeed in the new role / position / etc. Once you succeed in these new goals, it may be time to reach for your next promotion. Why read this / about the authors and have had wins in negotiating compensation for themselves and dozens of others over the last several years in both public companies and start-ups. Ben Faw Jeff Morelli Ben is a serial entrepreneur, an alumni of UBS (I Banking), Tesla, HBS, LinkedIn, and a Co Founder of BestReviews ( , and ). Sold to Tribune Publishing for over $100M sold again to Nexstar for $160M in November of 2020 Jeff was the first non-cofounder hire at (where Ben was an early advisor), and has built two multi-million dollar sales playbooks. On top of leading the company in sales every quarter, Jeff also closed the largest deal in Truework’s history. Truework

How to Negotiate Compensation During COVID (and at other times)

In this article, Thai Bui describes how Bazaarvoice leverages Alluxio as a caching tier on top of  AWS S3 to maximize performance and minimize operating costs on running Big Data analytics on AWS EC2. The original article can be found on . Alluxio's engineering blog This article aims to provide the following takeaways: Common challenges in building an efficient big data analytics platform on AWS How to setup Hive metastore to leverage Alluxio as the storage tier for frequently accessed tables, backed by all tables on AWS S3 as the source of truth How to setup tiered storage within Alluxio based on ZFS and NVMe on EC2 instances to maximize the read performance Benchmark results of micro and real-world workloads Our Big Data Workload and Platform on AWS is a digital marketing company based in Austin, Texas. We are a software-as-a-service provider that allows retailers and brands to curate, manage, and understand user-generated content (UGC) such as reviews for their products. We provide services such as  collecting and hosting reviews, programmatic management of UGC content, and deep analytics on consumer behavior. Bazaarvoice Our globally distributed services analyze UGC content for over 1900 of the biggest internet retailers and brands. Data engineering at Bazaarvoice requires handling data at massive internet-scale.  Take , for example, the combined 3-day holiday traffic generated 1.5 billion product page views and recorded $5.1 billion USD in online sales for our clients. Within a month, our services host and record web analytics events for over 900 million internet shoppers worldwide. 2018 Thanksgiving weekend To keep up with the web traffic, we host our services via Amazon Web Service. The big data platform completely relies on the open source Hadoop ecosystem, utilizing tools such as: Apache Hive, Spark for ETLs Kafka, Storm for unbounded dataset processing Cassandra, ElasticSearch and HBase for durable datastore and long-term aggregations The data generated by these various services is optimally formatted for analytics, such as Parquet or ORC, and is eventually stored in S3. Why Alluxio Using S3 to store our data enables us to increase our storage capacity effortlessly, but at the cost of incurring a performance bottleneck. Accessing sizable data from S3 is bottlenecked by the connection’s bandwidth. Since S3 is a managed service by AWS, it is not possible to write custom code to optimize to our specific environment, contrary to other open source technologies being utilized. One approach would be to provision more hardware to process data from S3 in parallel. But rather than spending more money on hardware, we decided to find a more ideal solution to solve this performance bottleneck. We realized that not all data access needs to be fast because the workloads typically involved a subset of data. However this selective subset is constantly changing as newer data is added. This led us to come up with a tiered storage system to accelerate our data access, with the following goals: It should be nimble by expanding as our business grows without any data movement or introducing ETL jobs It should provide the same level of interoperability for most of our existing systems It should allow us to scale by utilizing better hardware when our budget allows for it. With these criteria in mind, we decided to use to accelerate our data access. As a key component in the tiered storage system, Alluxio was picked as it is highly configurable and relatively cheap to reconfigure operational-wise. Alluxio Architecture with Alluxio We integrated Alluxio with Hive Metastore as a basis for the tiered storage S3 acceleration layer. Clusters can be elastically provisioned with support for our tiered storage layer. In each job cluster (Hive on Tez & Spark) or interactive cluster (Hive 3 on LLAP), a table could be altered to have more relevant data configured to use Alluxio, while having the rest of the data served by S3 directly. Since the data is not cached in Alluxio unless it is accessed via a Hive or Spark task, there’s no data movement. Since updating a table configuration is extremely cheap, adapting this pattern of changing queries  has allowed us to be very nimble. For example, let us look at our page view dataset. A page view is an event recorded when an Internet user visits a page in our client’s website. The raw data is collected, aggregated hourly, and converted into Parquet format. This data is stored in S3 in a year, month, day, and hour hierarchical structure: s3: |- pageview/
   |- /
     |- /
       |- /
         |- /
         |- /
         |- /
           |- pageview_file1.parquet
           |- pageview_file2.parquet //some-bucket/ 2019 02 14 01 02 03 The data is registered to one of our clusters via Hive Metastore and available to be used in Hive or Spark for analytics, ETLs, and other various purposes. An internal system registers the data and updates the Hive Metastore directly for every new dataset that it detects. # add partition from s3. equivalent to always referencing “cold” data
ALTER TABLE pageview 
  ADD LOCATION ‘s3: PARTITION (year= , month= , day= , hour= ) 2019 2 14 3 //<bucket>/pageview/2019/02/14/03’ Our system is also aware of the tiered storage configuration provided for each specific cluster and table. For instance, in an interactive cluster where analysts are analyzing the last few weeks of trending data, the table with partitions less than a month old is configured to read directly from the Alluxio filesystem. An internal system reads this configuration to mount the S3 bucket to the Alluxio cluster via REST API and automatically takes care of promoting or demoting tables and partitions using the following Hive DDLs: pageview # promoting. repeating query will cache the data from cold->warm-> hot
ALTER TABLE pageview SET LOCATION ‘alluxio: # demoting. 1 month older data goes back to the “cold” tier.
# protects our cache when people occasionally read older data
ALTER TABLE pageview SET LOCATION ‘s3: PARTITION (year= , month= , day= , hour= ) 2019 2 14 3 //<address>/mnt/s3/pageview/2019/02/14/03’ this PARTITION (year= , month= , day= , hour= ) 2019 1 14 3 //<bucket>/pageview/2019/01/14/03’ As more recent data arrives, older data is demoted, requiring the tiered storage configuration to be updated. This process happens asynchronously and continuously. Micro and Real-world Benchmark Results Our Alluxio + ZFS + NVMe SSD read micro benchmark is run on an i3.4xlarge AWS instance with up to 10 Gbit network, 128GB of RAM, and two 1.9TB NVMe SSDs. We mount a S3 bucket on Alluxio to perform 2 read tests. The first test copies 5GB of Parquet data using the AWS CLI into the instance’s ramdisk to measure only read performance. # using AWS cli to copy recursively to RAM disk
time aws s3 cp --recursive s3: //<bucket>/dir /mnt/ramdisk The second test uses the Alluxio CLI to copy the same Parquet data to ramdisk. This time, we perform the test 3 times to get the cold, warm, and hot numbers as shown in the chart above. # using AWS cli to copy recursively to RAM disk
time ./bin/alluxio fs copyToLocal /mnt/s3/dir /mnt/ramdisk Alluxio v1.7 with ZFS and NVMe takes about 78% longer to perform a cold read when compared to S3 (66s vs. 37s). The successive warm and hot reads are 2.5 and 4.6 times faster than reading directly from S3. However microbenchmarks don’t always tell the whole story, thus we will take a look at a few real-world queries executed by actual analysts at Bazaarvoice. We rerun the same queries on our production interactive cluster to compare the tiered storage system  against S3 by itself. The production cluster consists of  20 i3.4xlarge nodes running Hadoop 3, Hive 3.0 on Alluxio 1.7, and ZFS v0.7.12-1. Query 1 is a single table deduplication query with 6 columns to group by. It processes 95M input records and 8G of Parquet data from a 200G dataset. Query 2 is a more complex query, involving a join on 4 tables with aggregations across 5 columns. It processes 1.2B input records and 50G of Parquet data from several TB datasets. The two queries simplified for readability and their query plans are shown below. Compared to running the queries directly on S3, the tiered storage system speeds up query 1 by 11x and query 2 by 6x. # query simplified
SELECT dt, col1, col2, col3, col4, = lower( )
   AND month = AND year = AND (col7 IS NOT NULL OR col8 IS NOT NULL)
GROUP BY dt, col1, col2, col3, col4, col5 
ORDER BY dt DESC
LIMIT ; 1 col5 
FROM table_1 n
WHERE lower (col6) '<some_value>' 1 2019 100 Conclusion Alluxio, ZFS, and the tiered storage architecture have helped us save a significant amount of time for analysts at Bazaarvoice. AWS S3 is easy to scale in capacity and by augmenting it with a tiered storage configuration that is nimble and cheap to adapt, we can focus on growing our business and scaling storage as needed. Today at Bazaarvoice, the current production configuration can handle about 35TB of data in cache and half a petabyte of data on S3. In the future, we could simply add more nodes to grow the cache size or upgrade hardware for quicker localized access.

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution

All engineering teams strive to build the best product they can as quickly as possible. Some, though, stumble into a false dichotomy of choosing between speed and quality. While that choice may have been necessary in the past, it’s not the case today. What I’d like to do in this article is explain why. By reviewing the relationship between frequent software releases and increasing software quality and the dependency with telemetry, I’ll try to help readers understand that frequent releases coupled with data-backed insights are the best way to succeed in today’s marketplace. Pushing for faster value streams Value streams define engineering work. A stream includes all the work required to produce the end outcome. The outcome may be launching a new product, releasing a new feature, or simply churning through support tickets. Irrespective of their specific job title, everyone in engineering participates in a value stream. Organizations have always wanted to have faster value streams, and why wouldn’t they? Working faster means earlier time to market, beating competitors, and putting the product or service into customers’ hands quicker. Modern technology has turned every company into a technology company—whether they know it or not. In today’s market, building and shipping software successfully directly impacts the bottom line. The most successful companies differentiate themselves by building better engineering value streams. Their principles and practices of continuous delivery guided by real-time telemetry and continuous improvement have come to be known as DevOps. Indeed moving from waterfall to agile development and DevOps methodologies makes complete and undisputed sense. But if that’s the case, why are so many organizations still debating between speed and quality of their application releases? Is this not part of the same thing? The research behind engineering performance In their two books, and , Gene Kim, John Willis, Patrick Debois, Jez Humble, and Nicole Forsgren chart a clear course for improving engineering performance and debunk the idea that moving fast reduces quality. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Martin Fowler, the legendary software architect and author, claims that refuting this very idea is their most important contribution: “This huge increase in responsiveness does not come at a cost in stability, since these organizations find their updates cause failures at a fraction of the rate of their less-performing peers, and these failures are usually fixed within the hour. Their evidence refutes the bimodal IT notion that you have to choose between speed and stability—instead, speed depends on stability, so good IT practices give you both.” The findings in these research pieces are pretty convincing. High-performing teams, as compared to low-performing teams, deploy 46 times as often, are 440 faster from commit to production, 170 times faster in MTTR, and ⅕ as likely to encounter a failed deploy. These findings go hand in hand with Fowler’s assessment that good IT practices create speed and stability. The practices also generate business results. Accelerate compared data from Puppet Lab’s State of DevOps reports over three years. They found that high performance teams had 50% higher market capitalization growth compared to lower performers. Also -- high performance teams are twice as likely to exceed profitability, productivity, market share, and customer goals compared to lower performers. They’re “twice as likely to exceed noncommercial performance goals as low performers: quantity of products/services, operating efficiency, customer satisfaction, quality of products/services, achieving organizational/mission goals.” If this is not enough proof, two case studies in the DevOps Handbook -- Gary Gurver’s experience as the director of engineering for HP’s LaserJet Firmware division and Ernest Muller at Bazaarvoice -- further drive home the point. They demonstrate that using an established set of technical practices creates both speed and stability. This is the basis of The DevOps Handbook’s first principle: The Principle of Flow. It focuses on improving time from development to production. However, the benefits are contingent on the second principle. The Principle of Feedback The Principle of Feedback allows teams to course correct according to what’s happening in production. This requires telemetry (such as logs and metrics) across the value stream. The idea goes beyond the simple approach of monitoring uptime. Integrating telemetry across the value stream allows development and product management to quickly create improvements whether it’s from a production outage, deployment failure, A/B tests, or customer usage patterns. Focusing on telemetry makes decisions objective and data driven. Again, there are desirable knock-on effects across the organizations. Data aligns teams with objective goals, amplifies signals across the value streams, and sets the foundation for organizational learning and improvement. But designing for feedback doesn’t just happen. It’s a by-product of strong leadership that internalizes the goal to create telemetry within applications, environments, both in production and pre-production, and in the deployment pipeline. Scott Prugh, Chief Architect and Vice President of Development at CSG, said: “Every time NASA launches a rocket, it has millions of automated sensors reporting the status of every component of this valuable asset. And yet, we often don’t take the same care with software—we found that creating application and infrastructure telemetry to be one of the highest return investments we’ve made. In 2014, we created over one billion telemetry events per day, with over one hundred thousand code locations instrumented.” To resolve production incidents, high performance IT teams use telemetry 168 times faster than their peers with MTTR measured in minutes. Low performers, for contrast, had MTTR measured in days. This is no surprise when key business metrics are tracked, visualized, and monitored. The following image is taken from Ian Malpass’ “Measure Anything, Measure Everything.” It displays successful and unsuccessful logins. Vertical lines annotate deploys. It’s immediately apparent that there was a problem before 7 a.m. that may be related to the last deploy. Teams can discern that information in seconds. That means a quicker diagnosis, and paired with continuous delivery, faster resolutions. Etsy embodies this philosophy in what became known as the “Church of Graphs.” Ian Malpass, the Director of Engineering, puts it simply: “Tracking everything is key to moving fast, but the only way to do it is to make tracking anything easy...We enable engineers to track what they need to track, at the drop of a hat, without requiring time-sucking configuration changes or complicated processes.” Eventually they added enough tracking to make deployments safe. Building proper telemetry infrastructure is a key decision. The telemetry system must take in numeric data and logs across multiple components and present it in a unified way. Many teams start with a simple Statsd solution since adding telemetry requires one line of code and a Statsd server to aggregate the data. That’s enough to start, but it omits a huge data source. All applications produce logs. The logs may contain the most valuable insights into performance and are vital for effective troubleshooting. If adding instrumentation to code isn’t possible, it’s possible to extract telemetry from existing logs. Also, some failure scenarios cannot be instrumented in code. A log entry like “process 5 crashed from a segmentation fault” can be counted and summarized as a single segfault metric across all infrastructure. Transforming logs into data enables statistical anomaly detection, so changes from “30 segfaults last week” to “1000 of segfaults in the last hour” can trigger alerts. The debate is still alive As Dale Vile, CEO & Distinguished Analyst at Freeform Dynamics Ltd, explained in for Computer Associates,  organizations are still deciding on the necessary trade off between speed or quality despite the above wide-scale agreement that the two should really go hand and hand. an article The simple fact is that environments have grown even more complex with constant cycles of development, testing, releasing and ongoing support of the process. The growing adoption of microservices and an orchestration layer to manage the complexity has added a dynamic where speed is now inevitable and quality seems to be pushed by a plethora of vendors focusing on each part of this application release value chain. Every link in the chain is critical but gaining both high and low level visibility across all the constantly moving parts of the application delivery cycle is more challenging. Logging, metric monitoring, and increasingly distributed tracing as well, are common practice amongst engineering teams. By definition though, the best we can do is react as quickly as possible once notified a log- or metric-based event occurred. This falls short of being proactive enough to ensure safe and continuous operations. To help engineers be more proactive, the next generation of supporting technologies need to be able to identify and reveal events already impacting the environment automatically, together with the context required to see the bigger picture. Roi Ravhon, Core Team Lead here at explains: Logz.io “In today’s world of orchestration of hundreds of microservices, it is impossible to reach continuous operations with the ability to proactively see errors and exceptions before they impact production. Indeed, with these insights, the faster application release that can be achieved, the better the quality of the final product will be. Without proactive insights, the opposite is true” With this technological capacity, we can truly achieve our vision of continuous operations with the use of the modern technology stack - so long as it is enhanced with a layer of application insights that intelligently give us visibility into these errors and exceptions that have - and have not - been previously defined. In this state speed = quality. The debate should be OVER I think it’s time to finally bury the debate over speed and stability. The studies above demonstrate that applying The Principle of Flow via continuous delivery and backing with the Principle of Feedback via telemetry should produce both speed and quality. Cloud computing, containers and container orchestration and serverless have helped increase access to continuous delivery. Monitoring technologies have evolved as well with an eye to automation and advanced analytics based on machine learning. Today’s engineering teams are better poised than ever to build robust telemetry systems. There’s a plethora of paid products for all classes of vendors, open source platforms, and a host of integrations for all popular languages and frameworks. These platforms take advantage of the current landscape offering drop in telemetry and visualizing for servers, containers, orchestrators, virtual machines, and databases. Today’s systems create huge volumes of data, and data analysis tools must be able to keep pace. Automated metrics and trend prediction can amplify failure signals and reveal new ones that teams couldn’t find on their own. Advanced log analytics has become standard practice, helping teams to effectively dig deeper to discover the root cause in a fraction of the time this task used to take. In 2019, artificial intelligence is already adding even deeper visibility and providing the insights necessary to trust the speed and improve products safely.

Speed and Quality are Not Mutually Exclusive: Telemetry is the Key

The value of one’s opinion has increased once social media appeared. So what’s coming next? Before we get into that, though, let’s rewind for a bit. Ten years ago, it was pretty hard to become an opinion leader — there were only the message boards similar to Reddit where people shared their opinions, and it was chaos. It was pretty impossible to follow one particular user and everyone’s opinions were scattered around and hard to find. This all changed with the creation of social networks. In 2004, when Facebook was launched, people were given a channel to express their opinions and thoughts globally. Eventually some people started to gain followers who listened to them and they became opinion leaders (also known as influencers.) The growth of YouTube and Instagram, two platforms with an emphasis on visual content, only made this trend stronger. Now there are thousands of opinion leaders and many of us, mostly millennials, like to know what they wear, what gadgets they use, what food they eat. Some people don’t care about YouTubers or online celebrities, but they might be interested when some of their friends recommend a product, and that has changed the way we interact and the way we act as consumers. Sadly, large e-commerce platforms ignore that change — they take into account the interests of users by analyzing large chunks of data, but they don’t try to use other levers that may influence their decisions. With decentralization, that might change. and others blockchain-based projects can help us to make money by sharing our opinions online. Do you want to know how? Read further and find out! Winnest ​Opinions are more trusted than ads In this day and age, it’s not easy for any brand to stand out among the others and win over consumer attention — the advertising space has become too crowded. An average Internet user sees around , and their brains develop , filtering out any ads. Other users use ad blockers and some users even try browser, which natively blocks ads, leaving no chance for an advertiser to get to the consumer. 5,000 ads per day banner blindness Brave At the same time, there’s another option for brands wanting to successfully promote its products — leave it to the existing customers! Opinions are important nowadays because trust them as much as a personal recommendation. At the same time, 64% need to read between one and six online reviews to form an opinion. 84% of people Why is opinion sharing so appealing to many people? The Internet is currently inhabited by the younger generations — and . They have and they are always ready to recommend something they like and talk about something they don’t like. Also, they like to rate everything because nowadays “ .” Thus, sharing opinions on the Internet is perfectly natural for them. millennials generation Z different values it’s all about ratings Where does all this sharing happen? One of the most visited platforms for reviews is . It has over average unique monthly visitors and reviews of businesses. Yelp 145 million 163 million Another popular platform for reviews is — having over 4.5 million unique visitors per month and they allow e-commerce companies to integrate the Bazaarvoice platform into their marketing, providing tools for collecting reviews, gathering feedback, and keeping it all in a centralized database. The only disadvantage for a business is that the platform has complete control over all the data. Bazaarvoice Of course, we can’t exclude from this list. Amazon is the biggest e-commerce platform, having a yearly revenue of in 2017. More than users visit the site monthly, so the reviews for the products listed on the platform are quite handy given all those statistics we’ve already seen. At the same time, not all of them are genuine — Amazon has a allowing businesses to order paid reviews. The same problem with fake reviews haunts , Google, Facebook. There’s still no way to tell a genuine review apart from a paid one, but this problem can be solved by introducing decentralized blockchain solutions. Amazon Customer Reviews $3.03 billion 197 million program Yelp Blockchain for social platforms Blockchain technologies are still in the development phase, but it’s already clear that they have the potential to transform finances, bureaucracy, supply chains, e-commerce, and many other areas, including social networks. The immutability of the blockchain ledger has the quality of eliminating the trust issues found within a network, and trust is very important when we’re talking about online reviews. By using the blockchain in e-commerce we can address the following problems: Fake reviews — Every user’s identity would be assigned to a specific and unique profile via a , and it would be impossible to create a fake one. decentralized digital identity Centralized storage of customer reviews and data — All data would be stored in a decentralized manner and not a single entity is in control of that data. Right now, bringing more value to a business by giving it a good rating, users get nothing. The blockchain would reward users with tokens for sharing their opinions and helping businesses grow. Regular users will get an opportunity to earn money by sharing their honest opinions and their opinions will become more valuables. At the same time, advertisers currently receive less and less sales from regular display ads, so how can they change their marketing strategy? The new reality for advertisers Advertisers, companies, and brands usually tend to rely on centralized advertising platforms such as Facebook or Amazon’s inner advertising system in their marketing strategy, and in 2017 alone, advertisers spent worldwide — this metric is expected to reach in 2018. That’s too many ads for every single user, so users no longer care about the content of these ads. $209 billion $269 billion In this environment, advertisers are forced to look for new ways for advertising their goods and services, and this task will be even more important when e-commerce moves to the decentralized economy. When we talk about decentralized platforms, advertisers have two options: Paying users directly to watch ads. First option: This option is a mutually beneficial relationship between a potential buyer and an advertiser. An advertiser pays their potential customer to make him/her watch the ad and they gain the opportunity to sell the product. This model has a chance to become popular, because users use more and more ad-blocker plugins in their browsers or even such browsers as . Brave An average user spends on various websites — so make them pay attention to your content by literally paying them and their all yours. 1 hour per day Using opinion leaders. Second option: In a world where countless companies try to sell their products via millions of marketing activities, people need something to rely on when making a choice. like to wear the same clothes as their beloved celebrities; for example, the same sneakers the or drink the same beverage as Katy Perry. Of course, this is popular right now with millennials and Generation Z, but in the near future they will the majority of the purchasing power. Some people Kanye West has represent For those who don’t have any interest in following celebrities, there’s another option — following the opinions of their friends and family. According to : statistics 23% of online shoppers are influenced by social media recommendations. 55% of online shoppers tell friends and family when they are dissatisfied with a product or company. 84% of online shoppers in the United States review at least one social media site before making a purchase That’s where advertisers can make sales! Both mechanics are used by , a marketplace that allows its participants to use social media as the primary channel for promoting products. Those users who want to make money can recommend any product of their choice on their social media pages and get paid by companies when those products get purchased by their recommendation. In a decentralized environment, this model will have its obvious benefits: Winnest It filters out bad products. It’s unlikely that big influencers will risk their reputation by recommending low-quality goods and services, even in exchange for money. Advertisers would be able to cut their costs. Market participants won’t have to pay any fees to the platform (no centralized entity would be able to charge them.) Overall, with Winnest customers will get the products cheaply and safely and have complete control over their data. Another decentralized review platform, , utilizes a slightly different approach. They allow companies to build their own review systems on top of their Lina Network. Instead of posting subjective opinions, Lina has a list of criteria for every field of business and every review has to contain an evaluation of the customer’s experience by these criteria in order to be counted as valid. Coupled with the developer’s kit, this makes Lina a platform with big e-commerce potential. Lina Review The social recommendations could become one of the main advertisement methods instead of shoving banners in the customer’s face. And it would be more subtle, much like native ads, that sell than display advertising. a lot better A future where sharing is actually profitable Okay, this was a small trip to the not-so-distant future. Will the decentralized economy get more participants? It will. Will the decentralized model overtake the market soon? That depends. Decentralization becomes efficient only if all participants are motivated to play their roles within the ecosystem. After all, for a common user or an advertiser it doesn’t matter if the system is centralized or not, it only matters if he or she earns something by spending time interacting with it. People like to share and if they are getting a reward in return, that’s even better. Financial motivation coupled with an emotional motivation will keep us going further, and a decentralized marketplaces will help us build a new kind of economy that’s better for consumers and advertisers alike. About the author: — Founder of Geekforge.io and Howtotoken.com. Interviewing the top 10,000 worldwide experts who reveal the biggest issues on the way to technological singularity. Join my . Kirill Shilov #10kqachallenge: GeekForge Formula

Commerce Advertising: Share your Opinion and Make Money

This is a guest blog contributed by datasapiens’ Juraj Pohanka, Koen Michiels and Sam Gilbert. This article described how engineers at datasapiens brought down S3 API costs by 200x by implementing Alluxio as a data orchestration layer between S3 and Presto. . I Introduction Previous works There have been numerous articles and online webinars dealing with the benefits of using Alluxio as an between the S3 data storage and the data processing system used for ingestion or retrieval of data (i.e. Spark, Presto), as depicted in the picture below: intermediate storage layer To name a few use cases Alluxio-Presto use cases: Ryte ING Alluxio-Spark use cases: e Bazaarvoic The main conclusion from these use cases is that using Alluxio as a data orchestration layer has the following benefits: in data processing pipelines: lower latency Co-locating data and computation reduces network traffic for usage concurrency: horizontal scalability S3 API has limitations on the request rate for a given prefix All these benefits are critical when deciding a production-grade data processing architecture, however one important benefit has so far not been sufficiently highlighted when choosing such architecture – on the usage of the S3 API. This will be the focus of this article. cost reduction datasapiens is an international startup, based in Prague, that focuses primarily on helping companies to transform their business through data. We offer 3 key products: datasapiens our business intelligence platform with an embedded insights framework to help clients to become data-driven companies. Business insights commercialize clients’ data assets by driving cooperation with their suppliers. Brands insights increase customer engagement, sales and margin through a simple plan-do-review process that continuously measures and improves the loyalty lifecycle of clients’ customers. Personalized offers Our infrastructure journey of our infrastructure used AWS Redshift as the main data warehouse. However, when we started to acquire clients with larger data volumes (i.e. billions of records per year) we started to struggle with the following: The first generation storage-compute coupled architecture large infrastructure costs long query compilation times vendor-locking It was at this point that we decided to move to the . Hadoop ecosystem was the common software stack for cloud-based data lake architectures: The first version of our second generation S3 for data persistence Spark for ETL Hive for data warehousing Presto as a distributed query engine With this storage-compute separated architecture, we achieved faster performance and reduced compilation times. However, the infrastructure costs were above expectations due to S3 API cost increases, making up a large proportion of the overall cost (described in detail below). includes Alluxio as a data orchestration layer. We co-locate our data with our compute for in-memory data access. This architecture achieves even faster performance more consistently. In addition, we saw a drop in infrastructure costs due to S3 API costs dropping to a negligible level. The second version of our second generation . II Our observations A drastic increase in S3 API costs During the fine-tuning of our first Presto cluster, we conducted several performance tests of the cluster under various levels of user concurrency. The setup of the performance tests was the following: : dataset setup internal client data (1 billion rows of retail transactional data) persisted in Parquet workload setup 23 different queries used in client’s reports concurrency levels from 1 to 150 Ryte : cluster hardware setup worker nodes: count: 10 type: c5.4xlarge master nodes: count: 1 type: c5.4xlarge cluster software deployment setup PrestoDB 0.227 Apache Hive v2.3.5 For the design of our DWH, we wanted to avoid using the traditional star/snowflake schemas. Instead, we could pre-join the table entities in our ETL into a few OLAP cubes to avoid expensive table joins in Presto. Towards the end of the testing phase, we detected a dramatic increase in costs for the S3 API service. The following image depicts the incurred S3 API costs for each day of the performance tests: The following table summarizes the total number of executed queries with the S3 API costs and S3 API request counts per each day of performance tests: The next table shows the percentage of the S3 API costs out of the total costs: During this testing period, we tested several configuration variations of the Presto cluster in order to achieve a satisfying query speed at a specific concurrency level. In the first table above, we can observe the variance in the S3 API costs/requests per day. We attribute this variance to the varying Presto cluster deployments for each day in the testing period. After reviewing the incurred S3 API costs under the given workloads, we came to the conclusion that running a Presto cluster that read directly from S3 was not feasible, as it would impact our profit margin. Alluxio as a remedy We considered deploying Hadoop as an intermediate layer between Presto and S3 and synchronizing through simple jobs. We also looked at Alluxio, which provided similar functionality without the need for manual synchronization. Furthermore, Alluxio has additional benefits such as connections to various cloud providers & storage devices and having tiered storage functionality which we thought would improve query speed. After deploying a co-located Alluxio-Presto cluster, the performance of queries improved (especially for those with a higher number of execution stages), and the S3 API costs dropped to below $1/day. We have since implemented co-located Alluxio-Presto clusters in production. The infrastructure is successfully serving hundreds of users per client. We also learned that although manual synchronization is not needed. To prevent any performance degradation of the Presto clusters during startup and after ETL jobs it’s best to “hydrate” the cluster using “warm-up” jobs. . III A reproducible example using TPC-DS To get a better quantitative understanding of the S3 API costs and the cost reduction when using Alluxio, we conducted a simple experiment by querying data from S3 in two separate clusters: a co-located Alluxio-Presto cluster using PrestoSQL deployed in a Docker Swarm cluster an EMR cluster with PrestoDB without Alluxio In both clusters, we measured the API request count to Alluxio per each query. The following are the setup schemas for each cluster: Dataset setup: TPC-DS dataset with scale factor 100 stored in a S3 bucket Alluxio-Presto cluster setup: hardware setup worker nodes: count: 6 type: c5.4xlarge master nodes: count: 1 type: m5.2xlarge software deployment setup vertions of used systems: Alluxio 2.2.0 PrestoSQL 337 EMR-Presto cluster setup: hardware setup worker nodes: count: 6 type: c5.4xlarge master nodes: count: 1 type: m5.2xlarge software deployment setup PrestoDB 0.227 Query execution setup: set of queries: TPC-DS suite excluding Query no. 72 (due to longer query times) query executions: number of repeats: 10 concurrency level: 1 (no parallel executions) Measurements setup: Alluxio: logical Operations: ‘File Infos Got’ RPC Invocations: ‘GetFileInfo Operations’ S3: total request counts per request type total request costs per request type The deployment configuration for each of the clusters, as well as the complete test results, can be found in our . GitHub repository Here are the results from the conducted tests on the Alluxio-Presto cluster: 10 most API request-expensive queries: cumulative request counts: S3 API costs for caching the dataset into Alluxio: Here are the results from the conducted tests on the EMR-Presto cluster: As we can observe, the cumulative count for the `GetObject` request type and the `File Infos Got` request type are more-less compatible. The different values for the storage API calls in both cases can be attributed to: different nature of the Alluxio and S3 file system API different versions of used Presto clusters (PrestoSQL 337 vs PrestoDB 0.227) In the case of the S3 API, we also have an additional cost for the `ListBucket` request type which is more costly then the `GetObject` type. Now, we will try to estimate the per-query S3 API costs for the EMR-Presto cluster. We are limited by the following facts: we cannot tag individual requests to S3 we do not know the exact number of each API request type incurred by individual query To estimate the per-query S3 API costs, we will take a naïve approach and redistribute the costs to each query proportionally to the number of requests sent to Alluxio. This gives us the following table with the top 10 request-expensive queries: As can be seen, we can have quite significant costs for a single executed query. Let's do a simple comparison of the cluster infrastructure and S3 API costs for both cases with and without Alluxio: It is apparent that the S3 API costs for the cluster with Alluxio form a negligible (0.58%) part of the total costs, whereas for the cluster without Alluxio, they form almost half (48.83%) of the total costs in our simple test case. . IV General implications for cloud data lake architectures Looking at the pricing models for the storage services of three largest cloud providers ( , , and ), we see that they are very similar to each other, hence one can expect similar storage API costs when running analytical workloads with or without Alluxio. However, this claim still needs to be verified. AWS S3 Azure Blob Store Google Cloud Storage Our test case was a simple setup of a small cluster to which we were sending queries in a sequential manner. However, real-life production environments cope with workloads with far higher compute and storage demands. Frequently, we have: a large number of processes and users frequently and concurrently running queries workloads running real-time data streaming, ingesting and retrieving large amounts of data For these types of cloud-based production environments, using no intermediate data storage layer between the cloud storage service and the data processing system can lead to tremendous costs. Therefore we would be inclined to recommend that when running analytical workloads in a cloud-based environment, to have a data orchestration layer such as Alluxio. The original article was published on Alluxio’s Engineering Blog .

Stories, News & Mentions about BAZAARVOICE

Bazaarvoice’s Stories on HackerNoon