Cockroach Labs

cockroachlabs

Kristy is in her final year of computer science at the University of Waterloo (UW). Since her enrollment, she has interned for Paypal, Cognite AS, Peggy, Cockroach Labs and Google — all as a software engineer. When she graduates, she plans to work as an engineering manager. At the moment, Kristy is a technical project lead for UW Blueprint initiative, which focuses on accessible technology and promoting public and social welfare. The initiative partners with non-profit organizations to provide websites, mobile applications and analysis tools for free. Under UW Blueprint, Kristy leads a project that builds web applications to help Planet Read coordinate volunteers to translate and review children’s stories. “Currently how they process their translations is through email and Word documents, so we’re building a platform to hopefully change that,” she says. “It’s kind of like my brainchild for the last few months and we will be deploying in [January].” Kristy’s journey to UW’s computer science program was possible through a handful of positive reinforcements. Growing up in Guelph, ON. alongside her older sister to Chinese immigrant parents, Kristy fostered a keen interest in computer science through video games and mathematics. From catching Pokemons to rummaging through online indie games, Kristy began to see the correlation of mathematics in them. “I always liked tinkering with things…it seemed like computer science was a natural concept to try out,” she says. Although in high school, Kristy did not find the computer science courses to be challenging enough, she admits having an excellent teacher was a game-changer. She is also proud that she was part of a tight-knit comradery of the few female students looking to venture into computer science careers. “It gave me a lot of inspiration to try it out myself, and I really don’t know where I’d be if they were not there,” the 22-year-old says. “I think that it was really huge for me.” Then there were hackathons. “Hackathons really connected me with the community I wanted to be a part of, I met a lot of students from Waterloo while I was in high school in the hackathon community.” she says. For Kristy, hackathons were a way for her to connect with people who wanted to build and create. What attracted her to hackathons weren’t the elaborate and complex projects, but the simple ones that embodied the spirit of learning and having fun. Throughout high school, hackathons became Kristy’s staple extracurricular, but for her it was more than just an after-school activity. In 2016, she would become the founder of SpartaHacks, introducing Guelph to the first high school hackathon, organized alongside her friends from September to December. This was Kristy’s first time organizing and finding major success. “I really wanted to give back to a community that gave me so much.” Kristy says. “My school, on a smaller level, didn’t have much of a computer science environment or community, I felt really passionate about bringing it to my school and convincing my friends to enjoy these things.” As Kristy found herself studying at UW, her passion for hackathons continued. Soon, Kristy will be working with Hack the North as a logistics organizer and backend engineer. “I wanted to find people I could relate to more, I think it’s important to be able to have people in your life who can support you in the way that you need to be supported,” she says. “It’s hard.” Kristy does not shy away from talking about identity politics in the computer science industry, including her time at UW. As a queer woman of colour, finding a space in the industry isn’t as easy as it seems. In her experience, having mentors who either are allies or individuals from your community is a way to feel connected to a stereotypically heteronormative space. “It was huge for me that I had mentors in high school, who were women in computer science.” “It’s great to have a diverse network, and be friends with people from all walks of life, regardless of the background you come from.” Kristy spoke of the hope she feels in the growth of inclusion in the industry. “I think the onus is on everyone to make the space more welcoming and inclusive,” she says. When looking at Ontario, Kristy stated that there needs to be more change in regards to learning in programming, as computer science isn’t offered in every school, public or private. This would mean providing more resources, funding programming camps and implementing programming courses in schools that are subsidized. “If you grew up in rural Ontario, where there might not even be upper calculus classes, then it’s going to be hard to get a hold of programming classes,” she says. “Having the resources and the opportunities to go to programming camp, or go to hackathons, can be really influential with that.” With the ever-growing environment of STEM careers, Kristy believes there are many ways for students and people interested in computer science to increase their skills. For Kristy, co-op programs changed her life for the better, giving her a chance to explore different paths. Kristy spent time in California and Norway where she worked remotely for international companies during the pandemic. Through these experiences, her love for startups and entrepreneurship grew. “You receive valuable hands-on experiences on building software and websites from scratch,” she says. The desire and focus to build and create is what Kristy believes is the most important factor in figuring out what a student or someone just entering the STEM industry needs. She also emphasizes the need for expanding people’s creativity, and not just sticking to one type of building. “Take time to take care of yourself, do some art, some construction, woodwork, things like that,” she says. “Those are all important in keeping the spirit of building things alive, and to keep the spirit healthy.” There’s more interesting interviews like this at Thinking Cap!

In Conversation with a Computer Science Major, Kristy Gao

How can you tell which cloud provider is the right one for your business? And which one isn’t going to break your wallet? On the surface, AWS, Microsoft Azure, and Google Cloud are pretty similar. They all offer flexible compute, storage, and networking combined with everything engineers love about the cloud: self-service, instant provisioning, and autoscaling. But cloud providers differ in key areas that have a massive impact on your cloud bill. Most of your bill consists of d. That’s why I will focus on comparing the pricing of Instances and Virtual Machines across AWS, Azure, and Google Cloud to show you the nuanced differences between these providers. compute resources that represent even 75%-80% of a company’s cloud spen Cloud landscape overview: Top strengths of AWS, Azure, and Google Cloud Selecting one vendor over another comes down to knowing what your teams, applications, and workloads need. You need to have a full understanding of your requirements before setting out to explore the cloud landscape. Next, it’s time to examine each provider separately. The thing is, AWS, Azure, and Google Cloud come with different strengths and weaknesses. AWS is the leader for a good reason Let’s start with AWS that currently holds the largest chunk of the cloud market. Its undeniable strength is . You can choose from and pair them with a rich array of tools for databases, analytics, management, IoT, security, and enterprise applications. This list could go on almost forever. the sheer breadth and depth of its services almost 400 instance types AWS is the winner when it comes to developer functionality, thanks to the breadth of its services and ability to translate this scale into cost benefits. Still, this . One of the common pricing challenges companies face is understating AWS metrics and estimating the impact of architectural decisions on costs. complexity comes at a cost Azure is the go-to choice for enterprises. Microsoft Azure is quite a . That’s usually because they have a long-standing relationship with the provider and meeting all their enterprise computing needs in one place makes sense. popular choice with C-level executives The combination of Azure, Office 365, and Teams makes a safe option for enterprises looking to benefit from enterprise software all the way down to cloud computing resources with a single handshake. Google Cloud taps into the power of open source. All three major cloud vendors are strong in machine learning capabilities. But Google stands out among them thanks to its endless well of internal research and expertise – basically, the magic powering the search engine giant throughout the years. What makes Google different is its role in . I’m talking especially about containers and Google’s central role in building Kubernetes for orchestration and the Istio service mesh, today practically industry-standard technologies. developing various open source technologies Google’s culture of innovation lends itself really to startups and companies that prioritize these approaches and technologies. Naturally, companies often end up using different cloud providers for different parts of their operations or use cases. This lets them bring the best of all the (cloud) worlds together under one roof. multi cloud approach Billing in AWS vs. Azure vs. Google Cloud Per second billing is now something all the three cloud vendors offer. AWS introduced it in 2017 for EC2 Linux-based instances and EBS volumes at first – but today, it applies to many other services. Note that in AWS, per-second billing works with the minimum 60-second limit. Azure allows per-second charges as well, but this billing model isn’t available for all instances – mostly container-based instances. Google Cloud followed AWS in the introduction of per-second billing and now offers it for more than just instances based on Linux. In Google, this form of billing applies to all VM-based instances. Let’s take a look at the biggest cost driver: compute Compute is what makes your cloud bill so high. This isn’t to say that other services don’t contribute to it at all. Storage can get quite expensive, and moving data around might result in high egress costs. But it’s compute that comes with the biggest price tag – and greatest opportunity for cost optimization. Here’s a case study that shows the . incredible impact optimizing compute costs can have on a company’s bottom line How Netflix slashed its cloud costs by picking different machines To deliver its video streaming services across 190 countries, uses Amazon EC2 instances and Kinesis data streams. Recently, Netflix turned to the Graviton2-based EC2 instances. Switching to these brand-new machines helped Netflix improve performance by 50% and cut costs. Netflix Before, the company used m4.16xlarge instances that cost $3.2/hour. Now, Netflix uses m6g.16xlarge that costs $2.464/hour, and m6g.xlarge, priced at only $0.0154/hour. You can only imagine the cost savings Netflix enjoyed on its cloud billing. This example clearly shows how a minor difference in instance pricing can save you thousands of dollars. Comparing cloud pricing – our example setup To understand the pricing differences better, we’re going to compare instances and VMs within one region and across the same operating system. This is our example setup: Region: US West – Los Angeles (US West in Azure, US West 2 in Google Cloud) Operating System: Linux vCPUs: 4 Types of instances/VMs we will analyze: General-purpose Compute-optimized Memory-optimized For our comparison, we chose instances with similar RAM and 4 vCPUs. The obvious exception is Google Cloud’s memory-optimized instance that starts at 40 vCPUs. Here are the instances/VMs we selected for our cloud pricing comparison: AWS vs. Azure vs. Google Cloud: Comparing On-Demand pricing Here is the hourly On-Demand pricing of each of those instance-type scenarios across AWS, Azure, and Google Cloud. Cloud pricing based on On-Demand rates Note: In the above table, red marks the highest price and green the lowest price for a given instance type. Takeaways: As you can see, AWS and Azure offer quite a similar pricing for all these three instance types. For compute-optimized instances, Google Cloud has the highest pricing due to its scalable processors and all-core performance. The cost of Google Cloud for memory-optimized instances is way higher because it doesn’t run on 4 vCPUs, but 40vCPUs.A note about chips and processors Providers roll out instances on different computers, and the chips in them have different performance characteristics. So, you might end up with an instance type that has strong performance your teams don’t actually need, and you’ll have no idea about this. Reasoning about this on your own is challenging, and the best way to see what you’re really paying for is via benchmarking. The idea here is to drop the same workload on each machine and check its performance characteristics. You might discover something interesting, just like we did: This chart shows CPU operation in AWS (t2.2xlarge with 8 virtual cores) at different times after several idle CPU periods. Would you expect such unpredictable CPU behavior within a single cloud provider? Source: CAST AI The used the same method to evaluate 54 machines across AWS, Azure, and Google Cloud. They ran over 1000 microbenchmark tests to evaluate metrics like CPU, network, storage, and TPC-C performance. 2021 Cloud Report from CockroachLabs t achieved the best single-core CPU performance and offered the greatest throughput at every level. Google’s machines with general-purpose disks achieved the highest level of raw throughput but turned out to be the least cost-efficient option. One of their overall conclusions was that Google beats AWS and Azure on performance. I AWS vs. Azure vs. Google Cloud: Comparing discounted pricing with 1-year upfront commitment All three providers offer discounts if you commit to using them for at least 1 year. The pricing model is called Reserved Instances/Savings Plans in AWS, Reserved Savings in Azure, and Commitment Price in Google Cloud. These discounts incentivize businesses to commit to a specific level of usage for a fixed period in exchange for a discounted rate on some (not all) instances and VMs. To compare the discounted pricing among AWS, Azure, and Google Cloud, we examined a one-year commitment period without upfront cost – but this time, in another region: US East – Northern Virginia. Cloud pricing with 1-year commitment Takeaways: The general-purpose instances with a 1-year commitment are almost similar in AWS and Azure. AWS wins in the compute-optimized category, even if Azure and Google apply some pretty big discounts. Azure offers the cheapest alternative among memory-optimized instances. Not sure how Reserved Instances work and whether they really translate into discounts? Read this: Do AWS Reserved Instances and Savings Plans really reduce costs? How CPU bursting can help drive your costs down AWS, Azure, and Google Cloud all offer burstable performance instances. These instances provide you a baseline level of CPU performance, with the opportunity to burst to a higher level whenever your workload requires that. Burstable performance instances are . a good pick for low-latency interactive applications, microservices, small and medium databases, or product prototypes But to make the most of them, you need to keep a close eye on the credits you collect that enable bursting. For example, if you restart a T2 instance in AWS, you’re going to lose all the credits. Restarting a T3 or T4 instance means keeping credits for another 7 days and then losing them. Our research into AWS burstable instances showed that if you load your instance for 4 hours or more per day on average, you’re actually better off with a non-burstable one. But if you run an e-commerce business that gets a large stream of visitors once in a while, it’s a good match. AWS vs. Azure vs. Google Cloud: Comparing Spot Instances/Preemptible VMs Here’s another way to reduce your cloud bill – take advantage of capacity that’s currently not being used by any businesses. Cloud providers sell this excess capacity at incredibly high discounts – while AWS Spot Instances offer up to 90% off the On-Demand rates, Preemptible VMs in Google can be even 80% cheaper than regular VMs. Here’s a quick overview of the potential savings you can get for these instances in the US East – Northern Virginia region: Cloud pricing with Spot Instances/Preemptible VMs As you can see, all providers offer substantial discounts on these instances. But to take advantage of these savings, . How? Here’s a step by step guide: you need to make sure that your applications can handle interruptions Spot Instances: How to reduce AWS, Azure, and GCP costs by 90% Optimizing cloud costs is a point in time exercise By now, you’ve learned the basics of calculating compute charges. But Spot instance prices don’t always stay the same. In fact, they might change from one second to the next. Take a look at this example: At CAST AI, we were running an application on a mix of AWS On-Demand instances and Spot Instances. Then we used our own platform to analyze our setup. The platform looked for the most cost-effective Spot Instance alternatives for a machine with 8 CPUs and 16 GB. CAST AI suggested that we run our workload on an instance called INF1. But wait, It’s a supercomputer that usually costs a lot of money. why would we need a powerful GPU instance meant for machine learning applications? So, why did CAST AI pick it? We checked the pricing. As it turns out, at that time, INF1 just happened to be cheaper than the usual general purpose instances we used. We would have never guessed to look for Spot Instances in this category. As a result, we’d miss out on this incredible gem. That’s exactly the reason why you need automation to optimize cloud costs Even if you have DevOps or FinOps specialists managing your cloud expenses, . It’s high time you took control of your cloud bill with the help of automation and AI. you’re probably spending twice as much as you should CAST AI is a great place to start your path towards automated optimization if your teams work with Kubernetes. The solution automatically creates and implements cost-cutting tactics for guaranteed savings without any manual and repetitive work for your engineers. Visit CAST AI to learn how automation can reduce your cloud bill to the minimum regardless of the cloud provider you are working with.

AWS vs. Azure vs. Google Cloud: Which One Delivers Most Value for the Price

. Because one of the most common use cases of the new databases is storing data that is generated by high-throughput sources, it is important that the store engine is able to handle write-intensive workloads, all while offering acceptable read performance. RocksDB implements what is known in the database literature as a aka LSM tree. Technical design log-structured merge tree A Brief History of Distributed Databases The era of Web 2.0 brought with it a renewed interest in database design. While traditional RDBMS databases served well the data storage and data processing needs of the enterprise world from their commercial inception in the late 1970s until the dotcom era, the large amounts of data processed by the new applications—and the speed at which this data needs to be processed—required a new approach. For a great overview on the need for these new database designs, I highly recommend watching that database guru Michael Stonebraker delivered for Stanford’s Computer Systems Colloquium. this presentation The new databases that have emerged during this time have adopted names such as NoSQL and NewSQL, emphasizing that good old SQL databases fell short when it came to meeting the new demands. Despite their different design choices for particular protocols, these databases have adopted, for the most part, a shared-nothing, distributed computing architecture. While the processing power of every computing system is ultimately limited by physical constraints and, in cases such as distributed databases where parallel executions are involved, by the implications of , most of these systems offer the theoretical possibility of unlimited horizontal capacity scaling for both compute and storage. Amdahl's law Each node represents a unit of compute and storage that can be added to the system as needed. However, as Cockroach Labs CEO and co-founder Spencer Kimball explains in the case of CockroachDB, designing one of these new databases from scratch is a herculean task that requires highly knowledgeable and skillful engineers working in coordination and making very carefully thought decisions. here For databases such as CockroachDB, having a reliable, high-performance way to store and retrieve data from stable storage is essential. Designing a library that provides fast stable storage leveraging either filesystem or raw devices is a very difficult problem because of the elevated number of edge cases that are required to get right. Providing Fast Storage with RocksDB is a library that solves the problem of abstracting access to local stable storage. It allows software engineers to focus their energies on the design and implementation of other areas of their systems with the peace of mind of relying on RocksDB for access to stable storage, knowing that it currently runs some of the most demanding database workloads anywhere on the planet at and other equally challenging . RocksDB Facebook environments The advantages of RocksDB over other store engines are: . Because one of the most common use cases of the new databases is storing data that is generated by high-throughput sources, it is important that the store engine is able to handle write-intensive workloads, all while offering acceptable read performance. RocksDB implements what is known in the database literature as a aka LSM tree. Technical design log-structured merge tree Going into the details of LSM trees, and RocksDB’s implementation of the same, is out of the scope of this blog, but suffice it to say that it’s an indexing structure optimized to handle high-volume—sequential or random—write workloads. This is accomplished by treating every write as an append operation. A mechanism, that goes by the name of runs—transparently for the developer—in the background, removing data that is no longer relevant such as deleted keys or older versions of valid keys. compaction Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ Through the clever use of , RocksDB also offers great read performance making RocksDB the ideal candidate on which to base distributed databases. bloom filters The other popular choice to base storage engines on is b-trees. InnoDB, MySQL’s default storage engine, is an example of a store engine implementing a b-tree derivative, in particular, what is known as a b+tree. . The choice of a given technical design for performance reasons needs to be backed with empirical verification of the choice. During his time at Facebook, in the context of the project, a fork of MySQL that replaces InnoDB with RocksDB as MySQL’s storage engine, Mark Callaghan performed extensive and rigorous performance measurements to compare MySQL performance on InnoDB vs on RocksDB. Details can be found . Performance MyRocks here Not surprisingly, RocksDB regularly comes out as vastly superior in write-intensive benchmarks. Interestingly, while InnoDB was also regularly better than RocksDB in read-intensive benchmarks, this advantage, in relative terms, was not as big as the advantage RocksDB provides in the case of write-intensive tasks over InnoDB. Here is an example in the case of a I/O bound benchmark on : Intel NUC Source: https://smalldatum.blogspot.com/2017/11/insert-benchmark-io-bound-intel-nuc.html . RocksDb provides several tunable parameters to extract the best performance on different hardware configurations. While the technical design provides an architectural reason to favor one type of solution over another, achieving optimal performance on particular use cases usually requires the flexibility of tuning certain parameters for those use cases. Tunability RocksDB provides a long list of parameters that can be used for this purpose. Samsung’s Praveen Krishnamoorthy presented at the 2015 annual meetup an extensive on how RocksDB can be tuned to accommodate different workloads. study . In mission-critical solutions such as distributed databases, it is essential to have as much control and monitoring capabilities as possible over critical components of the system, such as the storage engine in the nodes. Manageability Facebook introduced several important to RocksDB, such as dynamic option changes and the availability of detailed statistics for all aspects of RocksDB internal operations including compaction, that are required by enterprise grade software products. improvements . The world of enterprise software, particularly when it comes to databases, is extremely risk averse. For totally understandable reasons—risk of monetary losses and reputational damage in case of data loss or data corruption—nobody wants to be a guinea pig in this space. Production references RocksDB was developed by Facebook with the original motivation of of its massive MySQL cluster hosting its user production database from InnoDB to RocksDB. The migration was completed by resulting in a 50% storage savings for Facebook. switching the storage engine 2018 Having Facebook lead the development and maintenance of RocksDB for its most critical use cases in their multibillion dollar business is a very important endorsement, particularly for developers of databases that lack Facebook’s resources to develop and maintain their own storage engines. . RocksDB offers a key-value API, available for C++, C and Java. These are the most widely used programming languages in the distributed database world. Language bindings When considering all these 6 areas holistically, RocksDB is a very appealing choice for a distributed database developer looking for a fast, production tested storage engine. Who Uses RocksDB? Over the years, the list of known uses of RocksDB has increased dramatically. Here is a non-exhaustive list of databases that embed RocksDB that underscores its suitability as a fast storage engine: Apache Cassandra CockroachDB MySQL (MyRocks) Rockset While all these database providers probably have similar reasons for picking RocksDB over other options, Instagram's replacement of Apache Cassandra’s own Java written LSM tree with RocksDB, which is now available to all other users of Apache Cassandra, is significant. Apache Cassandra is one of the most popular NoSQL databases. RocksDB has also found wide acceptance as an embedded database outside the distributed database world for equally important, mission-critical use cases: - In the Apache Kafka ecosystem, Kafka Streams is a client library that is commonly used to build applications and microservices that consume and produce messages stored in Kafka clusters. Kafka Streams supports fault-tolerant stateful applications. RocksDB is used by default to store state in such configurations. Kafka Streams - Apache Samza offers similar functionality as Kafka Streams and it also uses RocksDB to store state in fault-tolerant configurations. Apache Samza - After looking at several options, Netflix picked RocksDB to support their SSD caching needs in their global caching system, EVCache. Netflix - Cloudera Professional Services built a near-real-time transactional analytics system for Santander UK, backed by Apache Hadoop, that implements a streaming enrichment solution that stores its state on RocksDB. Santander Group is one of Spain's largest multinational banks. As of this writing, its revenues are close to 50 billion euros with assets under management approaching 1.5 trillion euros. Santander UK - Cherami is Uber’s own durable distributed messaging system equivalent to Amazon’s SQS. Cherami chose to use RocksDB as their storage engine in their storage hosts for its performance and indexing features. Uber RocksDB: Powering High-Performance Distributed Data Systems From its beginnings as a fork of LevelDB, a key-value embedded store developed by Google infrastructure experts Jeff Dean and Sanjay Ghemawat, through the efforts and hard work of the Facebook engineers that transformed it into an enterprise-class solution apt for running mission-critical workloads, RocksDB has been able to gain widespread acceptance as the storage engine of choice for engineers looking for a battle-tested embedded storage engine.

RocksDB Is Steadily Eating the Database World

What’s shaping our technology world GitHub is not just a code hosting service with version control — it’s also an enormous developer network. The sheer size of GitHub at over 30 million accounts, more than 2 million organizations, and over 96 million repositories translates into one of the world’s most valuable development networks. How do you quantify the value of this network? And is there a way to get the top repositories? Here at U°OS, we ran the GitHub network through a simplified version¹ of and produced the top 100 most valuable repositories. our reputation algorithm The result is as fascinating as it is eclectic in the way that it does feel like a good reflection of our society’s interest in the technology and where it moves. There are the big proprietary players with open source projects — Google, Apple, Microsoft, Facebook, and even Baidu. And at the same time, there’s a Chinese anti-censorship tool. There’s Bitcoin for cryptocurrency. There’s a particle detector for CERN’s Large Hadron Collider. There are gaming projects like Space Station 13 and Cataclysm: Dark Days Ahead and a gaming engine Godot. There are education projects like freeCodeCamp, Open edX, Oppia, and Code.org. There are web and mobile app building projects like WordPress, Joomla, and Flutter to publish your content on. There are databases to store your content for the web like Ceph and CockroachDB. And there’s a search engine to navigate through the content — Elasticsearch. There are also, perhaps unsurprisingly, jailbreak projects like Cydia compatibility manager for iOS and Nintendo 3DS custom firmware. And there’s a smart home system — Home Assistant. All in all, it’s really a great outlook for the technology world: we learn, build stuff to broadcast our unique voices, we use crypto, break free from proprietary software on our hardware, and in the spare time we game in our automated homes. And the big companies open-source their projects. Before I proceed with the list, a result of running the Octoverse through the reputation algorithm also produced a value score for every individual GitHub contributor. So, if you have a GitHub account and curious, you can get your score at and convert it to a Universal Portable Reputation. https://u.community/github Top 100 projects & repositories Out of over 96 million repositories Repository: Website: Google Kubernetes Container scheduling and management https://github.com/kubernetes/kubernetes https://kubernetes.io/ Repository: Website: Apache Spark A unified analytics engine for large-scale data processing https://github.com/apache/spark http://spark.apache.org/ Repository: Website: Microsoft Visual Studio Code A source-code editor https://github.com/Microsoft/vscode https://code.visualstudio.com/ Repository: Website: NixOS Package Collection A collection of packages for the Nix package manager https://github.com/NixOS/nixpkgs https://nixos.org Repository: Website: Rust Programming language https://github.com/rust-lang/rust https://www.rust-lang.org/ Repository: Website: Firehol IP Lists Blacklists for Firehol, a firewall builder https://github.com/firehol/blocklist-ipsets https://iplists.firehol.org/ Repository: Website: Red Hat OpenShift A community distribution of Kubernetes optimized for continuous application development and multi-tenant deployment https://github.com/openshift/origin https://www.openshift.com/ Repository: Website: Ansible A deployment automation platform https://github.com/ansible/ansible https://www.ansible.com/ Repository: Website: Automattic WordPress Calypso A JavaScript and API powered front-end for WordPress.com https://github.com/Automattic/wp-calypso https://developer.wordpress.com/calypso/ Repository: Website: Microsoft .NET CoreFX Foundational class libraries for .NET Core https://github.com/dotnet/corefx https://docs.microsoft.com/en-us/dotnet/core/ Repository: Website: Microsoft .NET Roslyn .NET compiler https://github.com/dotnet/roslyn https://docs.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/ Repository: Website: Node.js A JavaScript runtime built on Chrome’s V8 JavaScript engine https://github.com/nodejs/node https://nodejs.org/en/ Repository: Website: TensorFlow Google’s machine learning framework https://github.com/tensorflow/tensorflow https://www.tensorflow.org/ Repository: Website: freeCodeCamp Code learning platform https://github.com/freeCodeCamp/freeCodeCamp https://www.freecodecamp.org/ Repository: Website: Space Station 13 A round-based roleplaying game https://github.com/tgstation/tgstation https://www.tgstation13.org/ Repository: Website: Apple Swift Apple’s programming language https://github.com/apple/swift https://swift.org/ Repository: Website: Elasticsearch A search engine https://github.com/elastic/elasticsearch https://www.elastic.co/products/elasticsearch Repository: Website: Moby An open framework to assemble specialized container systems https://github.com/moby/moby https://mobyproject.org/ Repository: Website: CockroachDB A cloud-native SQL database https://github.com/cockroachdb/cockroach https://www.cockroachlabs.com/ Repository: Website: Cydia Compatibility Checker A compatibility checker for Cydia — a package manager for iOS jailbroken devices https://github.com/jlippold/tweakCompatible https://jlippold.github.io/tweakCompatible/ Repository: Website: Servo A web browser engine https://github.com/servo/servo https://servo.org/ Repository: Website: Google Flutter Google’s mobile app SDK to create interfaces for iOS and Android https://github.com/flutter/flutter https://flutter.dev/ Repository: Website: macOS Homebrew Package Manager Default formulae for the missing package manager for macOS https://github.com/homebrew/homebrew-core https://brew.sh/ Repository: Website: Home Assistant Home automation software https://github.com/home-assistant/home-assistant https://www.home-assistant.io/ Repository: Website: Microsoft .NET CoreCLR Runtime for .NET Core https://github.com/dotnet/coreclr https://docs.microsoft.com/en-us/dotnet/core/ Repository: Website: CocoaPods Specifications Specifications for CocoaPods, a Cocoa dependency manager https://github.com/CocoaPods/Specs https://cocoapods.org/ An analytics and search dashboard for ElasticsearchRepository: Website: Elastic Kibana https://github.com/elastic/kibana https://www.elastic.co/products/kibana Repository: Website: Julia Language A technical computing language https://github.com/JuliaLang/julia https://julialang.org/ Repository: Website: Microsoft TypeScript A superset of JavaScript that compiles to plain JavaScript https://github.com/Microsoft/TypeScript https://www.typescriptlang.org/ Repository: Website: Joomla A content management system https://github.com/joomla/joomla-cms https://www.joomla.org/ Repository: Website: DefinitelyTyped A repository for TypeScript type definitions https://github.com/DefinitelyTyped/DefinitelyTyped http://definitelytyped.org/ Repository: Website: Homebrew Cask A CLI workflow for the administration of macOS applications distributed as binaries https://github.com/Homebrew/homebrew-cask https://brew.sh/ Repository: Website: Ceph A distributed object, block, and file storage platform https://github.com/ceph/ceph https://ceph.com/ Repository: Website: Go Programming language https://github.com/golang/go https://golang.org/ AMP HTML Builder Repository: Website: A way to build pages for Google AMP https://github.com/ampproject/amphtml https://amp.dev/ Repository: Website: Open edX An online education platform https://github.com/edx/edx-platform https://open.edx.org/ Repository: Website: Pandas A data analysis and manipulation library for Python https://github.com/pandas-dev/pandas https://pandas.pydata.org/ Repository: Website: Istio A platform to manage microservices https://github.com/istio/istio https://istio.io/ Repository: Website: ManageIQ A containers, virtual machines, networks, and storage management platform https://github.com/ManageIQ/manageiq http://manageiq.org/ Repository: Website: Godot Engine A multi-platform 2D and 3D game engine https://github.com/godotengine/godot https://godotengine.org/ Repository: Website: Gentoo Repository Mirror A Gentoo ebuild repository mirror https://github.com/gentoo/gentoo https://www.gentoo.org/ Repository: Website: Odoo A suite of web based open source business apps https://github.com/odoo/odoo https://www.odoo.com/ Repository: Website: Azure Documentation Documentation of Microsoft Azure https://github.com/MicrosoftDocs/azure-docs https://docs.microsoft.com/azure Repository: Website: Magento An eCommerce platform https://github.com/magento/magento2 https://magento.com/ Repository: Website: Saltstack Software to automate the management and configuration of any infrastructure or application at scale https://github.com/saltstack/salt https://www.saltstack.com/ Repository: Website: AdGuard Filters Ad blocking filters for AdGuard https://github.com/AdguardTeam/AdguardFilters https://adguard.com/en/welcome.html Repository: Website: Symfony A PHP framework https://github.com/symfony/symfony https://symfony.com/ Repository: Website: CMS Software for the Large Hadron Collider Particle detector software components for CERN’s Large Hadron Collider https://github.com/cms-sw/cmssw http://cms-sw.github.io/ Repository: Website: Red Hat OpenShift OpenShift installation and configuration management https://github.com/openshift/openshift-ansible https://www.openshift.com/ Repository: Website: ownCloud Personal cloud software https://github.com/owncloud/core https://owncloud.org/ Repository: Website: gRPC A remote procedure call (RPC) framework https://github.com/grpc/grpc https://grpc.io/ Repository: Website: Liferay An enterprise web platform https://github.com/brianchandotcom/liferay-portal https://www.liferay.com/ Repository: Website: CommCare HQ A mobile data collection platform https://github.com/dimagi/commcare-hq https://www.commcarehq.org/ Repository: Website: WordPress Gutenberg An editor plugin for WordPress https://github.com/WordPress/gutenberg https://wordpress.org/gutenberg/ Repository: Website: PyTorch A Python package for Tensor computation and deep neural networks https://github.com/pytorch/pytorch https://pytorch.org/ Repository: Website: Kubernetes Test Infrastructure A test-infra repository for Kubernetes https://github.com/kubernetes/test-infra https://kubernetes.io/ Repository: Website: Keybase Keybase client repository https://github.com/keybase/client https://keybase.io/ Repository: Website: Facebook React A JavaScript library for building user interfaces https://github.com/facebook/react https://reactjs.org/ Repository: Website: Code.org Code learning resource https://github.com/code-dot-org/code-dot-org https://code.org/ Repository: Website: Bitcoin Core Bitcoin client software https://github.com/bitcoin/bitcoin https://bitcoincore.org/ Repository: Website: Arm Mbed OS A platform operating system for the Internet of Things https://github.com/ARMmbed/mbed-os https://www.mbed.com Repository: Website: scikit-learn A Python module for machine learning https://github.com/scikit-learn/scikit-learn https://scikit-learn.org Repository: Website: Nextcloud A self-hosted productivity platform https://github.com/nextcloud/server https://nextcloud.com/ Repository: Website: Helm Charts A curated list of applications for Kubernetes https://github.com/helm/charts https://kubernetes.io/ Repository: Website: Terraform An infrastructure management tool https://github.com/hashicorp/terraform https://www.terraform.io/ Repository: Website: Ant Design A UI design language https://github.com/ant-design/ant-design https://ant.design/ Repository: Website: Phalcon Framework Documentation Documentation for Phalcon, a PHP framework https://github.com/phalcon/docs https://docs.phalconphp.com Repository: Website: Documentation for CMS Software for the Large Hadron Collider Documentation for CMS Software for CERN’s Large Hadron Collider https://github.com/cms-sw/cms-sw.github.io http://cms-sw.github.io/ Repository: Website: Apache Kafka Mirror A mirror for Apache Kafka, a distributed streaming platform https://github.com/apache/kafka https://kafka.apache.org/ Repository: Website: Electron A framework to write cross-platform desktop applications using JavaScript, HTML and CSS https://github.com/electron/electron https://electronjs.org/ Repository: Website: Zephyr Project A real-time operating system https://github.com/zephyrproject-rtos/zephyr https://www.zephyrproject.org/ Repository: Website: The web-platform-tests Project A cross-browser testsuite for the Web-platform stack https://github.com/web-platform-tests/wpt https://www.w3.org/ Repository: Website: Marlin Firmware Optimized firmware for RepRap 3D printers based on the Arduino platform https://github.com/MarlinFirmware/Marlin http://marlinfw.org/ Repository: Website: Apache MXNet A library for deep learning https://github.com/apache/incubator-mxnet https://mxnet.apache.org/ Repository: Website: Apache Beam A unified programming model https://github.com/apache/beam https://beam.apache.org/ Repository: Website: Fastlane A build and release automaton for iOS and Android apps https://github.com/fastlane/fastlane https://fastlane.tools/ Repository: Website: Kubernetes Website and Documentation A repository for the Kubernetes website and documentation https://github.com/kubernetes/website https://kubernetes.io Repository: Website: Ruby on Rails A web-application framework https://github.com/rails/rails https://rubyonrails.org/ Repository: Website: Zulip Team chat software https://github.com/zulip/zulip https://zulipchat.com/ Repository: Website: Laravel A web application framework https://github.com/laravel/framework https://laravel.com/ Repository: Website: Baidu PaddlePaddle Baidu’s deep learning framework https://github.com/PaddlePaddle/Paddle http://www.paddlepaddle.org/ Repository: Website: Gatsby A web application framework https://github.com/gatsbyjs/gatsby https://www.gatsbyjs.org/ Repository: Website: Rust Crate Registry Rust’s community package registry https://github.com/rust-lang/crates.io-index https://crates.io/ Repository: Website: Nintendo 3DS Custom Firmware A complete guide to 3DS custom firmware https://github.com/hacks-guide/Guide_3DS https://3ds.hacks.guide/ Repository: Website: TiDB A NewSQL database https://github.com/pingcap/tidb https://pingcap.com Repository: Website: Angular CLI CLI tool for Angular, a Google web application framework https://github.com/angular/angular-cli https://cli.angular.io/ Repository: Website: MAPS.ME Offline OpenStreetMap maps for iOS and Android https://github.com/mapsme/omim https://maps.me/ Repository: Website: Eclipse Che A cloud IDE for Eclipse https://github.com/eclipse/che http://www.eclipse.org/che/ Repository: Website: Brave Browser A browser with native BAT cryptocurrency https://github.com/brave/browser-laptop https://www.brave.com/ Repository: Website: Patchwork A repository to learn Git https://github.com/jlord/patchwork http://jlord.us/patchwork/ Repository: Website: Angular Material Component infrastructure and Material Design components for Angular, a Google web application framework https://github.com/angular/components https://material.angular.io/ Repository: Website: Python Programming language https://github.com/python/cpython https://www.python.org/ Repository: Website: Space Station 13 A round-based roleplaying game https://github.com/vgstation-coders/vgstation13 http://ss13.moe/ Repository: Website: Cataclysm: Dark Days Ahead A turn-based survival game https://github.com/CleverRaven/Cataclysm-DDA http://cataclysmdda.org/ Repository: Website: Material-UI React components that implement Google’s Material Design https://github.com/mui-org/material-ui https://material-ui.com/ Repository: Website: Ionic A Progressive Web Apps development framework https://github.com/ionic-team/ionic https://ionicframework.com/ Repository: Website: Oppia A tool for collaboratively building interactive lessons https://github.com/oppia/oppia https://www.oppia.org Repository: Website: Alluxio A virtual distributed storage system https://github.com/Alluxio/alluxio https://www.alluxio.io/ Repository: Website: None XX Net A Chinese web proxy and anti-censorship tool https://github.com/XX-net/XX-Net Repository: Website: Microsoft .NET CLI A CLI tool for .NET https://github.com/dotnet/cli https://docs.microsoft.com/en-us/dotnet/core/tools/ [1] The explanation of the calculation of the simplified version is at the . U°OS Network GitHub repository

GitHub’s Top 100 Most Valuable Repositories Out of 96 Million

Apache Cassandra — The minimum internals you need to know

and Register for the Event on EventBrite, meet the attendees on Facebook. This message is brought to you by , Quantcast , & Meraki Developers . The Inaugural Hacker Noon Party starts at 6 on June 20th in San Francisco. COVO Quantcast, the way to understand, reach and influence all of your customers, will help pick up the tab, and is excited to meet you there. To learn more about , tweet their Head of Talent Strategy, job opportunities at Quantcast @Lumen. He’s smarter than he looks, and he looks pretty smart. Grandma Queen Elizabeth is into it. Cisco Meraki is giving away one million dollars worth of Meraki gear to help developers to use their APIs to automate networks, measure foot traffic, engage with their users via proximity marketing, and locate or track things with BLE and WiFi. Come meet ’s Meraki Colin Lowenberg at the party or tweet him . @colo Cisco Meraki is the leader in cloud managed IT with over one million active networks. The party will be hosted at COVO, San Francisco’s premier co-working venue. To learn more about their desks, offices, and event space call 415–886–6206. You must and can register for the Event on EventBrite, meet the attendees on Facebook. *, maybe even more* *… Does your brand want to become a sponsor of the Hacker Noon party? Sponsor benefits include your VIP area at the event, adverts in our emails to 100,ooo+ tech people (that milestone is kinda the reason we’re throwing this party). Interested, please email* We’d like to give away more drinks, more swag jobs , or Twitter DM me Partners@AMIpublications.com David Smooke (@davidsmooke). If you’re not in SF and can’t party with us, no worries! We wanted to invite you anyways because we’re one community. We should be hosting more of these elsewhere in the world, so drop us a line and let us know where we should throw one next. Trying something new, some of our friends are hiring. Heres our list for this month, with links to the listings on their name: — Solve big data problems in the cloud while changing banking for good. Hiring all over the US and the UK. Capital One — An open source, cloud-native SQL database for building global, scalable cloud services that survive disasters. Cockroach Labs — the way to understand, reach and influence all of your customers. Quantcast Want to get on the list of who’s hiring? Email Partners@AMIpublications.com Sorry if we disturbed your inbox! We want to say, thanks for reading , & here are ten stories of positivity in tech: Hacker Noon 10: A New Kind of Social Network: Emotional Intelligence by . Mark Nadal Could there be social networks without the haters, trolls, and downvotes? And how would a tech product go about supporting only positivity? Dig in. 9: Give Back In A Big Way by Peter Schroeder 🚀 Learn from Carnegie’s Gospel of Wealth. 8: Eight Amazing Sites With Best Free Stock Photos by Damir Mujic Photos, photos, photos. Here are 8 places giving away a ton of them. 7: Seven Reasons to Give Employees More Equity by Ash Rust People — not capital — build companies. Don’t short change your people. 6: How I tweaked pomodoro technique for better results in productivity by Febin John James The pomodoro technique is to decide on a task, set a timer for 25–30 mins and work until it rings. In this case, it’s been modified for developers. 5: How tech is softening hearts by babulous Technology can also play the role of providing needy recipients a direct path to willing donors. 4: Thirteen Simple Rules for Good Coding (from my fifteen years of experience) by Andrey Nikisha ev Nothing like experience to K.I.S.S. 3: Make A Fresh Start With These Powerful Habits by , Faisal Hoque Change your outlook, change your results. Pretty much reading down this road. all of Faisal’s stories 2: The future is algorithms, not code by Intercom Developer Support Engineer Cathal Horan This post is positive because of how it emphasizes the thought and function behind the code. 1: Stop Paying for SSL Certificates by Christopher Watson We’ve all had our issues with SSL ;-) Here’s a post about getting yours for free. Until next time, don’t take the realities of the world for granted, David Smooke Hacker Noon P.S. Read more about , Artificial Intelligence. , APIs , Developer Bootcamp , Javascript , Learning AI if You Suck at Math , MVP Lessons , Product Management Life , The Countdown , & Tech Careers Venture Capital. P.P.S. If you want free tickets to the event just like our brand new Facebook page and message us your favorite Hacker Noon article (FYI here’s latest & ). Then we’ll give you a free ticket code — trending stories while supplies last (I always wanted to use that phrase on a game show of some kind). P.P.S. This message is brought to you by our event sponsors, Quantcast, Meraki Developers & . Don’t forget to COVO register for the Event on EventBrite, and meet the attendees on Facebook.

You’re Invited to Our Inaugural Hacker Noon Party

scurries onto the database scene as a resilient and scalable distributed SQL database. Drawing inspiration from the tenacity of its insect namesake, CockroachDB boasts high availability even in the face of hardware failures. Its distributed architecture spans multiple nodes, mirroring the adaptability of its insect counterpart. CockroachDB With strong consistency and ACID transaction support, CockroachDB becomes a reliable choice for applications requiring data accuracy and reliability, thriving in dynamic environments and effortlessly managing the complexities of distributed data. This blog introduces using MinIO as a changefeed sink for CockroachDB. By doing so, you not only benefit from CockroachDB's strengths but also leverage the , , and of MinIO. Use this as a guide to establishing an enterprise-grade CDC strategy. It is inspired by awesome repo. durability scalability performance this What is CDC? CDC is a smart database management technique that tracks and captures changes in a relational database like CockroachDB. It acts like a monitor, detecting CRUD Operations like INSERTS, UPDATES and DELETES in real time. CDC's strength lies in its ability to identify only the altered data, making it more efficient in terms of network bandwidth and expense than traditional methods of replication. This efficiency is crucial for tasks like data integration, real-time analytics, and maintaining consistency across distributed systems. CDC is truly a prerequisite for real-time connected data fabrics and remains a fundamental tool for keeping databases synchronized and maintaining reliability in dynamic data environments. This continuous stream of real-time data updates provides a rich foundation for training and optimizing machine learning models which require huge volumes of up-to-date data for success. Changefeed Sinks with CockroachDB in CockroachDB are like data pipelines that efficiently funnel CRUD operations happening in the database to an external destination. In this instance, MinIO serves as one such destination. When configured as a sink, MinIO becomes the repository for the continuous stream of changes, offering a durable and scalable storage solution for CDC operations. Changefeed sinks One benefit of this approach is that you can use CDC data in your MinIO bucket to replicate your data strategy across clouds. For example, If your CockroachDB is hosted in AWS, but you need your data on-prem for your models to run, you can do that with this approach. This can be an effective way to implement a multi-cloud data strategy. Prerequisites To follow this guide, make sure you have Docker Compose installed. You can install Docker Engine and Docker Compose separately or together using Docker Desktop. The easiest way is to go for Docker Desktop. Check if Docker Compose is installed by running this command: docker-compose --version You will need a self-hosted, Enterprise license of CockroachDB. For this local deployment of CockroachDB, please note that CockroachDB on ARM for MacOS is experimental and is not yet ready for production. Getting Started To get started clone or download the project folder from location. this Open a terminal window, navigate to the project folder and run the following command: docker-compose up -d This command instructs Docker Compose to read the configuration from the `docker-compose.yml` file, create and start the services defined in it, and run them in the background, allowing you to use the terminal for other tasks without being tied to the container's console output. After running the command you should be able to see the containers are up and running. You can access the Cockroach UI at . Verify that your node is live. http://127.0.0.1:8080 You can access the MinIO UI at . Log in with the username and password combination of . http://127.0.0.1:9001 minioadmin:minioadmin Once you log in, you should be able to verify that the mc command in the was executed and a bucket named was automatically created. docker-compose.yml cockroach SQL Commands Once your containers are up and running, you’re now ready to run SQL commands. Run the following command in a terminal window in the same folder where you downloaded the tutorial files and started your containers. This command executes an interactive SQL shell inside the container, connecting to the CockroachDB instance running in that container, and allowing you to enter SQL queries. crdb-1 docker exec -it crdb-1 ./cockroach sql --insecure You should see the following if you executed the shell correctly. #
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v19.2.2 (x86_64-unknown-linux-gnu, built 2019/12/11 01:33:43, go1.12.12) (same version as client)
# Cluster ID: 0a668a2d-056d-4203-a996-217ca6169f80
#
# Enter \? for a brief introduction.
# As noted in the prerequisites, you need an Enterprise CockroachDB account to set up CDC. Use the same terminal window to enter the next commands. SET CLUSTER SETTING cluster.organization = '<organization name>';

SET CLUSTER SETTING enterprise.license = '<secret>';

SET CLUSTER SETTING kv.rangefeed.enabled = true; Crushing CDC You can now build your database and tables. The SQL below creates a new database named , switches the current database context to , creates a table named to store information about machine learning models, and inserts two rows of data into this table representing the performance metrics of specific models. Execute these commands in the same terminal. ml_data ml_data model_performance CREATE DATABASE ml_data;

SET DATABASE = ml_data;

CREATE TABLE model_performance (
     model_id INT PRIMARY KEY,
     model_name STRING,
     accuracy FLOAT,
     training_time INT);

INSERT INTO model_performance VALUES
   (1, 'NeuralNetworkV1', 0.85, 120),
   (2, 'RandomForestV2', 0.92, 150); Next, run the following command to create a changefeed for the table in CockroachDB and configure it to stream updates to MinIO. model_performance CREATE CHANGEFEED FOR TABLE model_performance INTO 'experimental-s3://cockroach?AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin&AWS_ENDPOINT=http://minio:9000&AWS_REGION=us-east-1' with updated, resolved='10s'; Navigate to the Cockroach UI at to verify that your changefeed has been successfully created and a high-water timestamp has been established. http://127.0.0.1:8080 Make a change to your data to see the changefeed in action. UPDATE model_performance SET model_name = 'ResNet50' WHERE model_id = 1; Run the following command to make sure the command was executed. SELECT * FROM model_performance ;
  model_id |   model_name   | accuracy | training_time  
+----------+----------------+----------+---------------+
         1 | ResNet50       |     0.88 |           130  
         2 | RandomForestV2 |     0.92 |           150 In a production environment, your transactions would populate MinIO without the need for this step. Navigate back to the MinIO UI at‘ to see your changefeed in action. http://127.0.0.1:9001 End of the Line In this tutorial, you’ve gone through the process of creating a changefeed sink with MinIO to enable a CDC strategy for your enterprise-licensed CockroachDB. This CDC strategy sets the stage for a and continuously synchronized data fabric for your most critical data. When you absolutely need to have a perfect replica of your data for data exploration, analytics or AI applications the combination of CockroachDB and MinIO is your winning strategy. resilient Evolve what has been outlined here and you will continue to support better decision-making, facilitating AI/ML endeavors, and maintaining the reliability of your dynamic data environment. MinIO today for all your cockroach-related needs. MinIO - CockroachDBs check-in, but they don't check out. Reach out to us if you have any questions or trouble with bugs at or on . Download hello@min.io Slack I want to thank the folks at CockroachDB – big shoutout to John Billingsley – for helping us with this post.

Stories, News & Mentions about COCKROACHLABS

Cockroachlabs’s Stories on HackerNoon