Cockroach Labs is the creator of CockroachDB, the most highly evolved cloud-native, distributed SQL database on the planet. We help companies of all sizes — and the apps they develop — scale fast, survive anything, and thrive everywhere.

Cockroach Labs

Kristy is in her final year of computer science at the University of Waterloo (UW). Since her enrollment, she has interned for Paypal, Cognite AS, Peggy, Cockroach Labs and Google — all as a software engineer. When she graduates, she plans to work as an engineering manager. At the moment, Kristy is a technical project lead for UW Blueprint initiative, which focuses on accessible technology and promoting public and social welfare. The initiative partners with non-profit organizations to provide websites, mobile applications and analysis tools for free. Under UW Blueprint, Kristy leads a project that builds web applications to help Planet Read coordinate volunteers to translate and review children’s stories. “Currently how they process their translations is through email and Word documents, so we’re building a platform to hopefully change that,” she says. “It’s kind of like my brainchild for the last few months and we will be deploying in [January].” Kristy’s journey to UW’s computer science program was possible through a handful of positive reinforcements. Growing up in Guelph, ON. alongside her older sister to Chinese immigrant parents, Kristy fostered a keen interest in computer science through video games and mathematics. From catching Pokemons to rummaging through online indie games, Kristy began to see the correlation of mathematics in them. “I always liked tinkering with things…it seemed like computer science was a natural concept to try out,” she says. Although in high school, Kristy did not find the computer science courses to be challenging enough, she admits having an excellent teacher was a game-changer. She is also proud that she was part of a tight-knit comradery of the few female students looking to venture into computer science careers. “It gave me a lot of inspiration to try it out myself, and I really don’t know where I’d be if they were not there,” the 22-year-old says. “I think that it was really huge for me.” Then there were hackathons. “Hackathons really connected me with the community I wanted to be a part of, I met a lot of students from Waterloo while I was in high school in the hackathon community.” she says. For Kristy, hackathons were a way for her to connect with people who wanted to build and create. What attracted her to hackathons weren’t the elaborate and complex projects, but the simple ones that embodied the spirit of learning and having fun. Throughout high school, hackathons became Kristy’s staple extracurricular, but for her it was more than just an after-school activity. In 2016, she would become the founder of SpartaHacks, introducing Guelph to the first high school hackathon, organized alongside her friends from September to December. This was Kristy’s first time organizing and finding major success. “I really wanted to give back to a community that gave me so much.” Kristy says. “My school, on a smaller level, didn’t have much of a computer science environment or community, I felt really passionate about bringing it to my school and convincing my friends to enjoy these things.” As Kristy found herself studying at UW, her passion for hackathons continued. Soon, Kristy will be working with Hack the North as a logistics organizer and backend engineer. “I wanted to find people I could relate to more, I think it’s important to be able to have people in your life who can support you in the way that you need to be supported,” she says. “It’s hard.” Kristy does not shy away from talking about identity politics in the computer science industry, including her time at UW. As a queer woman of colour, finding a space in the industry isn’t as easy as it seems. In her experience, having mentors who either are allies or individuals from your community is a way to feel connected to a stereotypically heteronormative space. “It was huge for me that I had mentors in high school, who were women in computer science.” “It’s great to have a diverse network, and be friends with people from all walks of life, regardless of the background you come from.” Kristy spoke of the hope she feels in the growth of inclusion in the industry. “I think the onus is on everyone to make the space more welcoming and inclusive,” she says. When looking at Ontario, Kristy stated that there needs to be more change in regards to learning in programming, as computer science isn’t offered in every school, public or private. This would mean providing more resources, funding programming camps and implementing programming courses in schools that are subsidized. “If you grew up in rural Ontario, where there might not even be upper calculus classes, then it’s going to be hard to get a hold of programming classes,” she says. “Having the resources and the opportunities to go to programming camp, or go to hackathons, can be really influential with that.” With the ever-growing environment of STEM careers, Kristy believes there are many ways for students and people interested in computer science to increase their skills. For Kristy, co-op programs changed her life for the better, giving her a chance to explore different paths. Kristy spent time in California and Norway where she worked remotely for international companies during the pandemic. Through these experiences, her love for startups and entrepreneurship grew. “You receive valuable hands-on experiences on building software and websites from scratch,” she says. The desire and focus to build and create is what Kristy believes is the most important factor in figuring out what a student or someone just entering the STEM industry needs. She also emphasizes the need for expanding people’s creativity, and not just sticking to one type of building. “Take time to take care of yourself, do some art, some construction, woodwork, things like that,” she says. “Those are all important in keeping the spirit of building things alive, and to keep the spirit healthy.” There’s more interesting interviews like this at Thinking Cap!

In Conversation with a Computer Science Major, Kristy Gao

. Because one of the most common use cases of the new databases is storing data that is generated by high-throughput sources, it is important that the store engine is able to handle write-intensive workloads, all while offering acceptable read performance. RocksDB implements what is known in the database literature as a aka LSM tree. Technical design log-structured merge tree A Brief History of Distributed Databases The era of Web 2.0 brought with it a renewed interest in database design. While traditional RDBMS databases served well the data storage and data processing needs of the enterprise world from their commercial inception in the late 1970s until the dotcom era, the large amounts of data processed by the new applications—and the speed at which this data needs to be processed—required a new approach. For a great overview on the need for these new database designs, I highly recommend watching that database guru Michael Stonebraker delivered for Stanford’s Computer Systems Colloquium. this presentation The new databases that have emerged during this time have adopted names such as NoSQL and NewSQL, emphasizing that good old SQL databases fell short when it came to meeting the new demands. Despite their different design choices for particular protocols, these databases have adopted, for the most part, a shared-nothing, distributed computing architecture. While the processing power of every computing system is ultimately limited by physical constraints and, in cases such as distributed databases where parallel executions are involved, by the implications of , most of these systems offer the theoretical possibility of unlimited horizontal capacity scaling for both compute and storage. Amdahl's law Each node represents a unit of compute and storage that can be added to the system as needed. However, as Cockroach Labs CEO and co-founder Spencer Kimball explains in the case of CockroachDB, designing one of these new databases from scratch is a herculean task that requires highly knowledgeable and skillful engineers working in coordination and making very carefully thought decisions. here For databases such as CockroachDB, having a reliable, high-performance way to store and retrieve data from stable storage is essential. Designing a library that provides fast stable storage leveraging either filesystem or raw devices is a very difficult problem because of the elevated number of edge cases that are required to get right. Providing Fast Storage with RocksDB is a library that solves the problem of abstracting access to local stable storage. It allows software engineers to focus their energies on the design and implementation of other areas of their systems with the peace of mind of relying on RocksDB for access to stable storage, knowing that it currently runs some of the most demanding database workloads anywhere on the planet at and other equally challenging . RocksDB Facebook environments The advantages of RocksDB over other store engines are: . Because one of the most common use cases of the new databases is storing data that is generated by high-throughput sources, it is important that the store engine is able to handle write-intensive workloads, all while offering acceptable read performance. RocksDB implements what is known in the database literature as a aka LSM tree. Technical design log-structured merge tree Going into the details of LSM trees, and RocksDB’s implementation of the same, is out of the scope of this blog, but suffice it to say that it’s an indexing structure optimized to handle high-volume—sequential or random—write workloads. This is accomplished by treating every write as an append operation. A mechanism, that goes by the name of runs—transparently for the developer—in the background, removing data that is no longer relevant such as deleted keys or older versions of valid keys. compaction Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/ Through the clever use of , RocksDB also offers great read performance making RocksDB the ideal candidate on which to base distributed databases. bloom filters The other popular choice to base storage engines on is b-trees. InnoDB, MySQL’s default storage engine, is an example of a store engine implementing a b-tree derivative, in particular, what is known as a b+tree. . The choice of a given technical design for performance reasons needs to be backed with empirical verification of the choice. During his time at Facebook, in the context of the project, a fork of MySQL that replaces InnoDB with RocksDB as MySQL’s storage engine, Mark Callaghan performed extensive and rigorous performance measurements to compare MySQL performance on InnoDB vs on RocksDB. Details can be found . Performance MyRocks here Not surprisingly, RocksDB regularly comes out as vastly superior in write-intensive benchmarks. Interestingly, while InnoDB was also regularly better than RocksDB in read-intensive benchmarks, this advantage, in relative terms, was not as big as the advantage RocksDB provides in the case of write-intensive tasks over InnoDB. Here is an example in the case of a I/O bound benchmark on : Intel NUC Source: https://smalldatum.blogspot.com/2017/11/insert-benchmark-io-bound-intel-nuc.html . RocksDb provides several tunable parameters to extract the best performance on different hardware configurations. While the technical design provides an architectural reason to favor one type of solution over another, achieving optimal performance on particular use cases usually requires the flexibility of tuning certain parameters for those use cases. Tunability RocksDB provides a long list of parameters that can be used for this purpose. Samsung’s Praveen Krishnamoorthy presented at the 2015 annual meetup an extensive on how RocksDB can be tuned to accommodate different workloads. study . In mission-critical solutions such as distributed databases, it is essential to have as much control and monitoring capabilities as possible over critical components of the system, such as the storage engine in the nodes. Manageability Facebook introduced several important to RocksDB, such as dynamic option changes and the availability of detailed statistics for all aspects of RocksDB internal operations including compaction, that are required by enterprise grade software products. improvements . The world of enterprise software, particularly when it comes to databases, is extremely risk averse. For totally understandable reasons—risk of monetary losses and reputational damage in case of data loss or data corruption—nobody wants to be a guinea pig in this space. Production references RocksDB was developed by Facebook with the original motivation of of its massive MySQL cluster hosting its user production database from InnoDB to RocksDB. The migration was completed by resulting in a 50% storage savings for Facebook. switching the storage engine 2018 Having Facebook lead the development and maintenance of RocksDB for its most critical use cases in their multibillion dollar business is a very important endorsement, particularly for developers of databases that lack Facebook’s resources to develop and maintain their own storage engines. . RocksDB offers a key-value API, available for C++, C and Java. These are the most widely used programming languages in the distributed database world. Language bindings When considering all these 6 areas holistically, RocksDB is a very appealing choice for a distributed database developer looking for a fast, production tested storage engine. Who Uses RocksDB? Over the years, the list of known uses of RocksDB has increased dramatically. Here is a non-exhaustive list of databases that embed RocksDB that underscores its suitability as a fast storage engine: Apache Cassandra CockroachDB MySQL (MyRocks) Rockset While all these database providers probably have similar reasons for picking RocksDB over other options, Instagram's replacement of Apache Cassandra’s own Java written LSM tree with RocksDB, which is now available to all other users of Apache Cassandra, is significant. Apache Cassandra is one of the most popular NoSQL databases. RocksDB has also found wide acceptance as an embedded database outside the distributed database world for equally important, mission-critical use cases: - In the Apache Kafka ecosystem, Kafka Streams is a client library that is commonly used to build applications and microservices that consume and produce messages stored in Kafka clusters. Kafka Streams supports fault-tolerant stateful applications. RocksDB is used by default to store state in such configurations. Kafka Streams - Apache Samza offers similar functionality as Kafka Streams and it also uses RocksDB to store state in fault-tolerant configurations. Apache Samza - After looking at several options, Netflix picked RocksDB to support their SSD caching needs in their global caching system, EVCache. Netflix - Cloudera Professional Services built a near-real-time transactional analytics system for Santander UK, backed by Apache Hadoop, that implements a streaming enrichment solution that stores its state on RocksDB. Santander Group is one of Spain's largest multinational banks. As of this writing, its revenues are close to 50 billion euros with assets under management approaching 1.5 trillion euros. Santander UK - Cherami is Uber’s own durable distributed messaging system equivalent to Amazon’s SQS. Cherami chose to use RocksDB as their storage engine in their storage hosts for its performance and indexing features. Uber RocksDB: Powering High-Performance Distributed Data Systems From its beginnings as a fork of LevelDB, a key-value embedded store developed by Google infrastructure experts Jeff Dean and Sanjay Ghemawat, through the efforts and hard work of the Facebook engineers that transformed it into an enterprise-class solution apt for running mission-critical workloads, RocksDB has been able to gain widespread acceptance as the storage engine of choice for engineers looking for a battle-tested embedded storage engine.

RocksDB Is Steadily Eating the Database World

and Register for the Event on EventBrite, meet the attendees on Facebook. This message is brought to you by , Quantcast , & Meraki Developers . The Inaugural Hacker Noon Party starts at 6 on June 20th in San Francisco. COVO Quantcast, the way to understand, reach and influence all of your customers, will help pick up the tab, and is excited to meet you there. To learn more about , tweet their Head of Talent Strategy, job opportunities at Quantcast @Lumen. He’s smarter than he looks, and he looks pretty smart. Grandma Queen Elizabeth is into it. Cisco Meraki is giving away one million dollars worth of Meraki gear to help developers to use their APIs to automate networks, measure foot traffic, engage with their users via proximity marketing, and locate or track things with BLE and WiFi. Come meet ’s Meraki Colin Lowenberg at the party or tweet him . @colo Cisco Meraki is the leader in cloud managed IT with over one million active networks. The party will be hosted at COVO, San Francisco’s premier co-working venue. To learn more about their desks, offices, and event space call 415–886–6206. You must and can register for the Event on EventBrite, meet the attendees on Facebook. *, maybe even more* *… Does your brand want to become a sponsor of the Hacker Noon party? Sponsor benefits include your VIP area at the event, adverts in our emails to 100,ooo+ tech people (that milestone is kinda the reason we’re throwing this party). Interested, please email* We’d like to give away more drinks, more swag jobs , or Twitter DM me Partners@AMIpublications.com David Smooke (@davidsmooke). If you’re not in SF and can’t party with us, no worries! We wanted to invite you anyways because we’re one community. We should be hosting more of these elsewhere in the world, so drop us a line and let us know where we should throw one next. Trying something new, some of our friends are hiring. Heres our list for this month, with links to the listings on their name: — Solve big data problems in the cloud while changing banking for good. Hiring all over the US and the UK. Capital One — An open source, cloud-native SQL database for building global, scalable cloud services that survive disasters. Cockroach Labs — the way to understand, reach and influence all of your customers. Quantcast Want to get on the list of who’s hiring? Email Partners@AMIpublications.com Sorry if we disturbed your inbox! We want to say, thanks for reading , & here are ten stories of positivity in tech: Hacker Noon 10: A New Kind of Social Network: Emotional Intelligence by . Mark Nadal Could there be social networks without the haters, trolls, and downvotes? And how would a tech product go about supporting only positivity? Dig in. 9: Give Back In A Big Way by Peter Schroeder 🚀 Learn from Carnegie’s Gospel of Wealth. 8: Eight Amazing Sites With Best Free Stock Photos by Damir Mujic Photos, photos, photos. Here are 8 places giving away a ton of them. 7: Seven Reasons to Give Employees More Equity by Ash Rust People — not capital — build companies. Don’t short change your people. 6: How I tweaked pomodoro technique for better results in productivity by Febin John James The pomodoro technique is to decide on a task, set a timer for 25–30 mins and work until it rings. In this case, it’s been modified for developers. 5: How tech is softening hearts by babulous Technology can also play the role of providing needy recipients a direct path to willing donors. 4: Thirteen Simple Rules for Good Coding (from my fifteen years of experience) by Andrey Nikisha ev Nothing like experience to K.I.S.S. 3: Make A Fresh Start With These Powerful Habits by , Faisal Hoque Change your outlook, change your results. Pretty much reading down this road. all of Faisal’s stories 2: The future is algorithms, not code by Intercom Developer Support Engineer Cathal Horan This post is positive because of how it emphasizes the thought and function behind the code. 1: Stop Paying for SSL Certificates by Christopher Watson We’ve all had our issues with SSL ;-) Here’s a post about getting yours for free. Until next time, don’t take the realities of the world for granted, David Smooke Hacker Noon P.S. Read more about , Artificial Intelligence. , APIs , Developer Bootcamp , Javascript , Learning AI if You Suck at Math , MVP Lessons , Product Management Life , The Countdown , & Tech Careers Venture Capital. P.P.S. If you want free tickets to the event just like our brand new Facebook page and message us your favorite Hacker Noon article (FYI here’s latest & ). Then we’ll give you a free ticket code — trending stories while supplies last (I always wanted to use that phrase on a game show of some kind). P.P.S. This message is brought to you by our event sponsors, Quantcast, Meraki Developers & . Don’t forget to COVO register for the Event on EventBrite, and meet the attendees on Facebook.

You’re Invited to Our Inaugural Hacker Noon Party

What’s shaping our technology world GitHub is not just a code hosting service with version control — it’s also an enormous developer network. The sheer size of GitHub at over 30 million accounts, more than 2 million organizations, and over 96 million repositories translates into one of the world’s most valuable development networks. How do you quantify the value of this network? And is there a way to get the top repositories? Here at U°OS, we ran the GitHub network through a simplified version¹ of and produced the top 100 most valuable repositories. our reputation algorithm The result is as fascinating as it is eclectic in the way that it does feel like a good reflection of our society’s interest in the technology and where it moves. There are the big proprietary players with open source projects — Google, Apple, Microsoft, Facebook, and even Baidu. And at the same time, there’s a Chinese anti-censorship tool. There’s Bitcoin for cryptocurrency. There’s a particle detector for CERN’s Large Hadron Collider. There are gaming projects like Space Station 13 and Cataclysm: Dark Days Ahead and a gaming engine Godot. There are education projects like freeCodeCamp, Open edX, Oppia, and Code.org. There are web and mobile app building projects like WordPress, Joomla, and Flutter to publish your content on. There are databases to store your content for the web like Ceph and CockroachDB. And there’s a search engine to navigate through the content — Elasticsearch. There are also, perhaps unsurprisingly, jailbreak projects like Cydia compatibility manager for iOS and Nintendo 3DS custom firmware. And there’s a smart home system — Home Assistant. All in all, it’s really a great outlook for the technology world: we learn, build stuff to broadcast our unique voices, we use crypto, break free from proprietary software on our hardware, and in the spare time we game in our automated homes. And the big companies open-source their projects. Before I proceed with the list, a result of running the Octoverse through the reputation algorithm also produced a value score for every individual GitHub contributor. So, if you have a GitHub account and curious, you can get your score at and convert it to a Universal Portable Reputation. https://u.community/github Top 100 projects & repositories Out of over 96 million repositories Repository: Website: Google Kubernetes Container scheduling and management https://github.com/kubernetes/kubernetes https://kubernetes.io/ Repository: Website: Apache Spark A unified analytics engine for large-scale data processing https://github.com/apache/spark http://spark.apache.org/ Repository: Website: Microsoft Visual Studio Code A source-code editor https://github.com/Microsoft/vscode https://code.visualstudio.com/ Repository: Website: NixOS Package Collection A collection of packages for the Nix package manager https://github.com/NixOS/nixpkgs https://nixos.org Repository: Website: Rust Programming language https://github.com/rust-lang/rust https://www.rust-lang.org/ Repository: Website: Firehol IP Lists Blacklists for Firehol, a firewall builder https://github.com/firehol/blocklist-ipsets https://iplists.firehol.org/ Repository: Website: Red Hat OpenShift A community distribution of Kubernetes optimized for continuous application development and multi-tenant deployment https://github.com/openshift/origin https://www.openshift.com/ Repository: Website: Ansible A deployment automation platform https://github.com/ansible/ansible https://www.ansible.com/ Repository: Website: Automattic WordPress Calypso A JavaScript and API powered front-end for WordPress.com https://github.com/Automattic/wp-calypso https://developer.wordpress.com/calypso/ Repository: Website: Microsoft .NET CoreFX Foundational class libraries for .NET Core https://github.com/dotnet/corefx https://docs.microsoft.com/en-us/dotnet/core/ Repository: Website: Microsoft .NET Roslyn .NET compiler https://github.com/dotnet/roslyn https://docs.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/ Repository: Website: Node.js A JavaScript runtime built on Chrome’s V8 JavaScript engine https://github.com/nodejs/node https://nodejs.org/en/ Repository: Website: TensorFlow Google’s machine learning framework https://github.com/tensorflow/tensorflow https://www.tensorflow.org/ Repository: Website: freeCodeCamp Code learning platform https://github.com/freeCodeCamp/freeCodeCamp https://www.freecodecamp.org/ Repository: Website: Space Station 13 A round-based roleplaying game https://github.com/tgstation/tgstation https://www.tgstation13.org/ Repository: Website: Apple Swift Apple’s programming language https://github.com/apple/swift https://swift.org/ Repository: Website: Elasticsearch A search engine https://github.com/elastic/elasticsearch https://www.elastic.co/products/elasticsearch Repository: Website: Moby An open framework to assemble specialized container systems https://github.com/moby/moby https://mobyproject.org/ Repository: Website: CockroachDB A cloud-native SQL database https://github.com/cockroachdb/cockroach https://www.cockroachlabs.com/ Repository: Website: Cydia Compatibility Checker A compatibility checker for Cydia — a package manager for iOS jailbroken devices https://github.com/jlippold/tweakCompatible https://jlippold.github.io/tweakCompatible/ Repository: Website: Servo A web browser engine https://github.com/servo/servo https://servo.org/ Repository: Website: Google Flutter Google’s mobile app SDK to create interfaces for iOS and Android https://github.com/flutter/flutter https://flutter.dev/ Repository: Website: macOS Homebrew Package Manager Default formulae for the missing package manager for macOS https://github.com/homebrew/homebrew-core https://brew.sh/ Repository: Website: Home Assistant Home automation software https://github.com/home-assistant/home-assistant https://www.home-assistant.io/ Repository: Website: Microsoft .NET CoreCLR Runtime for .NET Core https://github.com/dotnet/coreclr https://docs.microsoft.com/en-us/dotnet/core/ Repository: Website: CocoaPods Specifications Specifications for CocoaPods, a Cocoa dependency manager https://github.com/CocoaPods/Specs https://cocoapods.org/ An analytics and search dashboard for ElasticsearchRepository: Website: Elastic Kibana https://github.com/elastic/kibana https://www.elastic.co/products/kibana Repository: Website: Julia Language A technical computing language https://github.com/JuliaLang/julia https://julialang.org/ Repository: Website: Microsoft TypeScript A superset of JavaScript that compiles to plain JavaScript https://github.com/Microsoft/TypeScript https://www.typescriptlang.org/ Repository: Website: Joomla A content management system https://github.com/joomla/joomla-cms https://www.joomla.org/ Repository: Website: DefinitelyTyped A repository for TypeScript type definitions https://github.com/DefinitelyTyped/DefinitelyTyped http://definitelytyped.org/ Repository: Website: Homebrew Cask A CLI workflow for the administration of macOS applications distributed as binaries https://github.com/Homebrew/homebrew-cask https://brew.sh/ Repository: Website: Ceph A distributed object, block, and file storage platform https://github.com/ceph/ceph https://ceph.com/ Repository: Website: Go Programming language https://github.com/golang/go https://golang.org/ AMP HTML Builder Repository: Website: A way to build pages for Google AMP https://github.com/ampproject/amphtml https://amp.dev/ Repository: Website: Open edX An online education platform https://github.com/edx/edx-platform https://open.edx.org/ Repository: Website: Pandas A data analysis and manipulation library for Python https://github.com/pandas-dev/pandas https://pandas.pydata.org/ Repository: Website: Istio A platform to manage microservices https://github.com/istio/istio https://istio.io/ Repository: Website: ManageIQ A containers, virtual machines, networks, and storage management platform https://github.com/ManageIQ/manageiq http://manageiq.org/ Repository: Website: Godot Engine A multi-platform 2D and 3D game engine https://github.com/godotengine/godot https://godotengine.org/ Repository: Website: Gentoo Repository Mirror A Gentoo ebuild repository mirror https://github.com/gentoo/gentoo https://www.gentoo.org/ Repository: Website: Odoo A suite of web based open source business apps https://github.com/odoo/odoo https://www.odoo.com/ Repository: Website: Azure Documentation Documentation of Microsoft Azure https://github.com/MicrosoftDocs/azure-docs https://docs.microsoft.com/azure Repository: Website: Magento An eCommerce platform https://github.com/magento/magento2 https://magento.com/ Repository: Website: Saltstack Software to automate the management and configuration of any infrastructure or application at scale https://github.com/saltstack/salt https://www.saltstack.com/ Repository: Website: AdGuard Filters Ad blocking filters for AdGuard https://github.com/AdguardTeam/AdguardFilters https://adguard.com/en/welcome.html Repository: Website: Symfony A PHP framework https://github.com/symfony/symfony https://symfony.com/ Repository: Website: CMS Software for the Large Hadron Collider Particle detector software components for CERN’s Large Hadron Collider https://github.com/cms-sw/cmssw http://cms-sw.github.io/ Repository: Website: Red Hat OpenShift OpenShift installation and configuration management https://github.com/openshift/openshift-ansible https://www.openshift.com/ Repository: Website: ownCloud Personal cloud software https://github.com/owncloud/core https://owncloud.org/ Repository: Website: gRPC A remote procedure call (RPC) framework https://github.com/grpc/grpc https://grpc.io/ Repository: Website: Liferay An enterprise web platform https://github.com/brianchandotcom/liferay-portal https://www.liferay.com/ Repository: Website: CommCare HQ A mobile data collection platform https://github.com/dimagi/commcare-hq https://www.commcarehq.org/ Repository: Website: WordPress Gutenberg An editor plugin for WordPress https://github.com/WordPress/gutenberg https://wordpress.org/gutenberg/ Repository: Website: PyTorch A Python package for Tensor computation and deep neural networks https://github.com/pytorch/pytorch https://pytorch.org/ Repository: Website: Kubernetes Test Infrastructure A test-infra repository for Kubernetes https://github.com/kubernetes/test-infra https://kubernetes.io/ Repository: Website: Keybase Keybase client repository https://github.com/keybase/client https://keybase.io/ Repository: Website: Facebook React A JavaScript library for building user interfaces https://github.com/facebook/react https://reactjs.org/ Repository: Website: Code.org Code learning resource https://github.com/code-dot-org/code-dot-org https://code.org/ Repository: Website: Bitcoin Core Bitcoin client software https://github.com/bitcoin/bitcoin https://bitcoincore.org/ Repository: Website: Arm Mbed OS A platform operating system for the Internet of Things https://github.com/ARMmbed/mbed-os https://www.mbed.com Repository: Website: scikit-learn A Python module for machine learning https://github.com/scikit-learn/scikit-learn https://scikit-learn.org Repository: Website: Nextcloud A self-hosted productivity platform https://github.com/nextcloud/server https://nextcloud.com/ Repository: Website: Helm Charts A curated list of applications for Kubernetes https://github.com/helm/charts https://kubernetes.io/ Repository: Website: Terraform An infrastructure management tool https://github.com/hashicorp/terraform https://www.terraform.io/ Repository: Website: Ant Design A UI design language https://github.com/ant-design/ant-design https://ant.design/ Repository: Website: Phalcon Framework Documentation Documentation for Phalcon, a PHP framework https://github.com/phalcon/docs https://docs.phalconphp.com Repository: Website: Documentation for CMS Software for the Large Hadron Collider Documentation for CMS Software for CERN’s Large Hadron Collider https://github.com/cms-sw/cms-sw.github.io http://cms-sw.github.io/ Repository: Website: Apache Kafka Mirror A mirror for Apache Kafka, a distributed streaming platform https://github.com/apache/kafka https://kafka.apache.org/ Repository: Website: Electron A framework to write cross-platform desktop applications using JavaScript, HTML and CSS https://github.com/electron/electron https://electronjs.org/ Repository: Website: Zephyr Project A real-time operating system https://github.com/zephyrproject-rtos/zephyr https://www.zephyrproject.org/ Repository: Website: The web-platform-tests Project A cross-browser testsuite for the Web-platform stack https://github.com/web-platform-tests/wpt https://www.w3.org/ Repository: Website: Marlin Firmware Optimized firmware for RepRap 3D printers based on the Arduino platform https://github.com/MarlinFirmware/Marlin http://marlinfw.org/ Repository: Website: Apache MXNet A library for deep learning https://github.com/apache/incubator-mxnet https://mxnet.apache.org/ Repository: Website: Apache Beam A unified programming model https://github.com/apache/beam https://beam.apache.org/ Repository: Website: Fastlane A build and release automaton for iOS and Android apps https://github.com/fastlane/fastlane https://fastlane.tools/ Repository: Website: Kubernetes Website and Documentation A repository for the Kubernetes website and documentation https://github.com/kubernetes/website https://kubernetes.io Repository: Website: Ruby on Rails A web-application framework https://github.com/rails/rails https://rubyonrails.org/ Repository: Website: Zulip Team chat software https://github.com/zulip/zulip https://zulipchat.com/ Repository: Website: Laravel A web application framework https://github.com/laravel/framework https://laravel.com/ Repository: Website: Baidu PaddlePaddle Baidu’s deep learning framework https://github.com/PaddlePaddle/Paddle http://www.paddlepaddle.org/ Repository: Website: Gatsby A web application framework https://github.com/gatsbyjs/gatsby https://www.gatsbyjs.org/ Repository: Website: Rust Crate Registry Rust’s community package registry https://github.com/rust-lang/crates.io-index https://crates.io/ Repository: Website: Nintendo 3DS Custom Firmware A complete guide to 3DS custom firmware https://github.com/hacks-guide/Guide_3DS https://3ds.hacks.guide/ Repository: Website: TiDB A NewSQL database https://github.com/pingcap/tidb https://pingcap.com Repository: Website: Angular CLI CLI tool for Angular, a Google web application framework https://github.com/angular/angular-cli https://cli.angular.io/ Repository: Website: MAPS.ME Offline OpenStreetMap maps for iOS and Android https://github.com/mapsme/omim https://maps.me/ Repository: Website: Eclipse Che A cloud IDE for Eclipse https://github.com/eclipse/che http://www.eclipse.org/che/ Repository: Website: Brave Browser A browser with native BAT cryptocurrency https://github.com/brave/browser-laptop https://www.brave.com/ Repository: Website: Patchwork A repository to learn Git https://github.com/jlord/patchwork http://jlord.us/patchwork/ Repository: Website: Angular Material Component infrastructure and Material Design components for Angular, a Google web application framework https://github.com/angular/components https://material.angular.io/ Repository: Website: Python Programming language https://github.com/python/cpython https://www.python.org/ Repository: Website: Space Station 13 A round-based roleplaying game https://github.com/vgstation-coders/vgstation13 http://ss13.moe/ Repository: Website: Cataclysm: Dark Days Ahead A turn-based survival game https://github.com/CleverRaven/Cataclysm-DDA http://cataclysmdda.org/ Repository: Website: Material-UI React components that implement Google’s Material Design https://github.com/mui-org/material-ui https://material-ui.com/ Repository: Website: Ionic A Progressive Web Apps development framework https://github.com/ionic-team/ionic https://ionicframework.com/ Repository: Website: Oppia A tool for collaboratively building interactive lessons https://github.com/oppia/oppia https://www.oppia.org Repository: Website: Alluxio A virtual distributed storage system https://github.com/Alluxio/alluxio https://www.alluxio.io/ Repository: Website: None XX Net A Chinese web proxy and anti-censorship tool https://github.com/XX-net/XX-Net Repository: Website: Microsoft .NET CLI A CLI tool for .NET https://github.com/dotnet/cli https://docs.microsoft.com/en-us/dotnet/core/tools/ [1] The explanation of the calculation of the simplified version is at the . U°OS Network GitHub repository

GitHub’s Top 100 Most Valuable Repositories Out of 96 Million

It’s been an exciting, but complex year in the data world. Just as , the data tech ecosystem has continued to “fire on all cylinders”. If nothing else, data is probably even more front and center in 2018, in both business and personal conversations. Some of the reasons, however, have changed. last year On the one hand, data technologies (Big Data, data science, machine learning, AI) continue their march forward, becoming ever more efficient, and also more widely adopted in businesses around the world. It is no accident that one of the key themes in the corporate world in 2018 so far has been “ ”. The term may feel quaint to some (“isn’t that what’s been happening for the last 25 years?”), but it reflects that many of the more traditional industries and companies are now fully engaged into their journey to become truly data-driven. digital transformation On the other hand, a much broader cross-section of the public has become aware of the pitfalls of data. Whether it is through the very public debate over the , the Cambridge Analytica scandal, the massive Equifax data breach, GDPR-related privacy discussions or reports of growing , the data world has started revealing some risks of AI government surveillance in China darker, scarier undertones. Both are the flipside of the same phenomenon, which has been brewing for many years but is now in full display: just about everything (whether personal or professional) is rapidly getting digitized, and data technologies are becoming more adept than ever at processing and analyzing this massive data exhaust, increasingly in real time. From this can result . The debate on how to combine this great power with a necessary sense of responsibility has become essential. both magic and abuse Let’s highlight some of the key trends and events of 2018. Infrastructure & Analytics From an industry standpoint, the data ecosystem remains as exciting and vibrant as ever, with a rich tapestry of innovative startups, mature “scale-ups”, and many aggressive public technology vendors. Most importantly, many customers large and small are those technologies , and reaping undeniable value from their efforts. deploying in production at scale As the cycle of replacing older IT technologies with more modern data products continues, it seems that the Big Data market (infrastructure, analytics) is cycling through the early majority of buyers and transitioning into the late majority of the traditional adoption curve. In addition, the data world continues its inexorable evolution towards the cloud. It is actually staggering to see how fast the large public cloud providers (AWS, Azure, Google Cloud Platform, IBM) are growing, considering they already each generate billions of dollars of revenues every quarter. The trend raises ongoing concerns around , and this may open up opportunities for startups offering solutions. However, to date even companies adopting multi-cloud strategies tend to still rely on one vendor as their primary provider. vendor lock-in multi-cloud As they keep growing, large cloud providers increasingly compete with each other by offering a wide array of Big Data, data engineering and machine learning tools through their platforms (e.g., Amazon Neptune, Google AutoML, etc.) — and often with aggressive pricing, to attract more developers, as their true business model is data storage. As the scope and sophistication of such tools keep growing, this has a big impact on the data technology landscape, making it arguably , at least for broad, horizontal opportunities. A bit more every year, the list of product announcements at big annual cloud vendor conferences (see AWS , for example) sends shockwaves in the startup industry, as they put cloud vendors in direct competition with dozens of VC-backed startups in one fell swoop. It will be interesting to see how public markets react to the upcoming Elastic IPO, an open-source software company that saw Amazon launch a direct competitor, Elasticsearch, three years ago. harder for startups to compete re:Invent Plenty of opportunities for startups remain, however, as long as they are sufficiently differentiated. Many in the space are scaling fast, and there are a number of particularly interesting, fast-growing segments in the infrastructure and analytics part of the ecosystem, including /real-time, , and . The explosion of interest in AI has also led to great opportunities (and a lot of funding) in , , tools, and platforms enabling the streaming data governance data fabrics/virtualization AI chips GPU databases AI devops deployment of data science and machine learning in the enterprise. Machine Learning & AI It’s certainly been a , with anything from the prowess of AlphaZero to the staggering pace of release of new advances — new forms of Generative Adversarial Networks, Vicarious’ new Recursive Cortical Networks, Geoff Hinton’s new Capsule Networks. AI conferences like NIPS have grown to attract 8,000 people and thousands of academic papers are being submitted every day. wild year in the world of AI research At the same time, the pursuit of , perhaps thankfully so. Much of the current wave of excitement (and fear) about AI results from the impressive performance of deep learning since 2012 but, in the AI research community, there’s a as some question the foundations of deep learning (backpropagation) and others look to move past what they consider “brute force” approaches (lots of data, lots of computing power), perhaps in favor of more neuroscience-based approaches. AGI remains elusive growing sense of “what now?” Far from fearing robot world domination, many in the AI research community are concerned that continued over-hyping of the field may eventually disappoint and lead to another AI nuclear winter. Outside of AI research, however, we are just at the beginning of a wave of deployment and application of deep learning in the real world across a variety of problems involving speech recognition, image classification, object recognition and language, in different industries. If the infrastructure and analytics part of the ecosystem is getting to the late majority, we’re still very much in early adopter territory for enterprise and vertical AI applications. The that started a year or two ago has mostly continued unabated, even though the AI startup market is (arguably) showing signs of finally cooling down. Expectations, round sizes and valuations remain high, but we are certainly past the phase where big Internet companies would snap up very early AI startups at high prices just for the talent. The air is also clearing up a bit and revealing “real” AI startups, versus a number of other companies that were leveraging the hype. Some of the AI startups that were founded in the 2014–2016 time frame are starting to hit early scale, and many are offering increasingly interesting products including health, finance, “industry 4.0” and back office automation. Deep learning will continue bringing a lot of value in real world applications for years to come, and vertical-focused AI startups have many great opportunities ahead of them. Cambrian explosion of deep-learning based startups across industries and verticals This continued explosion is very much a global phenomenon, with Canada, France, Germany, the U.K. and Israel being particularly active. However, , with reports of government-led pooling of data at mind-boggling scale (across Internet companies and municipalities), rapid advances in areas such as facial recognition and AI chips, and gigantic rounds of financing for its startups: according to CB Insights, China accounted for only 9% of global AI deal share but nearly 48% of global AI funding in 2017, up from 11% in 2016 (see some examples below). China seems to be playing at a completely different level in AI In the same vein, (and ownership and security) are emerging as a major concern around the world. In the early days of the Internet, data privacy was about protecting what we did online, a comparatively small portion of our activities. Correspondingly, only a small (albeit vocal and passionate) minority of people truly cared. As just about every aspect of our personal and professional lives is now connected to the Internet through an ever increasing array of connected devices, the stakes are changing. With its ability to spot anomalies in massive data sets, predict outcomes and recognize faces, issues of data privacy AI is compounding the data privacy problem. A separate but related concern is that a lot of this data is owned by large Internet companies (GAFA). Some, like Facebook, have proven to be a less than perfect steward of it. Nonetheless, this data provides them an unfair advantage in the race to produce ever more powerful AI. Against those issues, an emerging theme is to think of the , as well as a way for others, outside of GAFA, to produce great AI. Crypto economics are viewed as a way to incentivize individuals to provide their personal data and for machine learning engineers to build models by processing this data anonymously. It all remains very experimental, but some early marketplaces and networks are emerging blockchain as a possible foil against the risks of AI The 2018 Landscape Without further ado, here’s our 2018 landscape. Quite semantic note: buzz terms come and go. Fewer people speak about “Big Data”, many more about “AI”, often to described the same reality. Consequently, we have slightly rebranded our 2018 landscape: it is now called the “Big Data & AI” landscape! To see the landscape at full size, . click here (The image is high-res and should lend itself to zooming in well, including on mobile) To download the full-size image, . click here To view a full list of companies in spreadsheet format, . click here This year, my FirstMark colleague provided immense help with the landscape. Demi Obayomi We’ve detailed some of our methodology in the notes to this post. Thoughts and suggestions welcome — please use the comment section to this post. Who’s in, who’s out On the exit front, the last year (since our 2017 landscape) has seen solid, but not extraordinarily strong. A few key companies appearing on the landscape went public, in particular Cloudera, MongoDB Pivotal and Zuora. Others are preparing to go out at the time of this writing, such as Elastic. Some notable acquisitions also occurred, including in particular Mulesoft (acquired by Salesforce post-IPO, for $6.5B), Flatiron Health (acquired by Roched for $2.1B), Appnexus (acquired by AT&T for $1.6B), Syncsort and Vision Solutions (acquired for $1.2B by Centerbridge Partners), Moat (acquired by Oracle for $850M), Integral Ad Science (acquired by Vista Equity Partners for $850M), eVestment (acquired by NASDAQ for $705M) and Kensho (acquired by S&P Global for $550M). It is worth noting that, other than Mulesoft, all those companies are headquartered on the East Coast (New York, Boston and Atlanta). Many other companies were also acquired for smaller amounts: Gigya (SAP), Blue River Technology (Deere & Co), CoreOS (Red Hat), Guavus (Thales), Lattice Data (Apple), Socrata (Tyler Technologies) and PracticeFusion (AllScripts). On the investment front, this was a year of big financing rounds for some Big Data and AI startups, particularly in China, with a number of oversized investments including Bytedance ($3B in total across 2 rounds in 2017), NIO ($1.6B across two rounds in 2017), and SenseTime ($850M across two around in 2017 and 2018). Major rounds of US companies appearing on the landscape include Snowflake Computing ($263M Series A — see our recent at ), Cohesity ($250M Series D), Dataminr ($221M Series E), Affirm ($200M Series E), Rubrik ($180M Series D), Qualtrics ($180M Series C — see an older but still relevant at Data Driven NYC), Tanium ($180M private equity round), ThoughtSpot ($145M Series D) and Coveo ($100M private equity round) and C3IoT ($100M Series F). fireside chat Data Driven NYC fireside chat ____________________ NOTES: As every year, we couldn’t possibly fit all companies we wanted on the chart. While the general philosophy of the chart is to be as inclusive as possible, we ended up having to be somewhat selective. Our methodology is certainly imperfect, but in a nutshell, here are the main criteria: Everything being equal, we gave priority to companies that have reached some level of market significance. This is a reasonably easy exercise for large tech companies. For growing startups, considering the limited amounts of data available, we often used venture capital financings as a proxy for underlying market traction (again, probably imperfect). So everything else being equal, we tend to feature startups that have raised larger amounts, typically Series A and beyond. Occasionally, we made editorial decisions to include earlier stage startups when we thought they were particularly interesting. On the application front, we gave priority to companies that explicitly leverage Big Data, machine learning and AI as a key component or differentiator of their offering. As discussed in the piece, it is a tricky exercise at a time when companies are increasingly crafting their marketing around an AI message, but we did our best. This year as in previous years, we removed a number of companies. One key reason for removal is that the company was acquired, and not run by the acquirer as an independent company.. In some select cases, we left the acquired company as is in the chart when we felt that the brand would be preserved as a reasonably separate offering from that of the acquiring company. As always, it is inevitable that we inadvertently missed some great companies in the process of putting this chart together. Did we miss yours? Feel free to add thoughts and suggestions in the comments. The chart is in png format, which should preserve overall quality when zooming, including on mobile. As we get a lot of requests every year: feel free to use the chart in books, conferences, presentations, etc — two obvious asks: (i) do not alter/edit the chart and (ii) please provide clear attribution (Matt Turck, Demi Obayomi and FirstMark Capital). Disclaimer: I’m an investor through FirstMark in a number of companies mentioned on this Big Data Landscape, specifically: ActionIQ, Cockroach Labs, Dataiku, Frame.ai, Helium, HyperScience, Kinsa, Timber, Sense360 and x.ai. Other FirstMark portfolio companies mentioned on this chart include Bluecore, Engagio, HowGood, Payoff, Knewton, Insikt, Optimus Ride, and Tubular. I’m a small personal shareholder in Datadog.

Great Power, Great Responsibility: The 2018 Big Data & AI Landscape

Unsupervised Investments (I): A Guide to AI Investors

Apache Cassandra — The minimum internals you need to know

How can you tell which cloud provider is the right one for your business? And which one isn’t going to break your wallet? On the surface, AWS, Microsoft Azure, and Google Cloud are pretty similar. They all offer flexible compute, storage, and networking combined with everything engineers love about the cloud: self-service, instant provisioning, and autoscaling. But cloud providers differ in key areas that have a massive impact on your cloud bill. Most of your bill consists of d. That’s why I will focus on comparing the pricing of Instances and Virtual Machines across AWS, Azure, and Google Cloud to show you the nuanced differences between these providers. compute resources that represent even 75%-80% of a company’s cloud spen Cloud landscape overview: Top strengths of AWS, Azure, and Google Cloud Selecting one vendor over another comes down to knowing what your teams, applications, and workloads need. You need to have a full understanding of your requirements before setting out to explore the cloud landscape. Next, it’s time to examine each provider separately. The thing is, AWS, Azure, and Google Cloud come with different strengths and weaknesses. AWS is the leader for a good reason Let’s start with AWS that currently holds the largest chunk of the cloud market. Its undeniable strength is . You can choose from and pair them with a rich array of tools for databases, analytics, management, IoT, security, and enterprise applications. This list could go on almost forever. the sheer breadth and depth of its services almost 400 instance types AWS is the winner when it comes to developer functionality, thanks to the breadth of its services and ability to translate this scale into cost benefits. Still, this . One of the common pricing challenges companies face is understating AWS metrics and estimating the impact of architectural decisions on costs. complexity comes at a cost Azure is the go-to choice for enterprises. Microsoft Azure is quite a . That’s usually because they have a long-standing relationship with the provider and meeting all their enterprise computing needs in one place makes sense. popular choice with C-level executives The combination of Azure, Office 365, and Teams makes a safe option for enterprises looking to benefit from enterprise software all the way down to cloud computing resources with a single handshake. Google Cloud taps into the power of open source. All three major cloud vendors are strong in machine learning capabilities. But Google stands out among them thanks to its endless well of internal research and expertise – basically, the magic powering the search engine giant throughout the years. What makes Google different is its role in . I’m talking especially about containers and Google’s central role in building Kubernetes for orchestration and the Istio service mesh, today practically industry-standard technologies. developing various open source technologies Google’s culture of innovation lends itself really to startups and companies that prioritize these approaches and technologies. Naturally, companies often end up using different cloud providers for different parts of their operations or use cases. This lets them bring the best of all the (cloud) worlds together under one roof. multi cloud approach Billing in AWS vs. Azure vs. Google Cloud Per second billing is now something all the three cloud vendors offer. AWS introduced it in 2017 for EC2 Linux-based instances and EBS volumes at first – but today, it applies to many other services. Note that in AWS, per-second billing works with the minimum 60-second limit. Azure allows per-second charges as well, but this billing model isn’t available for all instances – mostly container-based instances. Google Cloud followed AWS in the introduction of per-second billing and now offers it for more than just instances based on Linux. In Google, this form of billing applies to all VM-based instances. Let’s take a look at the biggest cost driver: compute Compute is what makes your cloud bill so high. This isn’t to say that other services don’t contribute to it at all. Storage can get quite expensive, and moving data around might result in high egress costs. But it’s compute that comes with the biggest price tag – and greatest opportunity for cost optimization. Here’s a case study that shows the . incredible impact optimizing compute costs can have on a company’s bottom line How Netflix slashed its cloud costs by picking different machines To deliver its video streaming services across 190 countries, uses Amazon EC2 instances and Kinesis data streams. Recently, Netflix turned to the Graviton2-based EC2 instances. Switching to these brand-new machines helped Netflix improve performance by 50% and cut costs. Netflix Before, the company used m4.16xlarge instances that cost $3.2/hour. Now, Netflix uses m6g.16xlarge that costs $2.464/hour, and m6g.xlarge, priced at only $0.0154/hour. You can only imagine the cost savings Netflix enjoyed on its cloud billing. This example clearly shows how a minor difference in instance pricing can save you thousands of dollars. Comparing cloud pricing – our example setup To understand the pricing differences better, we’re going to compare instances and VMs within one region and across the same operating system. This is our example setup: Region: US West – Los Angeles (US West in Azure, US West 2 in Google Cloud) Operating System: Linux vCPUs: 4 Types of instances/VMs we will analyze: General-purpose Compute-optimized Memory-optimized For our comparison, we chose instances with similar RAM and 4 vCPUs. The obvious exception is Google Cloud’s memory-optimized instance that starts at 40 vCPUs. Here are the instances/VMs we selected for our cloud pricing comparison: AWS vs. Azure vs. Google Cloud: Comparing On-Demand pricing Here is the hourly On-Demand pricing of each of those instance-type scenarios across AWS, Azure, and Google Cloud. Cloud pricing based on On-Demand rates Note: In the above table, red marks the highest price and green the lowest price for a given instance type. Takeaways: As you can see, AWS and Azure offer quite a similar pricing for all these three instance types. For compute-optimized instances, Google Cloud has the highest pricing due to its scalable processors and all-core performance. The cost of Google Cloud for memory-optimized instances is way higher because it doesn’t run on 4 vCPUs, but 40vCPUs.A note about chips and processors Providers roll out instances on different computers, and the chips in them have different performance characteristics. So, you might end up with an instance type that has strong performance your teams don’t actually need, and you’ll have no idea about this. Reasoning about this on your own is challenging, and the best way to see what you’re really paying for is via benchmarking. The idea here is to drop the same workload on each machine and check its performance characteristics. You might discover something interesting, just like we did: This chart shows CPU operation in AWS (t2.2xlarge with 8 virtual cores) at different times after several idle CPU periods. Would you expect such unpredictable CPU behavior within a single cloud provider? Source: CAST AI The used the same method to evaluate 54 machines across AWS, Azure, and Google Cloud. They ran over 1000 microbenchmark tests to evaluate metrics like CPU, network, storage, and TPC-C performance. 2021 Cloud Report from CockroachLabs t achieved the best single-core CPU performance and offered the greatest throughput at every level. Google’s machines with general-purpose disks achieved the highest level of raw throughput but turned out to be the least cost-efficient option. One of their overall conclusions was that Google beats AWS and Azure on performance. I AWS vs. Azure vs. Google Cloud: Comparing discounted pricing with 1-year upfront commitment All three providers offer discounts if you commit to using them for at least 1 year. The pricing model is called Reserved Instances/Savings Plans in AWS, Reserved Savings in Azure, and Commitment Price in Google Cloud. These discounts incentivize businesses to commit to a specific level of usage for a fixed period in exchange for a discounted rate on some (not all) instances and VMs. To compare the discounted pricing among AWS, Azure, and Google Cloud, we examined a one-year commitment period without upfront cost – but this time, in another region: US East – Northern Virginia. Cloud pricing with 1-year commitment Takeaways: The general-purpose instances with a 1-year commitment are almost similar in AWS and Azure. AWS wins in the compute-optimized category, even if Azure and Google apply some pretty big discounts. Azure offers the cheapest alternative among memory-optimized instances. Not sure how Reserved Instances work and whether they really translate into discounts? Read this: Do AWS Reserved Instances and Savings Plans really reduce costs? How CPU bursting can help drive your costs down AWS, Azure, and Google Cloud all offer burstable performance instances. These instances provide you a baseline level of CPU performance, with the opportunity to burst to a higher level whenever your workload requires that. Burstable performance instances are . a good pick for low-latency interactive applications, microservices, small and medium databases, or product prototypes But to make the most of them, you need to keep a close eye on the credits you collect that enable bursting. For example, if you restart a T2 instance in AWS, you’re going to lose all the credits. Restarting a T3 or T4 instance means keeping credits for another 7 days and then losing them. Our research into AWS burstable instances showed that if you load your instance for 4 hours or more per day on average, you’re actually better off with a non-burstable one. But if you run an e-commerce business that gets a large stream of visitors once in a while, it’s a good match. AWS vs. Azure vs. Google Cloud: Comparing Spot Instances/Preemptible VMs Here’s another way to reduce your cloud bill – take advantage of capacity that’s currently not being used by any businesses. Cloud providers sell this excess capacity at incredibly high discounts – while AWS Spot Instances offer up to 90% off the On-Demand rates, Preemptible VMs in Google can be even 80% cheaper than regular VMs. Here’s a quick overview of the potential savings you can get for these instances in the US East – Northern Virginia region: Cloud pricing with Spot Instances/Preemptible VMs As you can see, all providers offer substantial discounts on these instances. But to take advantage of these savings, . How? Here’s a step by step guide: you need to make sure that your applications can handle interruptions Spot Instances: How to reduce AWS, Azure, and GCP costs by 90% Optimizing cloud costs is a point in time exercise By now, you’ve learned the basics of calculating compute charges. But Spot instance prices don’t always stay the same. In fact, they might change from one second to the next. Take a look at this example: At CAST AI, we were running an application on a mix of AWS On-Demand instances and Spot Instances. Then we used our own platform to analyze our setup. The platform looked for the most cost-effective Spot Instance alternatives for a machine with 8 CPUs and 16 GB. CAST AI suggested that we run our workload on an instance called INF1. But wait, It’s a supercomputer that usually costs a lot of money. why would we need a powerful GPU instance meant for machine learning applications? So, why did CAST AI pick it? We checked the pricing. As it turns out, at that time, INF1 just happened to be cheaper than the usual general purpose instances we used. We would have never guessed to look for Spot Instances in this category. As a result, we’d miss out on this incredible gem. That’s exactly the reason why you need automation to optimize cloud costs Even if you have DevOps or FinOps specialists managing your cloud expenses, . It’s high time you took control of your cloud bill with the help of automation and AI. you’re probably spending twice as much as you should CAST AI is a great place to start your path towards automated optimization if your teams work with Kubernetes. The solution automatically creates and implements cost-cutting tactics for guaranteed savings without any manual and repetitive work for your engineers. Visit CAST AI to learn how automation can reduce your cloud bill to the minimum regardless of the cloud provider you are working with.

COCKROACH LABS

cockroachlabs.com

501-1K employees

Since 2015

COMPANY RANKING

COCKROACH LABS

EVERGREEN INDEX #351

Representing this company? Click here to claim and customize this page!

Cockroach Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

COCKROACH LABS

cockroachlabs.com

501-1K employees

Since 2015

COMPANY RANKING

COCKROACH LABS

EVERGREEN INDEX #351

RELATED COMPANIESRANK

STORIES ABOUT COCKROACH LABS

Representing this company? Click here to claim and customize this page!

Cockroach Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!