Airspeed is a suite of Slack and mobile apps that provide fun and simple ways to connect with your teammates. Our family of apps improve employee engagement, retention, and productivity by creating a strong company culture. Airspeed makes it easy to welcome new hires, track employee milestones, send kudos, and learn who (and where) your coworkers are. Learn more about Airspeed’s app family below, or get connected with free resources at airspeed.com/blog.

Airspeed

Optimizing Data Center Efficiency: A Deep Dive into Freecooling Techniques

You can jump to any section of the National Aeronautics Science & Technology Priorities III. Promoting Connectivity and Speed here. Air travel quickly moves people and cargo around the nation and the world, and connects people with each other at unprecedented speeds. It plays a critical, unique role in the nation’s transportation system. The U.S. Government will continue exploring new technologies that will enhance global connectivity at greater speed. As the U.S. Government prioritizes the integration of new aerospace technologies into the NAS, these technologies have the potential to provide improved connectivity and speed to communities across the country. AAM vehicles operating as would-be air taxis have the potential to provide people with a new and exciting way to move around their communities. Drones flying at BVLOS can be used to transport high-value goods such as medical supplies point-to-point with unprecedented speed and efficiency. The U.S. Government will create pathways to ensure these technologies can be safely, equitably, and sustainably incorporated into the NAS, thereby promoting connectivity and speed. To remain a global leader in aeronautics, the U.S. Government will advance connectivity and speed, while maintaining the unprecedented safety of air travel, and ensuring the environmental sustainability of the aviation sector. Beyond improvements to operations for traditional subsonic aircraft, increasing airspeed provides another approach to enhancing connectivity. Supersonic flight is defined as speeds faster than the speed of sound (Mach 1). Supersonic flight has seen emerging commercial interest for cargo and passenger transportation. Hypersonic flight is defined by flight in the earth’s atmosphere at speeds greater than Mach 5 for extended periods of time. It also draws upon and drives development of technologies in both air and space domains. Hypersonic flight systems will shorten national security crisis response times, while providing unique capabilities to the aeronautics enterprise. The U.S. Government will develop these technologies and systems via national collaborations across the Federal government, industry, academia, National Laboratories and Federally Funded Research and Development Centers, and University Affiliated Research Centers, as well as collaboration with allies and international partners. For supersonic flight, the U.S. Government will enable the generation of key data to support the development of enroute certification standards based on acceptable noise levels. It will take a systematic approach involving aircraft development, acoustic validation, and community response testing. R&D will be required to solve the other challenges of sustainable supersonic transportation, including reducing landing and take-off noise and high-altitude emissions, and significantly increased airframe and engine efficiency. Hypersonic flight systems could, in the future, support national needs for time-critical transportation of cargo, people, and payloads. R&D related to hypersonic applications, ranging from aerodynamics to high-speed propulsion, advanced materials, and resilient guidance, navigation, and control, is critical for U.S. national security. The U.S. Government will continue to empower private-sector innovation to realize a vision for aircraft-enabled, reusable space vehicles through hypersonic flight. This was originally published in March 2023 on whitehouse.gov. This content has been broken-up into bite sized sections, each with unique headlines and AI-generated lead images, in line with the provisions of Creative Commons 3.0.

How the U.S. Plans to Make Air Travel 5 Times Faster Than the Speed of Sound

মার্কিন যুক্তরাষ্ট্র কীভাবে শব্দের গতির চেয়ে 5 গুণ দ্রুত বিমান ভ্রমণ করার পরিকল্পনা করেছে

Wie die USA den Flugverkehr fünfmal schneller als mit Schallgeschwindigkeit machen wollen

Cómo Estados Unidos planea hacer que los viajes aéreos sean 5 veces más rápidos que la velocidad del sonido

Comment les États-Unis envisagent de rendre le transport aérien 5 fois plus rapide que la vitesse du son

कैसे अमेरिका ने हवाई यात्रा को ध्वनि की गति से 5 गुना तेज़ बनाने की योजना बनाई है

米国はどのようにして飛行機の移動を音速の5倍にする計画なのか

미국이 항공 여행을 음속보다 5배 빠르게 만들 계획을 세우는 방법

Como os EUA planejam tornar as viagens aéreas cinco vezes mais rápidas que a velocidade do som

Как США планируют сделать авиаперелеты в 5 раз быстрее скорости звука

ABD, Hava Yolculuğunu Ses Hızından 5 Kat Daha Hızlı Hale Getirmeyi Nasıl Planlıyor?

Hoa Kỳ có kế hoạch thực hiện việc di chuyển bằng đường hàng không nhanh gấp 5 lần tốc độ âm thanh như thế nào

美国计划如何使航空旅行速度比音速快 5 倍

A Comprehensive Guide to UAV Digital Forensics With DJI Drones

Astounding Stories of Super-Science August 1931: VOL. VII, NO. 2 - The Port of Missing Planes

Not long ago, in 2009, a behavior in the system (i.e a mode confusion) was part of the events that led to the loss of . The pilots reacted to a loss of altitude by pulling on the stick, which would have been an appropriate reaction with the autopilot fully enabled, which would then have put the aircraft in a climbing configuration. However, the airplane’s systems had entered a mode of lesser automation (“direct law” in Airbus terms) due to a blocked airspeed sensor, allowing the pilots to put the plane in a nose-high stall configuration, from which they did not recover. Air France Flight 447 We have come a long way in systems to build reliable software and techniques, however, systems still fail all the time. What makes some systems more prone to failure than others? Often times, we attribute failure to complexity. That’s a fair answer but the experience and evolution of software says there’s more to it. Running large ( ), complex systems for more than a decade, one pattern I repeatedly see is failure modes or modes in general. And when not done right, modes can make a system intrinsically unstable. Every system has failure modes but the most common and nasty ones are introduced by bimodal behaviors. in some cases, literally the largest In the book “ ”, Steven Pinker talks about how today we may be living in the most peaceful time in our species’ history, despite what the news tells us. ( ) The Better Angels of Our Nature highly recommended if you haven’t read I know, it’s a cheesy, weird parallel to draw here (with system failures) but today we (systems operators) may be living in the most peaceful (i.e. less oncall pain) time in our species’ (systems) history. That’s because years of academic research has gone into this very topic. A mode is a distinct setting within a machine interface, in which the same user input will produce perceived results different from those that it would in other circumstances. e.g. for , there’s one mode for inserting text, and a separate mode for entering commands ( ). These are fairly benign modes that you deal with everyday and are mere nuances for beginners. vi sorry, Emacs users but I’m sure you get the point However, there are modes that can cause actual production downtime. You may recognize some: If a pod normally calls a cluster local service, but can fallback to an external service under a certain condition, that’s a bimodal behavior Kubernetes If you call your database every 5 minutes ( ) but in case of failure, you retry every 100 milliseconds, that’s a bimodal behavior of the system. happy path Take these failure modes seriously. Bimodal/fallback behaviors are harder to test. They exercise your system in ways where “fallback path” or “secondary mode” will become less tested over time. Your primary mode will become resilient, but the day the fallback behavior kicks in ( ), your system availability will be at risk and you will have nasty outages. and it has latent issues Here are some alternatives to avoid bimodal behaviors in the examples I shared above: If a Kubernetes pod calls a cluster local service and the service is not available, instead of falling back to an external service, failover to a replica of your cluster local service improve the reliability of your cluster local service. or If you call your database every 5 minutes for the happy path, keep the same frequency when it fails. With 100ms, your database might receive a thundering herd of 3000x calls, potentially triggering another set of cascading failures ( ) that’s a topic I will cover another day Avoid bimodal behaviors when building systems. Know your failure modes. Fail cleanly and predictably. It’s a simple concept that will bring more “peace” in running systems.

Failure Modes: Why You Need To Know Them

When lockdown restrictions came into effect, lots of people turned to gaming as an escape, or just simply a way to pass the time. I, and many of my fellow Hacker Noon staff, belong to that camp. Which games did we play the most? Let's find out. This Slack discussion by Jeffrey Harris, Jack Boreham, Jose Hernandez, Nicolas Ng, and I occurred in 's official #gaming channel, and has been edited for readability. slogging What video games got you through the pandemic? Limarc Ambalina Apr 4, 2021, 3:40 AM Because of lockdown restrictions, many people had to find other ways to entertain themselves at home. Being an introvert, I wasn't that bothered by it to be honest. But I'd say there were a few games that helped me spend the time in glorious fashion. These games were: Limarc Ambalina Apr 4, 2021, 3:46 AM 1. Call of Duty Warzone 2. Fortnite (before it died) 3. FitXR (for exercise) 4. Thrill of the fight (for exercise) 5. SkyrimVR 6. Among US 7. Pixel Ripped 1995 (super amazing hidden gem of a VR title) Ones that definitely helped JeffreyHarris-HackerNoon Apr 4, 2021, 8:02 PM JeffreyHarris-HackerNoon Apr 4, 2021, 8:02 PM The Witcher 3 JeffreyHarris-HackerNoon Apr 4, 2021, 8:02 PM 2. Doom JeffreyHarris-HackerNoon Apr 4, 2021, 8:03 PM 3. Doom Eternal; 4. Overwatch; 5. Mega Man X 1-3; Super Mario World; 6. Kirby's Planet Robobot; 7. Ghost of Tsushima So SkyrimVR helped me get through the pandemic because I was discovering a familiar world for the first time in an incredibly immersive way. It was the escape I needed to feel like I wasn't confined, despite the fact that I was playing in a 2 meter x 2 meter space within my tiny Japanese apartment. Riding a horse in SkyrimVR, and turning to shoot your bow and arrow at a dragon flying overhead is an experience I can't really put into words. In a lot of ways, video games, like any other form of storytelling, can give us hope in times that we need it. Could you tell me which one of those games helped you cope or escape JeffreyHarris-HackerNoon and what it was about it that resonated with you? Limarc Ambalina Apr 5, 2021, 7:47 PM Also Jose Hernandez Nicolas Ng Jack Boreham get in on this topic when you can! 😄 Limarc Ambalina Apr 5, 2021, 7:48 PM I couldn't agree more with your number 1 pick Limarc Ambalina. Call of Duty warzone helped me get through the lockdown and is still helping me cope with it (the UK is in its 3rd lockdown). There is nothing like teaming up with mates that you can't see in real life to win a battle royale and click some heads. It allows me to socialise with my pals, and it has been a godsend during the pandemic. Jack Boreham Apr 5, 2021, 9:35 PM I think a lot of people are worried when gamers like us say this FPS shooting game that "glorifies violence" helped us cope with a pandemic, but it's not about that. It's about whatever game gets you and the friends together, you know? It could have been Among Us, it could have been Fall Guys. It's more about finding a way to socialize online with friends while you feel like you're isolated. Limarc Ambalina Apr 5, 2021, 9:37 PM Exactly, warzone came at the perfect time. My friends and I all had little to do and were isolated. Warzone brought us together. It's the same as monster hunter. That game is helping me now to connect with friends and to get through this wave of lockdown. It's not all about the FPS nature of the game, it's the fun you can have with friends, and that can be with any game. Jack Boreham Apr 5, 2021, 9:47 PM A game that I played a lot in the pandemic was . It's an action-RPG that takes place in outer space. Jose Hernandez Apr 5, 2021, 10:06 PM The Outer Worlds So Obsidian also developed Fallout: New Vegas and I loved that game a lot. Is the combat similar to Fallout Jose Hernandez? Also, what about that game was especially good that helped you either destress or find some escape during the pandemic? Limarc Ambalina Apr 5, 2021, 11:00 PM So my own list was: Nicolas Ng Apr 6, 2021, 12:43 PM 1. Minecraft (With the SevTech Ages mod installed) 2. FFXIV: Shadowbringers 3. Ace Combat 7 4. Project Wingman 5. Microsoft Flight Simulator 6. Ring Fit Adventure 7. Not sure if this counts, but Dungeons and Dragons played over Roll20 And I agree with both of you, Jack and Limarc, it doesn't really matter what we do when we game, so long as we can do it with our friends. In my case, it was making nuclear reactors and going to space. I barely felt the pandemic simply because more of my friends were on more often. When they weren't online, there's always the next world to explore with some interesting characters. Nicolas Ng Apr 6, 2021, 12:46 PM Limarc Ambalina I've never played Fallout, so I can't compare the two. But what helped me escape the pandemic was the story: traveling through outer space trying to take down a corrupt government captivated me. I hated putting the controller down Jose Hernandez Apr 6, 2021, 7:46 PM Interesting Jose and on that note Nicolas Ng how does Ace Combat or Microsoft Flight Sim help you escape? Or is it a more relaxation thing. Flight Sim like just seems to be so boring to me haha but maybe I'm wrong. Limarc Ambalina Apr 6, 2021, 8:43 PM Ace Combat is more of something that I did for excitement. It gets pretty intense and kept my hands busy. I just happened to be really addicted at the time. Flight Sim can be really boring but I kept it to short flights through really interesting places like the Alps or the Grand Canyon. Plus, I don't actually know how to do things normally and tend to end up in crazy situations like getting stuck in a snowstorm without an airspeed indicator while weaving between mountains. Nicolas Ng Apr 6, 2021, 9:06 PM

22 Video Games that Got Us Through the Pandemic

Have you ever thought about where you get your food from? No, I’m not talking about using Instacart on your phone or even going to the local Walmart or Safeway. Let’s take the example of a burger. What are the components of a burger? Cheese, lettuce, tomato, bacon, onion, pickles and chilles. There are also condiments such as mustard, mayonnaise, ketchup, relish, or “special sauces”, and last but not least, the bun. Where do you really get your food from? Let’s take just one ingredient: tomatoes — the number one plant in the edibles category. They are planted either late Spring or early Fall. . Keep monitoring for 80 days, and you got your ripe tomatoes! Of course, you also have to keep pests out, make sure it does not get humid and keep water levels optimal. All this just for one ingredient of one food item that you consume. They are quite greedy — they require 6–8 hours of afternoon sunlight, a 3-feet pot for each plant, pH levels only between 6.2 to 6.8 and at least 4 inches of compost voila, Farming is a There are currently over people in the world, with an expected population of over With such a monumental need of food, we need to ensure that we have enough resources, and that we are using them in the most optimal way. Fruit and vegetable farmers responding to a Feedback survey reported that they This quantity would be enough to provide 250,000 people with their recommended five portions of fruit and vegetables a day for a year. huge industry. 7.3 billion 8.5 billion by 2030 and 9.7 in 2050. wasted up to 37,000 tonnes of produce every year — around 16 per cent of their crop. What can we do? We need better irrigation management, better mapping and better control over how we grow the crops and vegetables. Enter Drones. 1. Drones for Agriculture , the use of drones in the agriculture can basically be boiled down to four segments: Crop field scanning with compact multispectral imaging sensors, GPS map creation through on-board cameras, heavy payload transportation, and livestock monitoring with thermal-imaging camera-equipped drones. According to a Dronefly press release In simple terms, you can scan your crop, map the surface, deliver items and count animals using drones. Farmers needed huge air-crafts, heavy lifting machinery, trucks, and manual labor to accomplish the above mentioned tasks. And now they are solved using a What is interesting is the sheer labor and effort it took to accomplish the above mentioned four tasks before drones came into the picture. 50 pound device that has four vertically oriented propellers. Source: Dronefly Infographics And when you have time, check out the full, beautifully designed infographics that tells you how exactly drones achieve their use cases. 2. Drones for Surveying— Story of A Switzerland Startup When I started writing this article, I thought the main focus should be agriculture. However, as I kept searching Fortunately, I remembered a friend of mine, , who works for a company that is the located in Switzerland — I realized just how versatile drones are. Adyasha Dash world’s leading VTOL drone producer Wingtra AG. I got to chat with her and learn some interesting information on how Wingtra grew, faced the challenges in creating an autonomous drone and where this could lead us. Below are some of the highlights from the conversation! What was the idea of Wingtra? When you think about drones, . Multicopters — these take-off and land vertically, have excellent precision, but cannot cover lot of area. And Fixed Wings — these use wings to generate lift, can travel long distance but hard to use for inexperienced drone users since they need to be hand launched. We wanted to bridge this gap. We wanted to leverage the usability factor of a Quadcopter with the large coverage that you obtain from Fixed Wings. there are two major types When I started, we were around 15 people 3 years back, when we had a pilot at all times to monitor the drone. Back then the drone was basically a cardboard with 2 motors. And now, we have a fully autonomous product. We wanted our main use case to be mapping and surveying for professional use such as mining, animal counting, mapping a topography, and so on. It started with us flying the device once a week to now it being flown more than 1000 times a week around the world. Is Agriculture your target market? We thought it would be, but we found greater traction in . Imagine miners who want to know how much earth has been exactly dug out and get a topographic map of the area — that’s a major focus. mining and surveying On the other hand, you have researchers. Zoology researchers who would like to survey a land or sea to count the number of animals/mammals present and researchers in Greenland who study how climate change effects glaciers are a huge group as well! What was the most challenging part of building this? The most important part of the technology is the transition phase — to be able to combine the functionality of a quadcopter and a fixed wing. This took us the longest time to achieve. The second most gripping challenge was safety, since this device is interacting with people out there. For academic purposes, it was okay when there were a few failures. But, when you start to sell commercially, it had to be 100/100 so people who’ve never seen it before can still use it safely. How does the technology work to create this magic? There are multiple sensors on the device that make it work — an air speed sensor, a magnetic compass to measure orientation, inertial measurement unit and a small micro-controller that takes in all of this input. On the user’s side, there is a tablet which has a very simple UI. All they have to do is denote the area they want to map in terms of a polygon, and press Start. The plane begins to deconstruct and figure out the most efficiency way to map, takes hundreds of pictures and analyzes this data to show to user . If the battery is fully charged, it can basically cover 240 American Football fields. ( Do you know the area of one field? ) Now that you have a fully autonomous Drone, what’s the next leap? I think we have made good progress in the aerial imagery and data collection part. However, the data we collect even in a single flight can be quite large: 1000s of images summing to tens of GB. So, an interesting challenge to solve next would be to make the data handling and transfer from the drone to cloud work seamlessly. And of course, you can never forget the safety side. Talking about the drone world itself, up until now most drones have been used within the line of sight. The challenge would be build drones that are safe enough to be operated beyond visual of sight as well. Another challenge is to incorporate the air traffic to find the best possible routes of travel. 3. Drones for Delivery There is no story complete without Amazon peeking in for a glance. From two-day delivery to two-hour delivery, Amazon’s next vision is to get this down to less than 30 minutes. Amazon was granted a new patent by the U.S. Patent and Trademark Office for a delivery drone that can respond to human gestures. There’s a happy looking drone! According to the patent, the drone’s communication system would include an array of sensors, including a depth sensor and cameras to detect visible, infrared and ultraviolet light. The drones would be able to recognize hand and body gestures, human voices and movement, such as a person walking closer to the drone or away from it (It can even detect when people are shooing it away!). But, it’s not just the e-commerce giant in the US. , the Amazon equivalent in China, has also However, they view drones not as a last-mile problem solver, but more so as an intermediate heavy-traffic-skipping vehicle. Alibaba got its hands dirty in the drone delivery business. Currently, drone delivery is only for the 22.5-square-mile Shanghai Jinshan Industrial Park, but there are plans to expand the program to other cities in coming years. **************************************************************** To conclude, the era of drones has shifted from being merely a device used to take pictures to a device that will (hopefully) solve some of the gripping problems around the world surrounding food, safety and transportation. ****************************************************************** If you found this to be useful, do Follow me for more articles. Did you know you can clap more than once? Try it out! :) I love writing about social issues, products, the technology sector and my graduate school experience in the US. Here is my personal blog . The best way to get in touch with me is via Instagram and Facebook . I share some interesting content there. To know more about my professional life, check out my LinkedIn . Happy reading!

How Drones Are Changing The World

At AISPEECH, we provide conversational AI services and natural language interaction solutions for a wide range of entities including financial institutions, the government, and IoV and IoT companies. If you believe in the idea that big data is the fuel for artificial intelligence, you will see how important a performant data processing architecture is for us. In this post, I am going to share with you what data tools and techniques we use to power our AI services. Real-Time vs. Offline: From Separation to Unity Before 2019, we used Apache Hive + Apache Kylin to build our offline data warehouse, and Apache Spark + MySQL for real-time analytic data warehouse: Roughly, we have three data sources: Business databases such as MySQL Application systems such as K8s container logs Automotive T-Box logs We write such data into Kafka via MQTT/HTTP protocol, business database Binlog, and Filebeat log collection. Then, the data will be diverted to two links: real-time and offline. : Data cached by Kafka will be computed by Spark and put into MySQL for further analysis. Real-time data link : Data cleaned by Kafka will be put into Hive. Then, we used Apache Kylin to create Cubes, but before that we needed to pre-construct a data model, which contains association tables, dimension tables, index fields, and relevant aggregation functions. Cube creation is triggered by a scheduling system on a regular basis. The created Cubes will be stored in HBase. Offline data link This architecture is a seamless integration with Hadoop technologies, and Apache Kylin delivered excellent performance in pre-computing, aggregation, exact deduplication, and high-concurrency scenarios. But as the days go by, some unpleasant issues reveal themselves: : Kylin 2.x and 3.x relies heavily on Hadoop and HBase. Too many components bring longer development links, higher instability, and more maintenance costs. Too many dependencies : This is a tedious process with all the flat table creation, column deduplication, and Cube creation. Every day, we ran 1000~2000 tasks but at least 10 of them failed so we had to spend a big chunk of time writing auto O&M scripts. Complicated Cube creation in Kylin : Dimension expansion means the prolonged creation time of Cubes when a data analysis model involves too many fields without data pruning; dictionary expansion happens when global exact deduplication takes too long and results in larger dictionaries and longer creation time. Both can drag down the overall data analytic performance. Dimension/dictionary expansion : Any changes in the computation fields or business scenarios could entail backtracking of data. Low flexibility of data analysis model : We were unable to query data breakdown with this architecture. A possible solution was to push these queries down to Presto, but the introduction of Presto would mean more O&M troubles. No support for breakdown queries So we started to find a new OLAP engine that could best serve our needs. Later, we narrowed down our choices to ClickHouse and Apache Doris. Due to the high O&M complexity, numerous table types, and lack of support for associated queries, we opted for Apache Doris in the end. In 2019, we built an Apache Doris-based data processing architecture, where both real-time and offline data will be poured into Apache Doris for analysis: We could have created an offline data warehouse in Apache Doris directly, but due to legacy reasons, it would be difficult to migrate all our data there, so we decided to preserve the upper half of our previous offline data link. What's different was that the offline data in Hive would then be written to Apache Doris instead of Apache Kylin. The Broker Load method of Doris was fast. Everyday, it took 10~20 minutes to ingest 100~200G data into Doris. As for the real-time data link, we used Doris-Spark-Connector to ingest data from Kafka to Doris. What's Good About the New Architecture? Deploying Apache Doris is easy since it only has Frontend and Backend processes. Both types of processes can be scaled out, so we can create only one cluster to handle hundreds of machines and dozens of PBs of data. We have been using Apache Doris for three years but only spent very little time on maintenance. Simple O&M with no dependencies on Hadoop components. Having a one-stop data warehouse that is capable of real-time data services, interactive data analysis, and offline data processing makes the development link short and simple. If anything goes wrong, we only need to check a few spots to find the root cause. Easy trouble shooting. This is similar to table association in MySQL. It is helpful in scenarios that require frequent change of data analysis models. Support for JOIN queries in Runtime format. Support for JOIN, aggregate, and breakdown queries. These include rollup index and materialized view. Rollup index allows us to implement secondary index to speed up queries. Support for multiple query acceleration methods. such as Hive, Iceberg, Hudi, and databases like MySQL and Elasticsearch. Support for federated queries across data lakes How Apache Doris Empowers AI In AISPEECH, we use Apache Doris in real-time data query and user-defined conversational data analytics. Real-Time Data Query The real-time data processing pipeline works as above. We ingest offline data via Broker Load, and real-time data via Doris-Spark-Connector. We put almost all our real-time data in Apache Doris, and part of our offline data in Airflow for DAG batch tasks. Our real-time data is in huge volume so we really need a solution that ensures high query efficiency. Meanwhile, we have a 20-member team for data operation and we need to provide data dashboarding services for all of them. That can be challenging for real-time data writing and demanding in query concurrency. Luckily, Apache Doris makes all this happen. User-Defined Conversational Data Analysis This is my favorite part because we have proudly leveraged our natural language processing (NLP) capability in data queries. A normal BI scenario works this way: Data analysts customize the dashboards on a BI platform based on the needs of data users (e.g. financial department and product managers). But we wanted more. We wanted to serve more unique needs of data users, such as roll-up and drill-down at any designated dimension. That's why we enable user-defined conversational data analysis. Unlike normal BI cases, it is the data users but not data analysts who are talking to BI tools. They only need to describe their requirements in natural language and we, utilizing our NLP capabilities, will turn them into SQL. Sounds familiar? Yep, . Except that we've been optimizing it for a long time and fine-tuned it for better integration with Apache Doris so our data users can expect higher hit rates and more accurate parsing results. Then, the generated SQL will be sent to Apache Doris for execution. In this way, data users can view any breakdown details and roll up or drill down any fields. this is like a domain-specific GPT-4 How Apache Doris makes this work well? Apache Doris has the following strengths compared to pre-computed OLAP engines such as Apache Kylin and Apache Druid: Flexible query models that support user-defined scenarios Support for table association, aggregate computing, and breakdown queries Quick response time Our internal users have been giving positive feedback regarding this conversational data analysis. Suggestions We accumulate some practical experience regarding the usage of Apache Doris that we believe can save you some detours. Table Design Use Duplicate table for small data volumes (e.g. less than 10 million rows). Duplicate tables support both aggregate queries and breakdown queries so you don't need to produce an extra table with all the detailed data. Use Aggregate table for large data volumes. Then you can do roll-up indexing, speed up queries via materialized view, and optimize aggregation fields on top of the Aggregate tables. A downside is that since Aggregate tables are pre-computed tables, you will need an extra breakdown table for detail queries. When handling huge amounts of data with many associated tables, generate a flat table via ETL, and then ingest it to Apache Doris, where you can do further optimizations based on the Aggregate table type. Or you can follow the recommended by the Doris community. Join optimizations Storage We isolate hot and warm data in storage. Data of the past year is stored in SSD, and data older than that is stored in HDD. Apache Doris allows us to set a cool-off period for partitions, but the relevant configuration can only be made upon partition creation. Our current solution is automatic synchronization, in which we migrate a certain portion of the historical data from SSD to HDD to make sure that the data of the past year is all placed on SSD. Upgrade Make sure to back up the metadata before upgrading. An alternative method is to start a new cluster, back up the data files to a remote storage system such as S3 or HDFS via Broker, and then import data from the old cluster data into the new cluster by way of backup recovery. Performance After Version Upgrade Starting with Apache Doris 0.12, we've been catching up with the releases of Apache Doris. In our usage scenarios, the latest version can deliver several times higher performance than the early ones, especially in queries with complex functions and statements involving multiple fields. We really appreciate the efforts by the community and strongly advise you to upgrade to the latest version. Apache Doris Summary To wrap up what we've gained from Apache Doris: Apache Doris can be a real-time + offline data warehouse so we only need one ETL script. It saves us a lot of development work and storage costs and also avoids inconsistency between real-time and offline metrics. Apache Doris 1.1.x supports vectorization and thus can deliver 2~3 times higher performance than older versions. Test results show that Apache Doris 1.1.x is on par with ClickHouse in flat table query performance. Apache Doris is powerful itself with no dependency on other components. Unlike Apache Kylin, Apache Druid, and ClickHouse, Apache Doris requires no other component to fill any technical gap. It supports aggregation, breakdown queries, and associated queries. We have migrated over 90% of our analysis to Apache Doris and that has made our O&M much lighter and easier. Apache Doris is easy to use as it supports MySQL protocol and standard SQL.

AIRSPEED

getairspeed.com

11-50 employees

Since 2021

COMPANY RANKING

AIRSPEED

EVERGREEN INDEX #1570

Representing this company? Click here to claim and customize this page!

Airspeed WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AIRSPEED

getairspeed.com

11-50 employees

Since 2021

COMPANY RANKING

AIRSPEED

EVERGREEN INDEX #1570

RELATED COMPANIESRANK

STORIES ABOUT AIRSPEED

Representing this company? Click here to claim and customize this page!

Airspeed WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!