Shape

United States v. Apple INC Court Filing, retrieved on March 21, 2024 is part of . You can jump to any part in this filing . This part is 5 of 25. HackerNoon’s Legal PDF Series here B. Apple invited third-party investment on the iPhone and then imposed tight controls on app creation and app distribution 41. Apple controls how developers distribute and create apps for iPhone users. For example, developers can only distribute native iPhone apps through Apple’s App Store, which is the only way for users to download native iOS apps. Limiting distribution to the Apple App Store enables Apple to exert monopoly power over developers by imposing contractual restrictions and rules that limit the behavior of non-Apple apps and services. Specifically, Apple sets the conditions for apps it allows on the Apple App Store through its App Store Review Guidelines. Under these guidelines, Apple has sole discretion to review and approve all apps and app updates. Apple selectively exercises that discretion to its own benefit, deviating from or changing its guidelines when it suits Apple’s interests and allowing Apple executives to control app reviews and decide whether to approve individual apps or updates. Apple often enforces its App Store rules arbitrarily. And it frequently uses App Store rules and restrictions to penalize and restrict developers that take advantage of technologies that threaten to disrupt, disintermediate, compete with, or erode Apple’s monopoly power. 42. Apple also controls app creation by deciding which APIs are available to developers when they make third-party apps. For example, developers cannot provide native apps on the iPhone unless they enter into Apple’s non-negotiable Developer Program License Agreement (DPLA). That agreement requires developers to use public APIs only “in the manner prescribed by Apple.” It also prohibits third-party apps from using APIs that Apple designates as “private.” Apple selectively designates APIs as public or private to benefit Apple, limiting the functionality developers can offer to iPhone users even when the same functionality is available in Apple’s own apps, or even select third-party apps. Similar to Apple’s App Store restrictions, Apple uses its DPLA to impose restrictions that penalize and restrict developers that take advantage of technologies that threaten to disrupt, disintermediate, compete with, or erode Apple’s monopoly power. 43. Developers cannot avoid Apple’s control of app distribution and app creation by making web apps—apps created using standard programming languages for web-based content and available over the internet—as an alternative to native apps. Many iPhone users do not look for or know how to find web apps, causing web apps to constitute only a small fraction of app usage. Apple recognizes that web apps are not a good alternative to native apps for developers. As one Apple executive acknowledged, “[d]evelopers can’t make much money on the web.” Regardless, Apple can still control the functionality of web apps because Apple requires all web browsers on the iPhone to use WebKit, Apple’s browser engine—the key software components that third-party browsers use to display web content. 44. Nor can developers rely on alternative app stores even though this would benefit developers and users. For example, developers cannot offer iPhone users an app store that only offers apps curated for use by children, which would provide opportunities to improve privacy, security, and child safety. By contrast, Apple allows certain enterprise and public sector customers to offer versions of app stores with more curated apps to better protect privacy and security. 45. Apple’s control over both app distribution and app creation gives Apple tremendous power. For example, Apple designates as “private” the APIs needed to send Short Message Service, or SMS, text messages, which is a protocol used by mobile carriers since the early 1990s to allow users to send basic text messages to other mobile phone numbers using their own mobile phone numbers. Developers have no technical means to access these private APIs, but even if they did, doing so would breach their developer agreement with Apple, and therefore put the developer at risk of losing the ability to distribute apps through the App Store. For example, Apple prohibits third-party iPhone apps from sending or receiving SMS text messages even though this functionality is available through Apple Messages. Likewise, Apple can control the functionality of third-party apps and accessories through its control of app distribution because if an app includes functionality that Apple does not like, Apple can and does exercise its discretion to simply block the app from the App Store. 46. Apple’s dominance is such that neither app developers nor iPhone users can benefit from lower cost or higher quality means of distributing apps or purchasing and providing digital products and services. Instead, Apple guarantees that it continues to benefit from the contributions of third-party developers and other platform participants while also protecting itself from the competitive threats and pressure those participants pose to Apple’s smartphone monopoly. 47. This complaint focuses on Apple’s use of its dominance to impose contracts and rules that restrict the behavior and design decisions of companies other than Apple. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on March 21, 2024, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. justice.gov

Inside Apple's Stranglehold on App Development and How it Shapes Consumer Choices

Artificial Intelligence is no longer a distant dream or faraway fear: it’s impacting the way we learn today. How do I know? Well, I’ve worked for for six years now, helping to connect people who want to learn with online courses provided by professional education providers. CoursesOnline During my time in the industry, I have witnessed firsthand how the e-learning industry has developed and been impacted by other industries and wider societal events. From the to the , there is no doubt that AI is going to be sooner than we might think. steadily growing focus on STEM subjects rise in online learning post-covid changing the face of e-learning 1. AI Teachers and Teaching Assistants Will Become The Norm AI is being used more and more in the delivery of education, on both in-person and e-learning platforms. Because of this, policymakers in the UK have discussed the future of AI in education, and particularly, the development of AI-tutors. When it comes to e-learning, AI tutors would deliver the educational materials through PowerPoint, video lectures, or live seminars, and create learning resources for students. AI would also be able to provide 1 to 1 chat support in clear and natural-sounding language as well as offer private student tuition. AI tutors can be a very useful tool for e-learning due to the time-effectiveness, cost-effectiveness, and consistency which it can provide, although there is concern over human online educators’ job roles. However, It is unlikely that teachers and teaching assistants who deliver e-learning courses will completely shift over to AI, as , increase student motivation, and enhance student-tutor relations. human educators will still be needed to clarify more complex questions, provide pastoral care 2. Grading and Marking Will Be Done Using AI Machine marking has been used for many years when it comes to grading papers, particularly exam papers that have one possible answer (such as multiple-choice tests or maths equations). However, thanks to the development of artificial intelligence and its ability to interpret nuanced answers, AI is likely to be used more and more in the grading and marking of papers. In some ways, this is a positive use of AI as it allows educators more time to teach and provide educational resources whilst also removing any personal bias that a human might have when marking student work. It also allows for for students and tutors. faster marking times, can provide consistent and high-quality student feedback, and is more convenient However, potential errors in marking and an overreliance on automated markers indicate that . regulations on AI should be put in place In some subjects, particularly the humanities such as English Literature and Philosophy, a degree of such as the use of metaphor or humor which, although AI is becoming more capable of, is unlikely to be as proficient as an experienced human examiner. marking is reliant on high-level language use 3. AI Will Be Used to Cheat and to Combat Cheating One of the difficulties with AI is that among the aspects of life it can make easier, it has made cheating in coursework and exams a lot easier. The rise in ChatGPT since 2022 has given way to , particularly on longer-form homework such as essays and coursework as well as exams. more students being able to successfully cheat In terms of e-learning, ChatGPT can be utilized even more easily due to the student having constant access to screens. Although this could be a setback for the e-learning industry, there are methods to combat cheating. One such method is by understanding why students cheat and by making sure any AI is used in a way to rather than hinder it. benefit the students' learning AI can also be used to help combat cheating practically and find areas where students have cheated, using other AI programs to detect changes in students' writing which could be an indicator of cheating. 4. AI Will Create a More Immersive E-Learning Experience Another way in which AI will impact e-learning is by allowing the creation of an immersive experience. Through virtual reality and virtual learning environments (VLEs), learners are able to benefit from a more . collaborative, interesting, and engaging method of online learning AI can be utilized to create realistic virtual learning environments which will help students to engage in a real virtual world. This can help provide a space free from distractions and assist in collaborative student work, as well as making e-learning a more exciting and appealing option. 5. AI Will Enable Personalized and Adaptive Learning In traditional e-learning environments, a tutor usually has several students to manage and therefore is unable to provide a personalized learning experience for every student. However, with the integration of AI into e-learning, for each student's learning style, interests, strengths, and past knowledge. learning resources, and materials can be personalized Unlike in-person learning, where this can pose physical challenges, AI can create different virtual learning environments, a variety of learning materials, and provide personalized learning routes for each student. This could help students understand complex topics more easily and could be especially useful for those who face challenges such as dyslexia or ADHD. However, it is essential to ensure that collaborative work is still incorporated when appropriate and that learning plans can be manually changed if needed. Whether you are keen for the future of AI to shape e-learning or are more apprehensive, there is no mistaking that AI will impact the way we learn online. I will no doubt see the effects of AI at Candlefox for the years to come, and it will influence not only the way that the courses we provide are delivered but also how students interact with them. If_ , then I am confident that we can benefit from the positive effects of AI on e-learning and create a more immersive, inclusive approach to learning without losing the humanity. we use AI in the right way and consider the potential pitfalls

5 Predictions For How AI Is Going To Shape The E-Learning Industry & Online Course Development

5টি ভবিষ্যদ্বাণী কিভাবে AI ই-লার্নিং ইন্ডাস্ট্রি এবং অনলাইন কোর্স ডেভেলপমেন্টকে রূপ দিতে যাচ্ছে

5 Vorhersagen darüber, wie KI die E-Learning-Branche und die Entwicklung von Online-Kursen prägen wird

Cinco predicciones sobre cómo la IA dará forma a la industria del aprendizaje electrónico y al desarrollo de cursos en línea

5 prédictions sur la façon dont l'IA va façonner l'industrie de l'apprentissage en ligne et le développement de cours en ligne

एआई ई-लर्निंग उद्योग और ऑनलाइन पाठ्यक्रम विकास को कैसे आकार देगा, इसके लिए 5 भविष्यवाणियां

AI が E ラーニング業界とオンライン コース開発をどのように形作るかについての 5 つの予測

AI가 이러닝 산업 및 온라인 코스 개발을 어떻게 형성할 것인지에 대한 5가지 예측

5 previsões de como a IA moldará a indústria de e-learning e o desenvolvimento de cursos online

5 прогнозов о том, как ИИ будет формировать индустрию электронного обучения и разработку онлайн-курсов

Yapay Zekanın E-Öğrenim Endüstrisini ve Çevrimiçi Kurs Geliştirmeyi Nasıl Şekillendireceğine İlişkin 5 Tahmin

5 dự đoán về cách AI sẽ định hình ngành học trực tuyến và phát triển khóa học trực tuyến

关于人工智能将如何塑造电子学习行业和在线课程开发的 5 个预测

The Noonification: Update Report: Trialing Basecamps Shape Up Methodology (1/29/2024)

Last year, I attempted to move my product team from the classic SCRUM approach to Basecamp's Shape Up methodology. It was an incredible experience; I've learned a lot from it and thought I would share some of my findings with you. If you've experimented with it yourself, I'd love to hear how it went. If you haven't, I'd love to hear why you stayed away. Part 1: Why Shape Up? My team had been running on SCRUM since forever. During our startup days, we were living the classic tech cycles: work fast, ship, and don't think about processes too much. Then, we got acquired. Once I had a bit more time to consider our product methodology options, I decided to give Shape Up a try. There were a few reasons: Up until then, we were attempting 2-week sprints and pretty consistently failing to finish them. Every week felt like a conveyor belt of tickets we'd never quite finish. The team felt like code monkeys. Pick ticket. Work ticket. Deliver ticket. Because we never finished the sprint, tasks would always spill to the next one. Eventually, the dam breaks. We all lacked focus. Each sprint was a pick & mix of things to work on across the entire code base. Very little teamwork. Each developer would work on their little piece of the pie, leaving little room for learning from each other and, frankly, a sense of community. Almost no customer understanding. Devs would pick up tickets assigned, get them done, and have no clue why/who/what. After re-reading , I thought I'd give it a try as it claimed to solve most of those issues. Basecamp's Shape Up Part 2: Pitching Internally One of the hardest parts of moving to Shape Up was pitching the idea internally. I spent extra time internalizing most of Shape Up's concepts to ensure I was ready for any questions. Unsurprisingly, developers loved the idea of this new approach (more time, more focus, more collaboration; why wouldn't they!). Also unsurprisingly, the hierarchy was more reticent. Ultimately, I managed to convince them by: Ensuring that it was just a trial. If things didn't work out, we'd go back to 'normal.' Ensuring I was going to remain as available as ever to help them. The devs would be focused and uninterruptible, but I wasn't. Highlighting Shape Up allows us to solve bigger problems which means bigger opportunities. Insisting on the pain product is currently experiencing and how it affects each team. I prepared a very clear slide deck and ran each head of department through it (customer service, sales, and C-suite). Part 3: Fears & Concerns My experience as a founder, marketer, and product manager has taught me it's always worth writing concerns down before starting an experiment. I had a few with Shape Up: I know I'm supposed to 'say no.' 'We're busy.' 'Next cycle, we can look into that.' That's the theory, and it works if the whole company is bought into this methodology. During my first cycle trial, I was concerned an emergency would pop up. How do we handle distractions? Shape Up recommends no backlogs (chapter 7). Since we were just trialing this, I obviously didn't go and cmd+a+delete our backlog, but still. If we were to adopt this methodology, I would feel somewhat lost without a backlog. Backlogs? This one really scared me. What happens if we reach the end of the 6-week cycle and we're tentatively there, but not quite? Basecamp says 'start again' (unless you're so super close). I was concerned we'd miss the mark by the annoying 20-30% range. 'Almost finished.' Ultimately, none of these fears/concerns could stop me from carrying on with the trial. It was worth keeping them in mind, though. Part 4: The Cycle And we're off! I kicked off the first cycle on a Monday morning, looking at six weeks of intense focus and teamwork. Here's a summary of what happened each week: : Kick-off and... silence. Leaving the team be, letting them research, and diving into the code on their own time are all a major part of this methodology. It was incredibly hard for me to let go during that first week and not ask for updates. I held strong! Week 1 : The team finally came out of their shell. Communication picked up. Design for the work started appearing, got to give some feedback and work together. Week 2 : In theory, week 3 should be the top of the cycle curve. By the end, the team should have a very good idea of how they're going to build what needs building. We created a cycle-specific Slack channel as communication started to properly ramp up. By the end of that week, we saw prototypes, designs, snippets of code, and more. We were on track! Week 3 : Quiet again. All the back and forth from week 3 produced a focused week 4 as everyone implemented their work. We kicked off a Friday 'show & tell.' Week 4 : Curveball week. One of the scopes started to generate quite a bit of chatter. It quickly became clear the scope wasn't clear enough. I hadn't been precise enough in my requirements, and what initially seemed nice and simple turned out to be complex. I had to make the tough decision to cut this scope. Week 5 : The remaining scopes were ticking away nicely. The intensity drastically picked up in week 6 as I was QA'ing all over the place, and the devs were iterating on my feedback incredibly quickly; we were all pulling together to reach the target. Week 6 In the end, we hit the target. We made it! It was super intense, and I was devastated we had to cut one of the scopes, but we made it. The following Monday at 10 a.m. we shipped into production. Part 5: A Few Lessons In no particular order, here are a few lessons and recommendations: . I thought I had done a decent job shaping most of the scary parts of the cycle. Turns out I missed something blatantly obvious which almost derailed the whole cycle. Shaping is hard . I shaped it mostly on my own, and sometimes, with my dev team lead. It would have been valuable for me to include other developers. Include your team during shaping . Stop everything. Your priority is to figure that thing out before it completely derails everything else. If you find yourself discussing or shaping mid-cycle, something's gone wrong . Whether it is between team members or throughout the cycle, the work intensity is going to greatly vary. As PM, it's your role to spot these pockets of intensity and pay special attention to the individuals going through them. Intensity is not evenly distributed . It made communication much easier but also much more fun. The cycle team quickly developed a shared language, memes related to the work we were working on, and so on. It basically felt like being a startup within the team. Create a separate Slack channel . We waited too long to do this. There should be enough to show or discuss from the end of week 1. It's also an opportunity to meet up, discuss, learn, etc. Implement show & tell meetings from week 1 . All the 'other work' had piled up for 6 weeks; it felt like going right back to SCRUM. This is something I'm still working on improving. The cooldown period turned out to be much harder than the cycle itself As you can probably tell, I was sold by this trial. Implementing Shape Up and adopting its quirks is certainly not an overnight thing. I suspect it'll be a long learning process. I have particularly appreciated the mindset shift this trial has allowed us. We (and the other teams, I hope) learned to see the work for what it is: an exciting challenge we'll overcome together. If you've trialed this (or not), I'd love to read some stories or feedback! If you enjoyed this post, you might enjoy . I write about product management, share actionable insights, and take on real-life product challenges for fun (I know, right?). my newsletter

Update Report: Trialing Basecamp's 'Shape Up' Methodology

আপডেট রিপোর্ট: ট্রায়ালিং বেসক্যাম্পের 'শেপ আপ' পদ্ধতি

Update-Bericht: Erprobung der „Shape Up“-Methodik von Basecamp

Informe de actualización: Prueba de la metodología 'Shape Up' de Basecamp

Rapport de mise à jour : essai de la méthodologie « Shape Up » de Basecamp

अद्यतन रिपोर्ट: बेसकैंप की 'शेप अप' पद्धति का परीक्षण

最新レポート: Basecamp の「シェイプアップ」方法論の試行

업데이트 보고서: Basecamp의 'Shape Up' 방법론 시험

Relatório de atualização: Testando a metodologia 'Shape Up' do Basecamp

Обновленный отчет: испытание методологии Basecamp «Shape Up»

Güncelleme Raporu: Basecamp'ın 'Şekillendirme' Metodolojisini Denemek

Báo cáo cập nhật: Thử nghiệm phương pháp 'Shape Up' của Basecamp

更新报告：试用 Basecamp 的“Shape Up”方法

Office of Science and Technology Policy Internships President Biden often says, “America is the only nation that can be defined by a single word: possibilities.” The White House Office of Science and Technology (OSTP) works to bring that idea to life by harnessing the power of science, technology, and innovation to achieve America’s greatest aspirations. OSTP’s mission includes: Providing advice to the President and the Executive Office of the President on matters related to science and technology; Strengthening and advancing American science and technology; Working with federal departments and agencies and with Congress to create bold visions, unified strategies, clear plans, wise policies, and effective, equitable programs for science and technology; Engaging with external partners, including industry, academia, philanthropic organizations, and civil society; state, local, Tribal and territorial governments; and other nations; and, Ensuring equity, inclusion, and integrity in all aspects of science and technology. OSTP has six core teams: Climate and Environment Energy Health Outcomes National Security Science and Society Office of the U.S. Chief Technology Officer (CTO) the President and First Lady’s , the (PCAST), the , the , the , the , and the . OSTP also houses or supports: Cancer Moonshot President’s Council of Advisors on Science and Technology National Science and Technology Council National Artificial Intelligence Initiative Office National Quantum Coordination Office National Nanotechnology Coordination Office Arctic Executive Steering Committee OSTP is led by a Senate-confirmed Director, , who also serves as Assistant to the President for Science and Technology, a member of the President’s Cabinet, and Co-Chair of PCAST. Arati Prabhakar OSTP was established by the National Science and Technology Policy, Organization, and Priorities Act of 1976. Before OSTP, presidents dating back to at least Franklin D. Roosevelt had various configurations of science advisors, committees of external science advisors, and White House offices dedicated to science and technology. Internship Details The Summer 2024 term begins on June 10, 2024 and ends on August 16, 2024. Interns will receive a stipend of $7,500 for participation in the program. Internships will be hosted in person on the White House campus, located in Washington, D.C. Please note that living accommodations and relocation support will not be provided. All internships are full-time and participants must be able to commit 35-40 hours per week (Monday-Friday). Applicants who receive an offer will be required to complete the Standard Form (SF) 86 to determine whether they meet security eligibility requirements.  Information requested on the SF86 that could affect eligibility includes, but is not limited to, an applicant’s connections to foreign governments, criminal history, financial debts and tax compliance, and prior drug use (including marijuana, regardless of whether the marijuana use was permitted under state law). Eligibility Requirements Must be a U.S. citizen Must be 18 years of age or older Males must be registered with Selective Service Must be currently enrolled in an accredited undergraduate or graduate college, community college, or university Must obtain a favorable security determination and pass a pre-employment drug test If you are not a U.S. Citizen you should not apply for this internship Key Dates for Upcoming Sessions OA offers three internship programs each year: Fall, Spring, and Summer. FALL 2023 September 11, 2023 Program Start Date: December 15, 2023 Program End Date: March 17, 2023 Application Open Date: April 14, at 11:59 PM ET Application Close Date: SPRING 2024 January 22, 2024 Program Start Date: May 10, 2024 Program End Date: July 24, 2023 Application Open Date: August 28, 2023 at 12:00 PM ET Application Close Date: SUMMER 2024 June 10, 2024 Program Start Date: August 16, 2024 Program End Date: October 16, 2023 Application Open Date: November 19, 2023 at 11:59 PM ET Application Close Date: This was originally published on whitehouse.gov.

Shape America's Future: Join the White House OSTP Internship Program

What data management in 2024 and beyond will look like hangs on one question. Can open data formats lead to a best-of-breed data management platform? It will take Interoperability across clouds and formats, as well as on the semantics and governance layer. Sixth Platform. Atlas. Debezium. DCAT. Egeria. Nessie. Mesh. Paimon. Transmogrification. This veritable word soup sounds like something that jumped out of a role-playing game. In reality, these are all terms related to data management. These terms may tell us something about the mindscape of people involved in data management tools and nomenclature, but that’s a different story. Data management is a long-standing and ever-evolving practice. The days of “Big Data” have long faded and data has now taken a back seat in terms of hype and attention. is the new toy people are excited about, shining a light on AI and machine learning for the masses. Generative AI Data management may seem dull in comparison. However, there is something people who have been into AI before it was cool understand well, and people who are new to this eventually realize too: AI is only as good as the data it operates on. There is an evolutionary chain leading from . Some organizations were well aware of this a decade ago. Others will have to learn the hard way today. data to analytics to AI Exploring and understanding how, where, and when data is stored, managed, integrated and used as well as related aspects of data formats, interoperability, semantics, and governance may be hard and unglamorous work. But it is what generates value for organizations, along with -like word soups. RPG Peter Corless and Alex Merced understand this. Their engagement with data goes back to way before their current roles as Director of Product Marketing at and Developer Advocate at , respectively. We caught up to talk about the state of data management in 2024, and where it may be headed. StarTree Dremio https://www.youtube.com/watch?v=Meq247f31-A&embedable=true The sixth data platform For many organizations today, data management comes down to handing over their data to one of the “Big 5” data vendors: , Azure, and , plus Snowflake and Databricks. However, analysts David Vellante and George Gilbert believe that the need of modern data applications coupled with the evolution of open storage management may lead to the emergence of a “sixth data platform”. Amazon Microsoft Google The notion of the “sixth data platform” was the starting point for this conversation with Corless and Merced. is that open data formats may enable interoperability, leading the transition away from vertically integrated vendor-controlled platforms towards independent management of data storage and permissions. The sixth data platform hypothesis It’s an interesting scenario and one that would benefit users by forcing vendors to compete for every workload based on the business value delivered, irrespective of lock-in. But how close are we to realizing this? To answer this question, we need to examine open data formats and their properties. In turn, to understand data formats a brief historical overview is needed. https://www.youtube.com/watch?v=j7s8xSTxkYk&embedable=true We might call this table stakes, and it’s not just a pun. Historically, organizations would resort to using data warehouses for their analytics needs. These were databases specifically designed for analytics, as opposed to transactional applications. Traditional data warehouses worked, but scaling them was an issue as Merced pointed out. Scaling traditional data warehouses was expensive and cumbersome because it meant buying new hardware with storage and computing bundled. This was the problem Hadoop was meant to solve when introduced in the early 2010s, by separating storage and compute. Hadoop made scaling easier, but it was cumbersome to use. This is why an SQL interface was eventually created for Hadoop via , making it more accessible. Hive used metadata and introduced a protocol for mapping files, which was what Hadoop operated on, to tables, which is what SQL operates on. That was the beginning of open data formats. Apache Hive Data lakes, data lakehouses, and open data formats Hadoop also signified the emergence of the data lake. Since storage was now cheap and easy to add, that opened up the possibility of storing all data in one big Hadoop store: the data lake. Eventually, however, , which is the realm the “Big 5” operate on. Hadoop and on-premise compute and storage gave way to the cloud “The idea of a data lakehouse was, hey, you know what? We love this decoupling of compute and storage. We love the cloud. But wouldn’t it be nice if we didn’t have to duplicate the data and we could just operate over the data store you already have, your data lake, and then just take all that data warehouse functionality and start trying to move it on there?”, as Merced put it. However, as he added, that turned out to be a tall order, because there were elements of traditional data management missing. Other cloud-based data management platforms introduced notions like automated sharding, partitioning, distribution, and replication. That came as part of the move away from data centers and file systems towards the cloud and APIs to access data. But for data lakehouses, these things are not a given. This is where open data formats such as and come in. They all have some things in common, such as the use of metadata to abstract and optimize file operations and being agnostic about storage. However, Hudi and Delta Lake both continued to build on what Hive started, while Iceberg departed from that. Apache Iceberg, Apache Hudi, Delta Lake https://www.youtube.com/watch?v=043GGYTZCbk&embedable=true A detailed open data format comparison is a nuanced exercise, but perhaps the most important question is: Does interoperability exist between open data formats, and if yes, on what level? For the sixth data platform vision to become a reality, Interoperability should exist among clouds and formats, as well as on the semantics and governance level. Interoperability of the same format between different cloud storage vendors is possible, but not easy. It can be done, but it’s not an inherent feature of the data formats themselves. It’s done on the application level, either via custom integration or by using something like Dremio. And there will be egress and network costs from transferring files outside the cloud where they reside. Interoperability of different formats is also possible, but not perfect. It can be done by using a third-party application, but there are also a couple of other options like or . Neither works 100%, as Delta Uniform batches transactions and OneTable is for migration purposes. Delta Uniform OneTable As Merced noted, creating a solution that works 100% would probably incur a lot of overhead and complexity as metadata would have to be synchronized across formats. Merced thinks these solutions are motivated at least partially by Iceberg’s growth and the desire to ensure third-party tools can work with the Iceberg ecosystem. To federate or not to federate? What’s certain is that there’s always gonna be sprinkles of data in other systems or multiple clouds. That’s why federation and virtualization that can work at scale is needed, as Corless noted. Schema management is complicated enough on one system, and trying to manage changes across three different systems would not make things easier. Whether data is left where it originally resides and used via federated queries, or ingested in one storage location is a key architectural question. As such, there are tradeoffs involved that need to be understood. Even within federation, there are different ways to go about it – Dremio, GraphQL, Pinot, Trino, and more. “How do data consumers want to size up these problems? How can we give them predictive ways to plan for these trade-offs? They could use federated queries, or a hybrid table of real time and batch data. We don’t even have a grammar to describe those kinds of hybrid or complex data products these days”, Corless said. Corless has a slightly different point of view, focusing on real-time data and processing. This is what StarTree does, as it builds on t. Pinot is a real-time distributed OLAP datastore, designed to answer OLAP queries with low latency. Apache Pino Corless also noted that the choice of data format depends on several parameters. Some formats are optimized for in-memory use, like Others like are optimized for disk storage. Apache Arrow. Apache Parquet There are attempts to have common representations of data both in-memory and on storage, motivated by the need to leverage tiered storage. Tiered storage may utilize different media, ranging from in-memory to SSD to some sort of blob storage. “People want flexibility in where and how they store their data. If they have to do transmogrification in real time, that largely defeats the purpose of what they’re trying to do with tiered storage. Is there something like the best, most universal format? Whenever you’re optimizing for something, you’re optimizing away from something else. But I’m very eager to see where that kind of universal representation of data is taking us right now.”, Corless said. Semantics and governance Regardless of where or how data is stored, however, true interoperability should also include semantics and governance. This is where the current state of affairs leaves a lot to be desired. As Merced shared, the way all table formats work is that the metadata has information about the table, but not about the overall catalog of tables. That means they are not able to document the semantics of tables and how they relate to each other. A standard format of doing that doesn’t exist yet. Merced also noted that there is something that addresses this gap, but it only works for Iceberg tables. is an open-source platform that creates a catalog, thus making tracking and versioning Iceberg tables and views possible. Project Nessie Nessie also incorporates elements of governance, and Merced noted that Nessie will eventually have Delta Lake support too. Databricks on its part offers which works with Delta Lake, but it’s a proprietary product. There is no lack of data catalog products in the market, but none of that can really be considered the solution to semantics and data interoperability. Unity Catalog Corless on his part noted that there a standard called DCAT. DCAT, which stands for Data Catalog Vocabulary, is an RDF vocabulary designed to facilitate interoperability between data catalogs published on the Web. DCAT was recently updated to v.3, it’s been around for over a decade, and it’s precisely aimed at interoperability. https://www.youtube.com/watch?v=Pb53L77_nS8&embedable=true The fact that DCAT is not used widely has probably more to do with vendors reinventing the wheel and/or aiming for lock-in, as well as clients not being more proactive in requiring interoperability standards than with DCAT itself, as per Corless. Unfortunately, it seems like what data is to AI, governance is to data for most organizations: a distant second at best. Forty-two percent of data and analytics leaders do not assess, measure or monitor their data and analytics governance, according to a . Those who said they measured their governance activity mainly focused on achieving compliance-oriented goals. 2020 Gartner survey The onset of . was an effort to standardize governance for data lakes leveraging DCAT and other metadata vocabularies. Today, however, both Data Lakes and Atlas seem to have fallen out of fashion. GDPR in 2018 marked the opportunity for governance to leverage metadata and semantics to rise up Apache Atlas A new project called was spun out of Atlas, aiming to address more than data lakes. Furthermore, there is another open standard for metadata and data lineage called and several lineage platforms that support it, including Egeria. Egeria OpenLineage Conclusion So, what is the verdict? Is the sixth data platform possible for data management in 2024 or is it a pipedream? Merced thinks that we’re already starting to see it. In his view, using open formats like Apache Iceberg, Apache Arrow, and Apache Parquet can create a greater level of interoperability. This interoperability is still imperfect, he noted, but it’s a much better state of things than in the past. Dremio’s thesis is to be an open data lakehouse, not just operating on the data lake, but across tools and data sources. Corless thinks that we’re going to see a drive toward clusters of clusters, or systems of systems. There’s going to be a reinforced drive towards the automation of integration. More like Lego bricks that snap together easily, rather than Ikea furniture that comes with a hex wrench. But for that to happen, he noted, we’ll need a language and a grammar so that systems and people can both understand each other. The ingredients for a sixth data platform are either there, or close enough. As complexity and fragmentation explode, a language and grammar for interoperability sounds like a good place to start. Admittedly, interoperability and semantics are hard. Even that, however, may be more of a people and market issue than a technical one. DCAT and OpenLineage are just some of the vocabularies out there. Even things as infamously hard to define as data mesh and data products have a vocabulary of their own – the . Data Product Descriptor Specification Perhaps then the right stance here would be cautious optimism. Also published . here

Data Management in 2024: Will Open Data Formats Shape a “Sixth Platform”?

You can jump to any section of the National Aeronautics Science & Technology Priorities here. Executive Summary In recent years, America’s communities have faced unprecedented shocks and stresses due to natural hazards, accidents, and human-caused disruptions, some occurring in tandem or as cascading disasters. Helping individuals, families, and communities recover from these immense challenges while building their resilience to historical, ongoing, and future stressors is a challenge that requires a wholeof-community effort, which includes harnessing science, technology, and innovation. The Bipartisan Infrastructure Law invests in addressing these challenges, including over $50 billion to build the nation’s resilience to the impacts of climate and extreme weather hazards, and directs investment in billions more in infrastructure that will need to be resilient to both natural and human-caused hazards. To contribute to addressing the challenge of building resilience, the United States National Science and Technology Council’s (NSTC) Subcommittee on Resilience Science and Technology (SRST) establishes a Resilience Science and Technology Grand Pathways Framework in this report. This Framework uses a focus on strengthening specific societal dimensions of resilience to better identify, align, and prioritize investments in science and technology to improve community resilience. The report defines and examines the interdependencies among five societal dimensions of resilience: safety and security, financial/economic resilience, healthy people, social cohesion, and trusted effective governance. These societal dimensions are compared to cross-cutting enablers, defined as the enabling resources and capabilities (for example, education, infrastructure, financial resources, and policy) that build towards the end states of a resilient community. The Framework weaves together the societal dimensions and cross-cutting enablers with key functions of a resilient community, which are defined as actions or services (e.g., continuity of government, emergency response operations) that use cross-cutting enablers to support one or more societal dimensions. Ultimately, the Framework integrates societal dimensions, cross-cutting enablers, and key functions to identify the critical gaps that impede resilience and that can be addressed by the advancement and application of resilience science and technology. The Grand Pathways Framework is intended to help users identify, align, and prioritize science and technology solutions to improve resilience. The Framework identifies three modes of resilience: the process of resilience (anticipating, avoiding, adapting, withstanding, and building back better), the property of resilience, and the desired outcomes of societal resilience. In applying the Framework, SRST encourages adopters to carefully consider the distinctions between and relationships among the crosscutting enablers, key functions, and societal dimensions of resilience. Resilience is an ever-evolving challenge, as novel and unpredictable stressors continue to emerge, and historical and environmental factors influence the path to resilience. Resilience can be improved by investments in science and technology. The Grand Pathways Framework looks to support decisionmakers in achieving the goal of resilience by helping them design and implement multipurpose science and technology solutions that strengthen a community’s ability to withstand a wide variety of acute shocks and chronic stressors. This was originally published in March 2023 on It has been broken-up into bite sized sections, each with unique headlines and AI-generated lead images, in line with the provisions of whitehouse.gov. Creative Commons 3.0.

Charting the Course to Community Resilience: How Science and Technology Shape the Future

As a tech that allows companies to build powerful systems and store data securely, blockchain has found , especially in the field of finance. The decentralization capabilities have also become a valuable asset in political, social, and economic areas. quick adoption Like so many young technologies, blockchain works very well on paper. In fact, blockchain technology has been reviewed from every possible angle and its future is . very promising Despite the glowing reviews, however, we have yet to experience a futuristic world run on blockchain systems. Risks often come from an unexpected source, one that is not necessarily considered when securing blockchain systems: the human factor. The human factor has prevented blockchain from flourishing. This article will look at how and why. System Building Decentralized systems are arguably the most interesting aspect of the blockchain ecosystem. Initially, it promised a true revolution as a financial system not controlled by banks. Human behavior, however, affects the rules and protocol configuration. Selecting or is always determined by the way humans interact with the system, which can allow people to take advantage of it. consensus algorithms security parameters While some architectures can successfully navigate the human factor—such as the , which was originally designed as a captcha mechanism in the early 2000s—multiple architectures are more susceptible to external attacks. For example, happen regularly as a result of asset concentration. proof of work consensus 51% of attacks Another factor to consider is how the blockchain is managed. There are also private blockchains and several decentralized platforms—such as Hyperledger, Corda, and Quorum—but a blockchain can’t be controlled and regulated by a single authority, as that would defeat the purpose. Security And Audits Despite its efficient operation, blockchain cannot create or evolve its security measures. During the process of development, human error can create vulnerabilities within the system. Regular and efficient audits are, therefore, vital. Ideally, every time a new string of code is written, it should be audited and tested separately and with the whole contract. But some cut corners, eager to release the product quickly. External attacks are , and the responsibility for them falls equally on the attacker and the protocol owner. You can’t prevent every attack, but there are mechanisms to contain it or stop it altogether. very common Security has to be an essential part of the whole development process. That’s where consensus audits and attack threat modeling come in. Unlike a code audit, this process involves the initial idea and possibly some block diagrams. Consensus audits review the main rules that need to be followed for a consensus to be reached. Something like passing a law or a bill in the government. Attack threat modeling follows that law analogy and shows what happens if the law is abused. By modeling such situations in advance, you can help prevent them. We encounter this all the time with , especially with experienced companies as they order attack threat modeling to be able to see in advance which types of attacks would be likely used against the architecture. Consulting during the early development stages can provide project owners with the necessary information and help them overview the scope of the work that lies ahead. HashEx There are methods to secure blockchains. All we need is to take advantage of them. Interface And Accessibility One of the primary goals for all blockchain developers is increasing the adoption of this technology. So far it’s been a slow and painful process, mostly due to blockchain’s complexity. Psychologically, we aren’t inclined to use a system that consumes a lot of time and requires extensive education. Humans want and will always choose user-friendly options. That’s what blockchain has to become: user-friendly. Sure, it starts with a code, but it doesn’t have to be complicated. There have been many suggested uses for blockchain in commerce, healthcare or when working with documents. In all cases, one of the main issues preventing adoption is usability. To understand this, here are a few examples that illustrate the practical applications of blockchain: Walmart has been at the into its supply chain management, particularly in the realm of food safety and traceability. By implementing a blockchain-based system, Walmart has enhanced the transparency and efficiency of its supply chain. This technology allows for precise tracking of food items throughout the entire supply chain, from farm to store shelves. The primary goal is to ensure food safety by quickly identifying and addressing any potential issues in the food supply chain. forefront of integrating blockchain technology Another sector where blockchain has made significant strides is in digital identity verification. Estonia's e-Residency program is a pioneering example of how . blockchain can secure personal data In the healthcare industry, blockchain-based systems are being developed to in order to improve the security and interoperability of healthcare information. manage electronic medical records By focusing on user-centric design, blockchain can be made more accessible, encouraging adoption across various industries. What Blockchain Needs Some may disagree, but blockchain doesn’t have the best reputation today. Yes, it’s been an incredible help to many areas, but its notoriety is mostly connected to financial scams and failures of the market. The key to changing that is education. And I don’t just mean educating people about the technology and its advantages. We need to educate professionals to take over the development and management of blockchain-based systems. We have to involve it so it can do more good. And it’s most necessary to fix our previous mistakes. Maximizing the positive effects of this technology should be a priority. Blockchain can’t remain a closed field for a group of skilled experts. For the technology to truly bloom, it has to become accessible. Lead image by Terry on Unsplash

How the Human Element Shapes and Limits Blockchain Development

SHAPE

shape.ci

-- employees

Since n.d.

COMPANY RANKING

SHAPE

EVERGREEN INDEX #9922

Representing this company? Click here to claim and customize this page!

Shape WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SHAPE

shape.ci

-- employees

Since n.d.

COMPANY RANKING

SHAPE

EVERGREEN INDEX #9922

RELATED COMPANIESRANK

STORIES ABOUT SHAPE

Representing this company? Click here to claim and customize this page!

Shape WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!