CertifyOS is a first-of-its-kind provider intelligence platform, powered by API integrations and hundreds of verified data points. We unlock insights and power performance for clinicians, teams and organizations, with frictionless licensing and enrollment, one-click credentialing and real-time network monitoring at your fingertips.

CertifyOS

certifyos

Say it with me: data engineers are not a data catalog. You would be hard-pressed to find “answering multiple Slack messages every week about which tables are good to use for this report,” in their job description, but it happens nonetheless. Data analysts aren’t psychic. Yet, they are often placed in the position of having to intuit if the data being piped is trustworthy. This misalignment has arisen as data teams are pushed to move faster, weave themselves across the , and enable increasingly self-service data platforms. data mesh It’s the data team’s equivalent of the classic document version control issues that have plagued knowledge workers for decades. What starts as a tight pitch deck evolves into: A million people making and sharing ad-hoc slides; Massaging content on those slides until it becomes an echo of its original intent; and Creating copies labeled V6_Final_RealFinal. The same thing happens across the data team. Everyone is trying to do the right thing (i.e., support your stakeholders, generate insights, pipe more data, etc.), but everyone is also moving fast. One day you look up and notice you have 6 different models with slight variations essentially doing the same thing…and no one knows which one is most up-to-date or even which field to use. This creates real operational problems downstream including: Inefficient cycles of redundant “traffic control;” Lower data quality; Time spent resolving problems created by analysts using improper/problematic data; Lower data trust across the organization; and Increased data downtime When you don’t trust your data or you have lower data reliability, organizations often pad the margins of error in their forecasts. As highlighted by Peleton’s , poor forecasting can be especially problematic during the pandemic when uncertainty across demand, supply chains, and the overall business environment is at an all-time high. recent production halt More data discovery, more problems Data discovery is a new approach to understanding the health of your distributed data assets in real-time, and it’s an essential part of the solution. Data discovery provides a domain-specific, dynamic understanding of your data based on how it’s being ingested, stored, aggregated, and used by a set of specific consumers. As with a data catalog, governance standards and tooling are federated across these domains (allowing for greater accessibility and interoperability), but unlike a data catalog, data discovery surfaces a real-time understanding of the data’s current state as opposed to its ideal or “cataloged” state. It is especially useful when teams take a distributed approach to that holds different data owners accountable for their data as products, which allows data-savvy users throughout the business to self-serve from those products. governance But as data becomes more accessible, how can downstream stakeholders determine what data sets have been served, transformed, and approved by a given domain’s data team? How can one domain be sure a common set of are being upheld across the organization? data quality standards, ownership, and communication processes One of my customers, a leading media company with a mature data organization, was facing these exact questions. As a result, we have been working with them and several others to implement a data certification program. What is data certification? is the process by which data assets are approved for use across the organization after having met mutually agreed-upon , or service-level agreements,  for data quality, observability, ownership/accountability, issue resolution, and communication. Data certification SLAs Similar to the concepts of data quality, data validation, or data verification, data certification layers on critical processes that align people, frameworks, and technology to central business policies. Data certification requirements vary based on the needs of the business, the capacity of the data engineering team, and the availability of data, but typically incorporate the following features: Data certification programs increase scalability by leveraging a consistent approach applied across multiple domains. They also increase efficiency by facilitating more trustworthy exchanges of information between domains with clear lines of communication. Here’s how it works. Step 1: Build out your data observability capabilities Implementing data observability–an organization’s ability to fully understand the health of the data in their system–is an important first step in the data certification process. Not only do you need insight into your current performance to set a baseline, but you also need a systemic end-to-end approach for proactive incident discovery, alerting, and triaging. If within the pipeline breaks–and it will break–you will be the first to know. This head start, along with a detailed understanding of the data ecosystem, will reduce time to detection and resolution by pinpointing where errors occur. anything Knowing what systems and data sets have a tendency to create the largest or most frequent problems downstream also helps inform the process of writing effective data SLAs (Step 4). Additionally, understanding the upstream dependencies of your most important tables or reports helps data teams understand what data to give the most attention. The bottom line is that a table or data set should be closely monitored for anomalies  (ideally continuously learning and evolving via machine learning) to be considered certified. Step 2: Determine your data owners Each certified data asset should have a responsible party across its lifecycle from the ingestion to the analytics layer. Some data teams may choose to implement a , others may build it directly into the specific SLA along with the expected communication procedures and resolution times. RACI (responsible, accountable, consulted, informed) matrix Step 3. Understand what “good” data looks like By asking your business stakeholders the “who, what, when, where, and why,” you can understand what data quality means to them and which data is actually the most important. This will enable you to develop key performance indicators such as: : Freshness Data will be refreshed by 7:00 am daily (great for cases where the CEO or other key executives are checking their dashboards at 7:30 am). Data will never be older than X hours. : Distribution Column X will never be null. Column Y will always be unique. Field X will always be equal to or greater than field Y. : Volume Table X will never decrease in size. : Schema No fields will be deleted from this table. : Lineage 100% of the data populating table X will have upstream sources and downstream ingestors mapped and include relevant metadata. (or availability) Data downtime : The number of incidents is multiplied by (the time to detection + time to resolution). An example of a data downtime SLA could be, table X will have less than Y hours of downtime a year. SLAs that measure each of the components of data downtime can be more actionable. Examples include: we will reduce our incidents X%, time to detection X%, and time to resolution X%. Query Speed: Our friends at : “Average query run time is a good place to start, but you may need to create a more nuanced metric (e.g., X% of queries finish in <Y seconds). Locally Optimistic suggest Ingestion (great for keeping external partners accountable): Data will be received by 5 am each morning from partner Y. This process also enables you to configure granular alerting rules tailored to what matters most to the business. Step 4: Set clear SLAs for your most important data sets Setting SLAs (service level agreements) for your data pipeline is a major step toward increasing your data reliability and is essential to a data certification program. SLAs need to be specific, measurable, and achievable. Not only do SLAs describe an agreed-upon standard of service, but they also define the relationship between parties. In other words, they outline who is responsible for what during normal operations as well as when issues occur. , a Senior Data Scientist with Red Ventures, suggests that an effective SLA is realistic. Simply saying “having reliable data at all times” is too vague to be useful; instead, Brandon suggests, that teams should set SLAs that are focused. Brandon Beidel “Good SLAs are specific and detailed. They will describe why it’s important to the business, what the expectations are when those expectations need to be met, how they will be met, where the data lives, and who is impacted by it.” Beidel includes within his SLAs how the team should respond if the SLA isn’t met. For example, “the data in table X will be refreshed every day by 7:00 am” will transform into, “Team Z will ensure the data in table X will be refreshed every day by 7:00 am. Within 2 hours of an anomaly alert, the team will verify, communicate to affected parties, and begin a root cause analysis of the issue. Within one business day, a ticket will be created and the wider team will be updated on the progress made toward resolution.” To achieve this level of specificity and organization, teams should align early – and often – with stakeholders to understand what good data looks like. That includes within the data team as well as the business. A good SLA needs to be informed by the realities of how the business operates and how your users consume the data. I take a slightly different approach and differentiate between what I consider the SLA of “table x will be updated by 7 am” and the SLO (Service Level Objective) of “we will aim to meet this SLA 99% of the time.” However you decide to approach it, I’d recommend against boiling the ocean. Most of my customers are implementing their data certification programs as “go forward” first and cleaning up older assets in a second wave. In fact, many of the best data teams will start certifying the most critical tables and data sets: the ones that add the most value to the business, have the most query activity, the number of users, or dependencies. Some are also implementing tiers of certification–bronze, silver, gold–that convey different levels of service and support. Step 5: Develop your communication and incident management processes Where and how will alerts be sent to the team? How will the next steps and progress be communicated internally and externally? While this may seem like table stakes, clear and transparent communication is essential to creating a culture of accountability. Many teams opt to have alerts and incident triage discussions take place in Slack, PagerDuty or Microsoft Teams. This enables rapid coordination while giving full transparency to the wider team as part of a health incident management workflow. It’s also important to consider how to communicate major outages to the rest of the organization. For example, if an alert turns out to be a huge production outage, how does the on-call engineer inform the rest of the company? Where do they make that announcement and how frequently do they provide updates? Step 6: Determine a mechanism to tag the data as certified At this point, you have created SLAs with measurable objectives, transparent ownership, clear communication processes, and strong issue resolution expectations. You have the tools and proactive measures in place to empower your teams to be successful. The final step is to certify and surface the approved data assets for your stakeholders. I recommend decentralizing the certification process. After all, the certification process is designed to help make teams faster and more scalable. Having centralized regulations, enacted at the domain level will achieve these goals and avoid creating too much red tape. For the certification process, data teams will tag, search and leverage their tables appropriately either using data discovery solutions, a home-grown tool, or some other form of data catalog. Step 7: Train your data team and downstream consumers Of course, just because tables are tagged as certified doesn’t guarantee analysts will stay inbound. The team will need to be trained in the proper procedures, which will need to be enforced as necessary. Fine-tuning the level of alerts and communication is important as well. Occasionally receiving alerts that don’t require action is healthy. For example, you may have a table that grows significantly in size, but it was expected because the team added a new data source. Nothing is broken and in need of fixing, but it’s still helpful for the team to know. After all, “expected” behavior to one person might still be newsworthy and critical to another member of the team – or even another domain. However, alert fatigue is real. If the team is starting to ignore alerts, it can be a sign to optimize your approach by either adjusting your monitors or bi-furcating communication channels to better surface the most important information. When it comes to your data consumers, don’t be shy! You have put in an incredibly robust system for data quality aligned to their needs. Help them move from a subjective to objective understanding of how your team is performing and start giving them the vocabulary to be part of the solution. It’s all about proactive communication Data certification can be a beautiful process to see in action. The data engineer tags the table as certified along with the owner of the data set and surfaces it within the data warehouse for an analyst to grab it and use in their dashboard. And voila! No more (or at least, a whole lot less) data downtime. At its core, this process underscores that without the proper processes and culture in place, certifying reliability and building organizational trust in your data is extremely difficult. Technology will never be a replacement for good data hygiene, but it certainly helps. Also published here.

Certify Your Data Assets to Avoid Treating Your Data Engineers Like Catalogs

Artificial intelligence is transforming industries across the board. From to finance, companies are racing to incorporate AI into their products and services. AI promises immense benefits like heightened efficiency, deeper insights, and automation of tedious tasks. However, amidst the AI gold rush, a troubling trend has emerged: AI washing. healthcare What Is AI Washing? AI washing refers to the deceptive practice of misrepresenting a product or service as utilizing more advanced AI than it truly does. Companies may exaggerate their AI capabilities and benefits to appear more technologically sophisticated. With AI dominating tech headlines, it has become a powerful marketing tool. But as tempting as it is to jump on the AI bandwagon, AI washing can have serious consequences for businesses and consumers alike. AI washing takes its name from the term , which refers to companies making dubious or misleading claims about having a positive environmental impact. Just as some brands exaggerate their eco-credentials through greenwashing, AI washing involves companies exaggerating their use of artificial intelligence for marketing purposes without substance behind the claims. Understanding this parallel helps explain why the practice of inflating AI capabilities became known as "AI washing." greenwashing A recent example of AI washing is Equinix, an American multinational company that claims to be a global leader in the data center market. published a report accusing Equinix of riding the AI bandwagon by making false claims about its power capacity to meet AI demands. This led to a 30% increase in Equinix's valuation to $80 billion in the past year. Hindenburg Research The Rise of AI Washing Vendors may claim a product uses AI when it actually relies on simple rules-based programming and algorithms. While rules and algorithms do play a role in AI, true AI systems involve technologies like machine learning and neural networks that enable more advanced capabilities like natural language processing, computer vision, and predictive analytics. AI washing takes advantage of the hype and ambiguity surrounding true AI. By making dubious AI claims, companies aim to appear more innovative, get more media coverage, attract investors, and charge higher prices for their products and services. Several factors have contributed to the proliferation of AI washing in recent years: Excitement and optimism around AI have led to intense media coverage and exaggerated claims. This hype motivates companies to jump on the AI bandwagon. AI hype: There is no single definition of what constitutes true AI. The lack of clarity enables companies to call any type of automation or rules-based software "artificial intelligence." AI ambiguity: As interest in AI grows, companies feel pressure to showcase AI capabilities even if their technology is not that advanced. AI washing fills this demand. Demand for AI: Currently, there are no laws or standards governing AI claims in marketing. This regulatory gap enables companies to freely engage in AI washing without legal consequences. Lack of regulation: As AI becomes more prevalent, consumers and businesses should be aware of these motivations and strategies for AI washing. Evaluating AI claims critically is key to avoiding deception. The US Securities and Exchange Commission (SEC) recently penalized two investment advisory companies, Delphia (USA) Inc. and Global Predictions Inc., with a $400,000 fine for AI washing. Delphia claimed to use AI to predict successful companies and trends for early investment, while Global Predictions boasted of being the first AI-regulated financial advisor. However, the and the public about their AI usage. SEC found that both companies misled clients Examples of AI Washing Many industries are now rife with AI washing. Here are some common examples of exaggerated or misleading claims about artificial intelligence: Customer service chatbots are often touted as using AI when they actually rely on simple rules and scripts. They may have limited abilities to understand natural language or handle new queries. True AI chatbots utilize machine learning to train on diverse conversations and improve their capabilities over time. Chatbots: Many companies promote product recommendation features as AI-driven. In reality, they frequently rely on simplistic rules like identifying commonly purchased items together. Genuine AI recommendation systems apply complex algorithms to large datasets to uncover non-obvious patterns and predictively model customer preferences. Recommendation engines: Basic image analysis software is sometimes positioned as artificial intelligence for marketing appeal. Most can only identify objects, people, or text that were explicitly programmed for recognition. Advanced AI image recognition leverages deep learning neural networks to understand images contextually and generalize to novel inputs. Image recognition: Software that forecasts outcomes like future sales by extrapolating historical data is not necessarily AI-enabled. True AI predictive analytics systems can process more variables, recognize complex patterns, and continuously learn from new data to improve predictions. Predictive analytics: While big data platforms enable the collection and storage of vast datasets, they do not automatically include AI capabilities. AI-powered big data analytics applies machine learning algorithms to large datasets to uncover insights that would not be visible to humans or standard analytics. Big data analytics: These examples reveal how the lines between conventional software and true artificial intelligence are often blurred. As AI washing becomes more common, it's essential for businesses to evaluate vendor claims critically before making purchase decisions. Looking past the AI hype to understand underlying technologies is key. How to Spot AI Washing Since there are no official auditing standards for AI, identifying exaggerated claims comes down to asking the right questions and investigating vague assertions about "AI" or "machine learning." Here are some tips for seeing through AI washing: If a company claims their product uses AI, ask for specifics on the techniques used and how they enable intelligent, adaptive behavior. The lack of technical details is a red flag. Ask how it works: Don't take capabilities like natural language processing at face value. Test the product extensively to verify whether supposed AI features actually work as advertised. Validate capabilities: Ask vendors if their AI claims have been validated by independent testing agencies like AI laboratories and universities. The lack of external validation is suspicious. Request third-party evaluation: Examine the qualifications of the vendor's AI team. Do they have recognized experts in AI and advanced degrees in the field? Insufficient in-house talent makes lofty AI assertions doubtful. Check qualifications: Ask how the AI model was trained. Thorough training on extensive, high-quality data is essential for developing systems that work well. Review training approach: Research competitor products making similar AI claims. If others have more modest claims for similar offerings, it may indicate exaggeration. Compare competitors: Ask for specific case studies and examples demonstrating the claimed AI capabilities in real-world applications over time. The lack of solid examples exposes weak claims. Get examples: By probing vendors with these questions and scrutiny, buyers can push past the hype and determine if true AI or mere AI washing is at play. What AI Really Looks Like Artificial intelligence is a continually advancing field with many techniques and applications. There is no single formula, but authentic AI solutions exhibit some key characteristics: The system’s algorithms have the ability to learn from data without explicit programming. Machine learning techniques enable adaptation and improvement. Learning ability: The AI demonstrates an understanding of context and meaning, not just data patterns. For example, comprehending language or analyzing images situationally. Contextual comprehension: The AI can extend its training on particular datasets to unfamiliar data and scenarios it has not directly experienced before. Generalizability: The system can store what it learns over time and apply that learning in the future, rather than just responding based on immediate inputs. Memory: The AI acts proactively to analyze ambiguous, complex problems and delivers solutions based on reasoning and judgment, not just preset rules. Problem-solving: The system can make reasonable predictions and suggestions even with uncertain, incomplete data. AI handles ambiguity well. Uncertainty handling: In many applications, the AI can explain its processes, rationale, and decisions in understandable terms to human users when needed for trust and accountability. Explanation: While the techniques behind AI solutions may differ, these hallmarks indicate that a system incorporates true artificial intelligence that evolves and adapts over time, rather than just simple automation. Understanding these AI standards helps separate legitimate applications from AI washing. The Hype and History of AI Washing AI washing is not a new phenomenon. It follows a similar pattern to previous technological hype, such as the metaverse mania of recent years, the dot-com craze of the late 1990s, and even the airplane hype of the 1920s. During the metaverse mania, companies like Facebook (now Meta) heavily promoted the concept, leading to a surge in mentions of the term by various companies, many of which may not have had serious metaverse ambitions. Similarly, during the dot-com bubble, companies added ".com" or "Internet" to their names to attract investors, resulting in significant stock price increases. In the 1920s, investors were drawn to companies associated with airplanes, even if their actual business had little to do with aviation. One example is Seaboard Airlines, which attracted investor interest despite being a railroad company. These historical examples demonstrate that companies often try to capitalize on the hype surrounding new technologies, even if their actual capabilities or involvement in those areas are limited. AI washing is the latest iteration of this trend, fueled by the excitement and potential of artificial intelligence. Moving Forward With True AI As artificial intelligence advances, AI washing is likely to remain an issue plaguing many industries. To avoid deception and move forward with authentic AI, consumers and businesses should: on the true nature of AI and its techniques like machine learning so they can scrutinize vendor claims knowledgeably. Educate themselves that force vendors to back up AI assertions and not rely on buzzwords. Ask tough questions to evaluate vendor offerings and determine if they constitute true AI versus exaggeration. Consult AI experts within industries to certify legitimate AI and combat false claims. Advocate for auditing standards in research and talent development to drive continued AI innovations that live up to the technology’s true potential. Support investments Ultimately, focusing on the reality of artificial intelligence capabilities, not the hype, will lead to smart adoption of AI that delivers genuine benefits and business value. With scrutiny and standards, AI washing can be recognized so authentic AI advances can flourish.

What Is AI Washing? How to See Through the Hype

For too long, the siren song of data mesh has lured companies into a false sense of security, promising a utopian future of seamless data management. But the harsh reality is sinking in: data mesh isn't the panacea it's cracked up to be. As costs soar, trust in data erodes, and decision-making stalls, it's time to shatter the illusion and explore a new path forward. For many organizations, the allure of data mesh, a decentralized approach to data management, has turned into a frustrating chase. While it promised data democratization and agility, the reality often involves siloed data, soaring costs, and eroded trust in data-driven decision-making. A recent implementing data mesh experienced significant data quality issues. Surveys paint a concerning picture: survey by Unisphere Research, a division of Information Today, Inc., in partnership with ChaosSearch, found that 68% of IT managers Another survey conducted by revealed that struggled with increased complexity and lack of collaboration after adopting data mesh. PwC in 2023 72% of businesses "Data mesh, in theory, sounds great," says Experts are also raising concerns: Zhamak Dehghani, a renowned data management consultant and author of the book 'Data Mesh: Delivering Data-Driven Value in the Age of Big Data.' "However, the reality is that many organizations lack the cultural and organizational maturity to successfully implement it, leading to fragmented data landscapes and inconsistent data practices." The Data Marketplace Data Marketplace vs. Data Mesh: A Comparative Look Feature Data Mesh Data Marketplace Data Ownership Decentralized (domain teams) Decentralized (domain teams own data, but data products are managed by data experts) Data Storage Distributed across domain-specific data products Centralized data lake with isolated access zones Data Governance Loosely coupled, relies on domain teams Robust and centralized, with standardized processes Data Access Open access to all data for domain teams Controlled access based on skills and expertise Data Products Developed and maintained by domain teams Created and managed by data experts based on defined needs Focus Agility and data ownership Focus, collaboration, and data quality Here's how the data marketplace overcomes the challenges of data mesh: By focusing on relevant data and eliminating duplication, the data marketplace minimizes storage and processing expenses. Reduced Costs: Robust governance and centralized data management ensure consistent data standards and quality across the organization. Improved Data Quality: The marketplace fosters communication and collaboration between data producers and consumers, leading to better insights and decision-making. Enhanced Collaboration: While data ownership remains with domain teams, the data marketplace ensures accountability for data products through designated data experts. Clearer Ownership: Data Marketplace Principals Here are my 8 principles for using the data marketplace concept that will ensure your organization leverages the right data at the right time to make the right decisions: - In the marketplace, we keep the data producers owners of the data they produce, but they do not own the data products built on top of it, which I think is a fear thing. Decentralized Data Ownership One of my biggest criticisms about data mesh is the decentralized ownership which requires now a front-end engineer and a product manager to own a data product based on their data, they are required to design and maintain it, and they have zero clue why they do it, or how it’s being used, and if I will be honest, none of the tools they used is actually native to their work. Instead, it just lets them set up the events or APIs calls, sets them with the tools to document what they send, to ingest the data without them touching anything, and from there, lets the experts work with it. This system makes more sense as they only need to worry that the data is correct and reliable and are not required to fix data products gone wrong. -  Yes, please stop storing all the data warehouses! Even if it’s S3 and “cheap,” there is no reason to store so much and process so much data; the reality is that most of this data is just sitting there getting dust. Transactional-based data By decentralizing the data, we allow the owners to decide what they store. With the marketplace, I wish to take a different approach: the analysts, the middle people between the data consumers and producers can see all the data produced in the company, but they can decide what data to buy and sign a “contract” with the producer which describe basically what they will use the data, what are their expectations and what happens in case of a data incident. By doing so, we ensure that the producers understand why they produce the data and what impact it has on the organization, and allow us to clean unused data either into deep storage or into the abyss by deleting it, and the analysts on the other side need to create a value case to why this data is important for them to use. It’s creating a win-win situation that also addresses the missing link between the data producers and analysts on who using the data in the data lake structure, these transactions are logged and stored. - Basic rules must be in the game, what is the metadata required, how and where to document the data ingested, produced, and visualized, what monitors should be in place to ensure quality, how the data contracts are created, and how data anomalies, incidents, and conflicts are handled; by doing, so we ensure we standardized the process. Robust Governance & Standards I see a big need to have 2-3 types of data contracts but enforce that every data created has a schema validation process to ensure quality, and in case of an issue, it’s rejected, and notifications are sent to the owner of it to be aware “Huston we have a problem.” Governance is important, and also, the enforcement of the regulation keeps this thing running with strong work-frames that address the use cases of data and serve as a system for future-proofing the data products. - Let’s talk architecture to make the data marketplace successful by creating three “isolated” levels of data on the data warehouse system; each level is accessible and usable based on the skills and expertise of the individuals and is meant to act as a safe net for data consumption: Isolated Architecture Environments serves as a secure storage for all data, regardless of quality, to ensure sensitive information protection. Think of it as a vault for safeguarding all types of data. Data then undergoes preparation steps like defining sessions, identifying bots, and ensuring compliance before being transformed into a usable format for analysis. This process can be compared to transforming raw ingredients into a ready-to-use meal for a chef. The Raw Data Zone empowers analysts by providing access to the prepared data through aggregated tables. Analysts can utilize exploration tools to query, analyze, and gain insights from the data. This zone, along with the tools it provides, plays a crucial role in enabling analysts to create valuable decision-making dashboards. The Marketplace Data Zone serves as a single source of truth for the entire company. This zone contains rigorously tested, secured, and verified data presented through user-friendly visualizations. Anyone in the company can access and explore this data in a controlled environment, either directly via the provided tools or by extracting it for use in tools like Excel. The Decision-making Zone The data here is deemed the "official" version - any conflicting information from other sources should be considered inaccurate. Data that enters the data lake has one access point where the data is validated, is it what needs to be here, the data is checked for anomalies, and only if it passes the inspection, it will arrive in a raw data landing zone; then from there, the data will have small components that will run and clean for bot traffic, add attribution to the session, or even enrich missing metadata to ensure the data can be used. When it arrives to the data consumers, they will only see the data that was certified pre-calculated, and built, to enable the look and feel of a self-exploration environment without the fear of losing control over the numbers because “Self-serve analytics” is a nice dream but will never become reality due to the level of data literacy in the organizations. - Based on the above and the transaction-based, why in hell should we have a push system where anyone interested can just push data into the lake or a dashboard? Pull System We need to bring back control over the data and have a purpose and a clear why; otherwise, we’re just wasting money on moving data around. I am sure your cloud provider is happy and also convincing you this is cheap if you buy the right saving plans, but in reality, you have nothing to do with the data sitting there collecting dust. Some industry standards estimate that around 87% of the data ingested into the data lake have no clear purpose or use case; think about the amount of money you are spending right now on having it stored. The pull system ensures that only those with purpose arrive at the common areas and are used. Otherwise, it will just become history in a short period saving you a lot of money on wasted accomilated data. - The data domain subject matter experts are the ones who can build data products from the raw data and move it into the marketplace zone, or approve a new data product from the marketplace zone into the decision-making zone. Data Parliaments In the data mesh, one of the biggest issues is that anyone can pull data and build a data product with it, but this sometimes causes cases of data not matching or worse decisions to be made over wrong data. The parliament members can be, for example, responsible only for the marketplace revenue tables; they are the only ones who know how to define the true source of the revenue, how to present it, and also how to handle issues with its quality when things go wrong. On the other side, a parliament can handle the company KPIs table and will know which data in the marketplace is the one true source. Contrary to data mesh, we create a function of data controllers who validate that the marketplace has one true source of data, so each analyst using the marketplace zone knows they are using the right data, and they are the ones that ensure the correct data is carried across the company. This also answers the governance being overwhelmed with so many topics they need to manage and helps them focus on the work-frames and the parliaments on the data products. - This is a vital part if we wish to ensure people in the organization consume the right data and make decisions based on the best available data to them. Data Certification We need to certify the data; the certification process allows us to, on the one side, ensure that we have one, revenue numbers used in the decision-making zone, and also ensure it has the right process behind it from data monitoring to error handling and up to how it should be documented. The certification of the data ensures that everyone knows what the correct number is and has the trust that the data has all the required steps to ensure its reliability and trustworthiness; one of the things I met the most is the missing trust of leaders in the data they consume to make decisions, and this mistrust causes them to doubt the data and ask analysts to check it constantly whenever it doesn’t fit their expectation, wasting a lot of time of the analysts on debugging. But by creating a data certification, we can ensure that, in case of a data incident, the user will know it before it’s being used, and the analysts will know where they need to look to solve it. Basically, creates better trust between the data consumers to the analysts, and in case the data incident is not fixed where the problem is and who is blocking it from being resolved, creating full transparency and accountability over the data. Return on investment! We spoke earlier on transactions and explained the use case. Well, with that, we need to have some system that evaluates the effort invested in creating a data product, the storage, the processing, the analyst time invested, the cost of maintaining it on the one side, and on the other, the possible profits by having the data. ROI - I agree here that not every data source will have it, but I will argue it’s possible :-). By having, for example, the campaign performance on Facebook, we can optimize the campaigns and invest more money in campaigns that drive more profit or cut campaigns that waste money. By doing so, we save and increase the revenue, and it can go into the calculation. Another example is by finance having the AWS costs available, they can find areas to improve savings either by buying saving plans or investigating high costs and possibly reducing them. Another method to evaluate the revenue of the data is by incrementally testing; having this data means you can save or generate X amount of extra money; if not, this is the amount of money lost. The ROI gives everyone in the chain of the data a know what the value of their data is and also the ability to evaluate how to prioritize a data incident with a clear or estimated monetary value next to it. More data does not equal better insights; it just means more time lost on making decisions, more time wasted on bad data, and the bad thing, loss of focus; here is what I am trying to correct with my data marketplace concept. - Focus on the data that make sense for the organization to make decisions. Focus Not all events and all data sources are relevant, and most definitely, having more data does not mean you will be able to make better decisions. Focusing on a small amount of highly relevant KPIs will help the business oriented around what is working or not, will ensure higher focus on the quality of the data, and ensure everyone knows how their work impacted or not the success of the organization. - If you produce data, you need to know WHY you produce it and what is the added value of having it. Without having it in place, you waste money. WHY I don’t know how many times in my life as an analyst I was required to run some super expensive and complicated query that requested me to give some number or shares, but after asking what they were used for or what decisions they drove, I discovered that my three days work was “Just for good to know” and had no impact what so ever on the decision making or actions taken. I am thinking a lot about the wasted time of analysts answering questions that do not serve a real purpose and the amount of time gone on checking the data quality and compatibility, understanding it, and then trying to run the most efficient query, and it all goes with action. - People should be able to self-serve themselves, and no, when I talk about self-serve, I am not talking about this drag and drop that we all fall in love with from Google Analytics. Data usability What I mean is that if you are an analyst or data consumer, you should have the right tools and access to data based on your skills and expertise. I do not wish to see data consumers playing with raw data and trying to drive learning without knowing what can be the results of them using the numbers they encounter; at the same time, I do wish to see it. Even if you create more work-frames around the data mesh, the reality is that when you ask the wrong people to do the right things without the expertise, you increase their anxiety levels to handle situations they are not ready for, and worse, you cause them to lose focus on what their unit purpose is, and for this, you are wasting more growth opportunities. Hiring more data engineers and data product managers won’t solve your problem; training unrelated people to own data products will fail. I call you to try something new, reduce the noise from data increase your focus, and if you want, after you master the data marketplace and the delicate balance between centralized to decentralized, you can go in the direction of data mesh. But all the time you don’t, you risk accumulating more costs in lost revenues and overspending on data the decision is yours! Embracing the data marketplace isn't just about efficiency; it's about empowering all stakeholders in the data ecosystem to harness the true potential of data. By focusing on collaboration, clarity, and purpose, we can unlock the power of data and navigate a path toward sustainable success. Share your thoughts on data management and explore whether the data marketplace could be a viable solution for your organization. Join the conversation!

From Data Chaos to Clarity: The Data Marketplace vs. Data Mesh

TOPTAL, LLC v. DENIS GROSZ Court Filing, retrieved on February 26, 2024, is part of . You can jump to any part in this filing . This part is 7 of 7. HackerNoon’s Legal PDF Series here CERTIFICATE OF ELECTRONIC SERVICE I certify that I am an employee of the Second Judicial District Court of the State of Nevada, in and for the County of Washoe that on the October 2023, I electronically filed the foregoing with the Second Judicial District Court's electronic filing system which will send a notice of electronic filing to the following: JOHN DESMOND, ESQ. KENT ROBISON, ESQ PHILIP MANNELLY, ESQ. ADAM HOSMER-HENNER, ESQ. HANNAH WINSTON, ESQ. BRIAN IRVINE, ESQ. About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on October 11, 2023, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. media.licdn.com

Certificate of Electronic Service for Upload of Toptal v. Denis Grosz Court Filing 

Drone deliveries could be the next big thing if logistics companies find a way to overcome the current hurdles. How long will it take them to figure things out? What Drone Delivery Could Look Like So far, the commercial sector has shown interest in various short-range drone applications. Last-mile delivery is one of the most likely uses for drones in logistics since the technology is better suited for brief trips. While conventional vans can travel incredibly long distances, the cost of deploying them in the last leg of shipping is absurdly high. Same-day delivery is another likely application because drones are small, fast, and lightweight. More importantly, they can typically only hold one package at a time, making them perfect for urgent order fulfillment. The food and grocery delivery surge highlights another potential application for cost-effective drones. While app-based services are popular, companies lose money paying their drivers. For example, Uber only — it lost millions every year prior. started making a profit in 2023 Pros of Using Drones for Package Deliveries There are numerous benefits to using drones for package deliveries. Fast Delivery Autonomous drones could cut delivery times in half. They can deliver lightweight packages because they’re agile and small. A large fleet could easily make thousands of deliveries daily. faster than conventional vehicles Inexpensive Operation Semi-trucks and delivery vans use diesel or gas, which is expensive and fluctuates in price often. On the other hand, drones use electricity, which is relatively inexpensive. In other words, their cost per trip is typically much lower. Higher Efficiency Drones are fast and use little fuel because roadways, traffic lights, congestion, or car accidents don’t hinder them. Instead, they can fly in a semi-direct path toward their target address. They're more efficient since they can achieve a higher delivery quota than trucks. Increased Accessibility Addresses in remote or congested areas can be challenging to get to by truck. Drones can travel to these spots with little trouble, increasing delivery accessibility — and potentially increasing order frequency. Automatic Delivery Autonomous drones pose less of a risk to public safety than self-driving vehicles. At the same time, they offer all of the benefits of automation — they can deliver packages around the clock, never have to take breaks, and won’t suffer from human-error-related mistakes. Cons of Using Drones for Package Deliveries As great as the benefits of drone deliveries are, downsides do exist. Unreliable Unfortunately, modern drones aren’t the most robust technology — a strong gust of wind or unpleasant weather can easily knock them off course. If conditions aren’t ideal, they won’t be able to leave the fulfillment center to make deliveries. Complex Currently, there are few out-of-the-box solutions for commercial drone delivery. Autonomous flight , requiring technology like cloud computing, artificial intelligence, computer vision, and GPS-independent navigation to work. Properly developing and integrating everything can be a highly complex undertaking. is very resource-intensive Involved Logistics companies can only deploy a fleet if they have storage space, chargers, navigation software, insurance, and maintenance. Developing the infrastructure to make drone deliveries is time-consuming and expensive. Expensive Commercial drones are expensive — on average, in 2024. Unfortunately, the price will only drop to $500 by 2028. Logistics companies will have to invest a significant portion of their budget into the upfront expenses alone. a single one costs $520 Error-Prone While conventional vehicles can experience a flat tire or a dead battery, the driver can fix it or call for assistance. If a drone suffers from a technical issue mid-air, it could land in a tree, on a roof or in the middle of a road, irreversibly damaging it and its package. Finding and retrieving it might be impossible if its geolocation features malfunction. Restricted Drones can only carry relatively lightweight packages. They’d have to be much larger to lift bulk or heavy objects — and who wants all that weight flying precariously above their head? Preventable Challenges to Widespread Adoption One of the most preventable challenges to the widespread adoption is drop-off uncertainty. Media often depicts drones gracefully swooping down and gingerly placing a package right where it needs to be. In reality, they need a pre-determined location like a parcel locker. Unoperated drones aren’t capable of the precision that placing a package at the right doorstep requires, so pick-up locations are ideal. However, they have to get relatively low to the ground to make these deliveries, meaning there’s a high likelihood of interference. While logistics companies can hope people remain civil, only some are fond of this novel delivery method. Residents of one town after the tech giant rolled them out. Although they’d face repercussions from destroying or damaging anything, the ensuing legal battles and replacement costs could be high. discussed shooting down Amazon’s drones At the very least, general public hesitation is another challenge halting widespread adoption. According to one survey, feel worried about potential drone accidents. Their concerns aren’t unfounded, either — drones are far more likely to crash than aircraft. nearly three in 10 people The issue of red tape is technically preventable but would be challenging in reality. The Federal Aviation Administration (FAA)’s Part 107 rule between the pilot and their drone at all times. Australia and the United Kingdom have similar regulations. This technicality all but prevents autonomous deliveries. requires a line of sight Moreover, operators must get a remote pilot certificate, register, certify, and pass tests before ever being able to fly — or watch over — a drone. Even if they have all these credentials, they can’t fly above 400 feet or over congested areas. The regulations are strict. While these challenges are all preventable with enough effort, money, and time, they still are massive obstacles to standardizing drone delivery. Unless a pioneering logistics company works to change things, they will remain in the way. Unavoidable Challenges to Widespread Adoption Logistics companies can’t remove every hurdle to adoption. Weather is one of the biggest obstacles — drones can’t fly in hail, fog, rain, sleet, or windy conditions. If it’s too hot out, their lifting capacity drops drastically, and their batteries drain much faster in the cold. Essentially, they are relatively useless unless it’s mild out. Their range is another longstanding challenge. Currently, autonomous drones can only get so far — a few miles, to be precise. For logistics companies to make an adequate return on investment, they’d need to build a micro-fulfillment center next to every major city. Obstacles are another frequently overlooked — and unavoidable — challenge. While autonomous delivery drones are often equipped with computer vision, artificial intelligence, and sensors, they won’t be able to recognize objects as well as humans. In other words, they have a much higher chance of running into power lines, low-flying aircraft, trees, birds, and people. What the Future of Drone Delivery Will Look Like While drone technology has potential, it’s facing some significant hurdles. Even if the FAA’s regulations suddenly disappeared, their success isn’t guaranteed. In fact, companies have yet to make much progress even though the FAA has already pulled back. Its 2021 rule change , enabling them to fly at night and over moving vehicles without a waiver. loosened commercial drone restrictions Still, the winds of change are definitely blowing. In 2021, registered their drones. More and more logistics enterprises are recognizing the potential of this technology. roughly 1.3 million commercial pilots Crucially, the general public’s awareness and acceptance of this technology is shifting. According to one study, to use package delivery drones if the option existed. While logistical adoption of the drone market, it likely won’t be an option at checkout for years. seven in 10 people would prefer is the fastest-growing portion Drones Could Be the Future of Logistics While delivery drones might not become a thing for years — possibly even decades — it’s clear that logistics brands are hard at work trying to make this technology happen. It could soon become the standard.

How Close Are We to Widespread Drone Deliveries?

আমরা ড্রোন ডেলিভারি ব্যাপকভাবে কতটা কাছাকাছি?

Wie nah sind wir an einer weit verbreiteten Verbreitung von Drohnenlieferungen?

¿Qué tan cerca estamos de las entregas generalizadas de drones?

À quel point sommes-nous proches des livraisons généralisées par drones ?

हम व्यापक ड्रोन डिलीवरी के कितने करीब हैं?

ドローン配送の普及はどのくらい近づいているのでしょうか?

드론 배송이 널리 확산되는 데 얼마나 가까이 다가왔나요?

Quão perto estamos das entregas generalizadas de drones?

Насколько мы близки к распространению доставки дронами?

Yaygın Drone Teslimatlarına Ne Kadar Yakınız?

Chúng ta đã tiến gần đến việc giao hàng bằng máy bay không người lái rộng rãi đến mức nào?

我们距离广泛的无人机送货还有多远？

581 Stories To Learn About Non Fiction

Authors: (1) Davide Viviano, Department of Economics, Harvard University; (2) Lihua Lei, Graduate School of Business, Stanford University; (3) Guido Imbens, Graduate School of Business and Department of Economics, Stanford University; (4) Brian Karrer, FAIR, Meta; (5) Okke Schrijvers, Meta Central Applied Science; (6) Liang Shi, Meta Central Applied Science. Table of Links Abstract & Introduction Setup (When) should you cluster? Choosing the cluster design Empirical illustration and numerical studies Recommendations for practice References A) Notation B) Endogenous peer effects C) Proofs References Alberto Abadie, Susan Athey, Guido W Imbens, and Jeffrey Wooldridge. When should you adjust standard errors for clustering? Technical report, National Bureau of Economic Research, 2017. Vivi Alatas, Abhijit Banerjee, Rema Hanna, Benjamin A Olken, and Julia Tobias. Targeting the poor: evidence from a field experiment in indonesia. American Economic Review, 102 (4):1206–1240, 2012. Peter M Aronow and Cyrus Samii. Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics, 11(4):1912–1947, 2017. Susan Athey, Dean Eckles, and Guido W Imbens. Exact p-values for network interference. Journal of the American Statistical Association, 113(521):230–240, 2018. Sarah Baird, J Aislinn Bohren, Craig McIntosh, and Berk Ozler. Optimal design of experi- ¨ ments in the presence of interference. Review of Economics and Statistics, 100(5):844–860, 2018. G Basse and A Feller. Analyzing multilevel experiments in the presence of peer effects. arXiv preprint arXiv, 1608, 2016. Guillaume W Basse and Edoardo M Airoldi. Limitations of design-based causal inference and a/b testing under arbitrary and network interference. Sociological Methodology, 48(1): 136–151, 2018a. Guillaume W Basse and Edoardo M Airoldi. Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika, 105(4):849–858, 2018b. Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008. Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0):0, 2000. Yann Bramoull´e, Habiba Djebbari, and Bernard Fortin. Identification of peer effects through social networks. Journal of econometrics, 150(1):41–55, 2009. Emily Breza, Arun G Chandrasekhar, Tyler H McCormick, and Mengjie Pan. Using aggregated relational data to feasibly identify network structure without network data. American Economic Review, 110(8):2454–2484, 2020. Jing Cai, Alain De Janvry, and Elisabeth Sadoulet. Social networks and the decision to insure. American Economic Journal: Applied Economics, 7(2):81–108, 2015. Aureo De Paula, Seth Richards-Shubik, and Elie Tamer. Identifying preferences in networks ´ with bounded degree. Econometrica, 86(1):263–288, 2018. Dean Eckles, Brian Karrer, and Johan Ugander. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference, 5(1), 2017. Dennis Egger, Johannes Haushofer, Edward Miguel, Paul Niehaus, and Michael Walker. General equilibrium effects of cash transfers: experimental evidence from kenya. Econometrica, 90(6):2603–2643, 2022. Stefan Faridani and Paul Niehaus. Rate-optimal linear estimation of average global effects. arXiv preprint arXiv:2209.14181, 2022. Paul Goldsmith-Pinkham and Guido W Imbens. Social networks and the identification of peer effects. Journal of Business & Economic Statistics, 31(3):253–264, 2013. Keisuke Hirano and Jack R Porter. Asymptotics for statistical treatment rules. Econometrica, 77(5):1683–1701, 2009. David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. Reducing interference bias in online marketplace pricing experiments. arXiv preprint arXiv:2004.12489, 2020. Daniel Hong, Hyunwoo Lee, and Alex Wei. Optimal solutions and ranks in the max-cut sdp. arXiv preprint arXiv:2109.02238, 2021. Michael G Hudgens and M Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):832–842, 2008. Ravi Jagadeesan, Natesh S Pillai, and Alexander Volfovsky. Designs for estimating the treatment effect in networks with interference. 2020. Igor Kabiljo, Brian Karrer, Mayank Pundir, Sergey Pupyrev, Alon Shalita, Alessandro Presta, and Yaroslav Akhremtsev. Social hash partitioner: a scalable distributed hypergraph partitioner. arXiv preprint arXiv:1707.06665, 2017. Hyunseung Kang and Guido Imbens. Peer encouragement designs in causal inference with partial interference and identification of local average network effects. arXiv preprint arXiv:1609.04464, 2016. Brian Karrer, Liang Shi, Monica Bhole, Matt Goldman, Tyrone Palmer, Charlie Gelman, Mikael Konutgan, and Feng Sun. Network experimentation at scale. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3106– 3116, 2021. Toru Kitagawa and Guanyi Wang. Who should get vaccinated? individualized allocation of vaccines over sir network. Journal of Econometrics, 2021. Lihua Lei. Unified ℓ2→∞ eigenspace perturbation theory for symmetric random matrices. arXiv preprint arXiv:1909.04798, 2019. Lihua Lei, Xiaodong Li, and Xingmei Lou. Consistency of spectral clustering on hierarchical stochastic block models. arXiv preprint arXiv:2004.14531, 2020. Michael P Leung. Treatment and spillover effects under network interference. Review of Economics and Statistics, 102(2):368–380, 2020. Michael P Leung. Causal inference under approximate neighborhood interference. Econometrica, 90(1):267–293, 2022. Michael P Leung. Network cluster-robust inference. Econometrica, 91(2):641–667, 2023. Shuangning Li and Stefan Wager. Random graph asymptotics for treatment effect estimation under network interference. The Annals of Statistics, 50(4):2334–2358, 2022. Tianxi Li, Lihua Lei, Sharmodeep Bhattacharyya, Koen Van den Berge, Purnamrita Sarkar, Peter J Bickel, and Elizaveta Levina. Hierarchical community detection by recursive partitioning. Journal of the American Statistical Association, 117(538):951–968, 2022. Shuyang Ling and Thomas Strohmer. Certifying global optimality of graph cuts via semidefinite relaxation: A performance guarantee for spectral clustering. Foundations of Computational Mathematics, 20(3):367–421, 2020. Elena Manresa. Estimating the structure of social interactions using panel data. Unpublished Manuscript. CEMFI, Madrid, 2013. Charles F Manski. Identification of endogenous social effects: The reflection problem. The review of economic studies, 60(3):531–542, 1993. Charles F Manski. Identification of treatment response with social interactions. The Econometrics Journal, 16(1):S1–S23, 2013. Karthik Muralidharan and Paul Niehaus. Experimentation at scale. Journal of Economic Perspectives, 31(4):103–24, 2017. Mark EJ Newman. Community detection and graph partitioning. Europhysics Letters, 103 (2):28003, 2013a. Mark EJ Newman. Spectral methods for community detection and graph partitioning. Physical Review E, 88(4):042822, 2013b. Elizabeth L Ogburn, Oleg Sofrygin, Ivan Diaz, and Mark J van der Laan. Causal inference for social network data. arXiv preprint arXiv:1705.08527, 2017. Jean Pouget-Abadie. Dealing with Interference on Experimentation Platforms. PhD thesis, 2018. Fredrik S¨avje, Peter Aronow, and Michael Hudgens. Average treatment effects in the presence of unknown interference. Annals of statistics, 49(2):673, 2021. Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000. Betsy Sinclair, Margaret McConnell, and Donald P Green. Detecting spillover effects: Design and analysis of multilevel experiments. American Journal of Political Science, 56(4):1055– 1069, 2012. Sean J Taylor and Dean Eckles. Randomized experiments to detect and estimate social influence in networks. In Complex Spreading Phenomena in Social Systems, pages 289– 322. Springer, 2018. Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329–337. ACM, 2013. Davide Viviano. Experimental design under network interference. arXiv preprint arXiv:2003.08421, 2020a. Davide Viviano. Policy choice in experiments with unknown interference. arXiv preprint arXiv:2011.08174, 2020b. Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17:395–416, 2007. Jeffrey M Wooldridge. Cluster-sample methods in applied econometrics. American Economic Review, 93(2):133–138, 2003. This paper is under CC 1.0 license. available on arxiv

Causal Clustering: Design of Cluster Experiments Under Network Interference: References

96 Stories To Learn About Data Engineering

89 Stories To Learn About Big Data Analytics

361 Stories To Learn About Big Data

This paper is under CC BY 4.0 DEED license. available on arxiv Authors: (1) Ehsan Toreini, University of Surrey, UK; (2) Maryam Mehrnezhad, Royal Holloway University of London; (3) Aad Van Moorsel, Birmingham University. Table of Links Abstract & Introduction Background and Related Work FaaS Architecture Implementation and Performance Analysis Conclusion Acknowledgment and References Acknowledgment The authors in this project have been funded by UK EPSRC grant “FinTrust: Trust Engineering for the Financial Industry” under grant number EP/R033595/1, and UK EPSRC grant “AGENCY: Assuring Citizen Agency in a World with Complex Online Harms” under grant EP/W032481/1 and PETRAS National Centre of Excellence for IoT Systems Cybersecurity, which has been funded by the UK EPSRC under grant number EP/S035362/1. References [1] Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. Auditing black-box models for indirect influence. Knowledge and Information Systems, 54(1):95–122, 2018. [2] Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. arXiv preprint arXiv:1704.01701, 2017. [3] Muhammad Ajmal Azad, Samiran Bag, Simon Parkinson, and Feng Hao. Trustvote: Privacy-preserving node ranking in vehicular networks. IEEE Internet of Things Journal, 6(4):5878–5891, 2018. [4] Olivier Baudron, Pierre-Alain Fouque, David Pointcheval, Jacques Stern, and Guillaume Poupard. Practical multi-candidate election system. In Proceedings of the twentieth annual ACM symposium on Principles of distributed computing, pages 274–283, 2001. [5] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153–163, 2017. [6] Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 797–806. ACM, 2017. [7] Ronald Cramer, Ivan Damg˚ard, and Berry Schoenmakers. Proofs of partial knowledge and simplified design of witness hiding protocols. In Annual International Cryptology Conference, pages 174–187. Springer, 1994. [8] Craig E. Carroll and Rowena Olegario. Pathways to corporate accountability: Corporate reputation and its alternatives. Journal of Business Ethics, 163(2):173–181, 2020. [9] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259–268. ACM, 2015. [10] Amos Fiat and Adi Shamir. How to prove yourself: Practical solutions to identification and signature problems. In Conference on the theory and application of cryptographic techniques, pages 186–194. Springer, 1986. [11] Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. On the (im) possibility of fairness. arXiv preprint arXiv:1609.07236, 2016. [12] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1–42, 2018. [13] Feng Hao, Matthew N Kreeger, Brian Randell, Dylan Clarke, Siamak F Shahandashti, and Peter Hyun-Jeen Lee. Every vote counts: Ensuring integrity in large-scale electronic voting. In 2014 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE 14), 2014. [14] Feng Hao, Peter YA Ryan, and Piotr Zieli´nski. Anonymous voting by two-round public discussion. IET Information Security, 4(2):62–67, 2010. [15] Moritz Hardt, Eric Price, Nati Srebro, and others. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pages 3315–3323, 2016. [16] Hui Hu, Yijun Liu, Zhen Wang, and Chao Lan. A distributed fair machine learning framework with private demographic data protection. In 2019 IEEE International Conference on Data Mining (ICDM), pages 1102–1107. IEEE, 2019. [17] Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi-Malvajerdi, and Jonathan Ullman. Differentially private fair learning. In International Conference on Machine Learning, pages 3000–3008. PMLR, 2019. [18] Jonathan Katz and Yehuda Lindell. Introduction to modern cryptography. CRC press, 2014. [19] Niki Kilbertus, Adria Gascon, Matt Kusner, Michael Veale, Krishna P Gummadi, and Adrian Weller. Blind justice: Fairness with encrypted sensitive attributes. In 35th International Conference on Machine Learning, pages 2630–2639. PMLR, 2018. [20] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016), 9(1), 2016. [21] Jiacheng Liu, Fei Yu, and Lixin Song. A systematic investigation on the research publications that have used the medical expenditure panel survey (MEPS) data through a bibliometrics approach. Library Hi Tech, 2020. [22] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. [23] Arwa Mahdawi. It’s not just A-levels – algorithms have a nightmarish new power over our lives. The Guardian, 2020. [24] Arvind Narayanan. Translation tutorial: 21 fairness definitions and their politics. In Proc. Conf. Fairness Accountability Transp., New York, USA, 2018. [25] Cecilia Panigutti, Alan Perotti, Andr´e Panisson, Paolo Bajardi, and Dino Pedreschi. Fairlens: Auditing black-box clinical decision support systems. Information Processing & Management, 58(5):102657, 2021. [26] Cecilia Panigutti, Alan Perotti, and Dino Pedreschi. Doctor xai: an ontology-based approach to black-box sequential data classification explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 629–639, 2020. [27] Saerom Park, Seongmin Kim, and Yeon-sup Lim. Fairness audit of machine learning models with confidential computing. In Proceedings of the ACM Web Conference 2022, pages 3488–3499, 2022. [28] Reuters. Amazon ditched AI recruiting tool that favored men for technical jobs. The Guardian, 2018. [29] Shahar Segal, Yossi Adi, Benny Pinkas, Carsten Baum, Chaya Ganesh, and Joseph Keshet. Fairness in the eyes of the data: Certifying machinelearning models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 926–935, 2021. [30] Keng Siau and Weiyu Wang. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal, 31(2):47– 53, 2018. [31] Douglas Robert Stinson and Maura Paterson. Cryptography: theory and practice. CRC press, 2018. [32] Ehsan Toreini, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad van Moorsel. The relationship between trust in AI and trustworthy machine learning technologies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 272–283, 2020. [33] Michael Veale and Reuben Binns. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society, 4(2):2053951717743530, 2017. [34] Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research, 18(1):2357–2393, 2017. [35] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.

Privacy-preserving Computation of Fairness for ML Systems: Acknowledgement & References

This paper is under CC BY 4.0 DEED license. available on arxiv Authors: (1) Ehsan Toreini, University of Surrey, UK; (2) Maryam Mehrnezhad, Royal Holloway University of London; (3) Aad Van Moorsel, Birmingham University. Table of Links Abstract & Introduction Background and Related Work FaaS Architecture Implementation and Performance Analysis Conclusion Acknowledgment and References 2 Background and Related Work One of the benefits of auditing ML-based products relates to trust. Trust and trustworthiness (in socio-technical terms) are complicated matters. Toreini et. al [32] proposed a framework for trustworthiness technologies in AI–solutions based on existing social frameworks on trust (i.e. demonstration of Ability, Benevolence and Integrity, a.k.a. ABI and ABI+ frameworks) and technological trustworthiness [30]. They comprehensively reviewed the policy documents on regulating AI and the existing technical literature and derived any ML–based Table 1: Features of FaaS and comparison with other privacy–oriented fair ML proposals (support: full: ✓, partial: ✙, none: ✗) solution needs to demonstrate fairness, explainability, auditability, and safety and security to establish social trust. When using AI solutions, one cannot be assured of the fairness of such systems without trusting the reputation of the technology provider (e.g., datasets and ML models). It is commonly believed that leading tech companies do not make mistake in their implementation [8]; however, in practice, we often witness that such products indeed suffer from bias in ML [28, 23]. 2.1 Fairness Metrics There exist several fairness definitions in the literature. Designing a fair algorithm requires measuring and assessment of fairness. Researchers have worked on formalising fairness for a long time. Narayanan [24] lists at least 21 different fairness definitions in the literature and this number is growing, e.g., [5, 6]. Fairness is typically expressed as discrimination in relation to data features. These features for which discrimination may happen are known as Protected Attributes (PAs) or sensitive attributes. These include, but are not limited to, ethnicity, gender, age, scholarity, nationality, religion and socio-economic group. The majority of fairness definitions expresses fairness in relation to PAs. In this paper, we consider Group Fairness, which refers to a family of definitions, all of which consider the performance of a model on the population groups level. The fairness definitions in this group are focused on keeping decisions consistent across groups and are relevant to both disparate treatment and disparate impact notions, as defined in [9, 15]. In this paper, we will focus on the computations based on the above three fairness metrics. For this computation, the auditor requires to have access to the three pieces of information for each elements in the dataset: (1) the sensitive group membership (binary value for A demonstrating if a sample does or does not belong to a group with PAs) (2) the actual labelling of the sample (binary value for Y ) (3) the predicted label of the sample (binary value for Yˆ ). The ML system transfers this information for each sample from their test set. Then, the auditor uses this information to compute the above fairness metrics. Note that while we consider the above metrics for our protocol and proof-ofconcept implementation in next sections, our core architecture is independent of metrics, and the metric set can be replaced by other metrics too (Fig. 1). 2.2 Auditing ML Models for Fairness The existing research in fair ML normally assumes the computation of the fairness metric to be done locally by the ML system, with full access to the data, including the private attributes [15, 6, 5]. However, there is a lack of verifiability and independence in these approaches which will not necessarily lead to trustworthiness. To increase trust in the ML products, the providers might make the trained model self–explaining (aka transparent or explainable). There is also the transparent–by–design approach [12, 2, 34]. While this approach has its benefits, it is both model–specific and scenario–specific [25]; thus it cannot be generalised. There is also no trusted authority to verify such claims and explanations. Moreover, in reality, the trained model, datasets and feature extraction mechanisms are company assets. Once exposed, it can make them vulnerable to the competitors. Another approach to provide transparency to the fairness implementation comes through the black–box auditing, also known as adhoc [12, 22, 26]. In this way, the model is trained and audited for different purposes [1]. This solution is similar to tax auditing and financial ledgers where accountants verify and ensure these calculations are legitimate.However, unlike the well–established body of certifications and qualifications for accountants in tax auditing and financial ledgers; there does not exist any established processes and resources for fairness computation in AI and ML. The concept of a service that calculates fairness has been proposed before, e.g., in [33]. The authors introduced an architecture to delegate the computation of fairness to a trusted third party that acts as a guarantor of its algorithmic fairness. In this model, the fairness service is trusted both by the ML system and the other stakeholders (e.g. users and activists). In particular, the ML system must trust the service to maintain the privacy of data and secrecy of its model, whilst revealing to the trusted third party the algorithm outcome, sensitive input data and even inner parameters of the model. This is a big assumption to trust that the third party would not misuse the information and hence the leakage of data and model information is not a threat. To address these limitations, Kilbertdus et al. [19] proposed a system known as ‘blind justice’, which utilises multi–party computation protocols to enforce fairness into the ML model. Their proposal considers three groups of participants: User (data owner), Model (ML model owner) and the Regulator (that enforces a fairness metric). These three groups collaborate with each in order to train a fair ML model using a federated learning approach [35]. The outcome is a fair model that is trained with the participation of these three groups in a privacy-preserving way. They only provide a limited degree of verifiability in which the trained model is cryptographically certified after training and each of the participants can make sure if the algorithm has not been modified. It should be noted that since they operate in the training stage of the ML pipeline, their approach is highly dependent on the implementation details of the ML model itself. Jagielski et al. [17] proposed a differential privacy approach in order to train a fair model. Similarly, Hu et al. [16] used a distributed approach to fair learning with only demographic information. Segal et al. [29] used similar cryptographic primitives but took a more holistic approach towards the computation and verification of fairness. They proposed a data-centric approach in which the verifier challenges a trained model via an encrypted and digitally certified dataset using merkle tree and other cryptographic primitives. Furthermore, the regulator will certify the model is fair based on the data received from the clients and a set of dataset provided to the model. Their approach does not provide universal verifiability as the regulator is the only party involved in the computation of fairness. More recently, Park et al. [27] proposed a Trusted Execution Environment (TEE) for the secure computation of fairness. Their proposal requires special hardware components which are cryptographically secure and provide enough guarantees and verification for the correct execution of the code. The previous research generally has integrated fairness into their ML algorithms; therefore, such algorithms should be redesigned to use another fairness metric set. As it can be seen in Table 1, FaaS is the only work which is independent of the ML model and fairness metric with universal verifiability, and hence, can be used as a service.

Background and Related Work on Privacy-preserving Computation of Fairness for ML Systems 

Craig Wright Is NOT Satoshi Nakamoto

ক্রেগ রাইট সাতোশি নাকামোতো নন

Craig Wright ist NICHT Satoshi Nakamoto

Craig Wright NO es Satoshi Nakamoto

Craig Wright n'est PAS Satoshi Nakamoto

क्रेग राइट सातोशी नाकामोटो नहीं हैं

クレイグ・ライトはサトシ・ナカモトではありません

크레이그 라이트는 나카모토 사토시가 아니다

Craig Wright NÃO é Satoshi Nakamoto

Крейг Райт НЕ Сатоши Накамото

Craig Wright Satoshi Nakamoto DEĞİLDİR

Craig Wright KHÔNG phải là Satoshi Nakamoto

克雷格·赖特不是中本聪

Blockchain technology has revolutionized the digital landscape, presenting both challenges and opportunities. The blockchain skills gap is a significant issue due to blockchain technology's specialized and unique aspects, which differ greatly from traditional web development. Unlike traditional web development, technology contains specialized aspects that demand a distinct skill set. blockchain The blockchain skills gap is a significant and growing concern, marked by a shortage of skilled professionals and a steep learning curve for developers aspiring to enter this innovative field. Addressing this gap is not only crucial for individual career growth but also essential for the sustainable advancement and wider adoption of blockchain technology across various sectors. The author acknowledges a vested interest in the organization(s) highlighted in this story. However, the views expressed within are delivered impartially and without bias. Disclosure: What are the Challenges? While the originates from multiple challenges, a few key obstacles discourage mainstream web developers. Firstly, they need to learn niche programming languages, like Solidity, which is very different from popular languages such as JavaScript. This high barrier to entry often discourages those already comfortable with their web development expertise. blockchain skills gap Additionally, blockchain development diverges significantly from conventional web development in terms of data handling and application structure. Developers must adapt to a decentralized ledger system, a stark contrast to the centralized models they are accustomed to. This shift necessitates a new paradigm in thinking about data storage, management, and security. Concepts unique to blockchain, such as immutability, consensus mechanisms, and peer-to-peer interactions, are seldom encountered in traditional web development environments. The complexity of varied platforms and protocols discourages those accustomed to universal web standards. The fragmentation across different blockchain ecosystems hampers developer familiarity and makes onboarding more difficult. There is also the difficulty of writing flawless for high-value transactions. With smart contracts being immutable, ensuring correctness from the onset is critical and introduces risks unfamiliar to traditional web developers. smart contracts Finally, developer-friendly tools to simplify blockchain builds are lacking. Resources to eliminate the underlying complexity have so far been limited in availability. While improvements have been made, there are still gaps in tools for debugging, monitoring, and optimizing blockchain applications. Agoric's Approach to Bridging the Gap Agoric, a blockchain platform utilizing the Cosmos proof-of-stake protocol, offers an approach to make blockchain technology more accessible to mainstream developers. This approach involves strategies and tools that might be more familiar to developers not specializing in blockchain, potentially easing their transition into this field. The protocol is intended to bridge the gap between conventional software development practices and the specific demands of blockchain technology, potentially broadening the pool of developers engaged in the blockchain ecosystem. Agoric is a platform in the blockchain sector featuring its native token, BLD, which is used for transactions within its ecosystem. The platform integrates several features designed to appeal to developers. One of the key aspects of Agoric is its use of JavaScript for writing smart contracts. JavaScript's widespread adoption makes it a familiar choice for many developers, potentially easing the transition into blockchain development. Regarding security, Agoric utilizes a secure subset of JavaScript known as HardenedJS. This is designed to safeguard the integrity of smart contracts by preventing malicious code interference. Additionally, Agoric introduces a smart contract framework named Zoe, which aims to ensure safety. This means users receive what they expect from a contract or are fully refunded. Agoric also provides a variety of tools for developers. These include reusable components like fungible tokens, automated market makers, and loan structures. These tools are intended to simplify developing and deploying decentralized applications (dApps). Agoric supports connections with other IBC-enabled chains, such as Ethereum and Chainlink, offering developers access to broader data and services within the blockchain ecosystem. The platform has seen adoption among mainstream developers, illustrated by the development of applications such as Inter Protocol. This decentralized stable token is designed to remain equivalent to the US dollar and is supported by collateral assets from various chains. Inter Protocol uses Agoric's components for liquidity and stability and benefits from the JavaScript smart contracts for ease of code writing and maintenance. Educational Resources and Training Programs The educational resources for learning blockchain development have grown extensively, providing various platforms and formats to suit different learning styles and needs. A key part of this educational landscape is the wealth of online courses on Coursera, Udemy, and edX platforms. These courses cover a broad range of introductory topics in blockchain to advanced subjects like smart contract development, cryptocurrency, and . decentralized application (DApp) creation They typically feature video lectures, interactive coding exercises, and real-world project examples, offering a thorough learning resource for beginners and experienced developers. Besides online courses, specialized workshops and bootcamps are organized by different organizations and technology institutes. These are usually more hands-on and intensive, offering an immersive experience in blockchain development. They aim to provide practical skills and knowledge through project-based learning and mentorship, ideal for those who prefer a structured, interactive learning environment and direct guidance from industry experts. Another important aspect of blockchain education is the resources blockchain platforms provide. Many blockchain companies and communities offer documentation, tutorials, and developer guides to assist developers interested in working with their platforms. These resources are crucial for staying current with the latest technologies and best practices in the fast-changing blockchain field. The role of structured training programs in closing the blockchain skills gap is crucial. While self-learning through online resources is valuable, structured programs offer a more comprehensive and guided learning approach. They present a curriculum that progressively builds skills, ensuring learners acquire a deep and practical understanding of blockchain technology. This is particularly important considering the complexity and subtleties of blockchain development, where a strong foundation is essential for effective and secure application development. Structured training programs also often include elements of peer interaction and networking, which are vital for professional growth in the blockchain area. Engaging with other learners and industry professionals can lead to new insights, collaborations, and career opportunities. Additionally, many of these programs provide certifications or credentials upon completion, beneficial for professionals seeking to certify their skills and advance their careers in the blockchain sector. Community Support and Collaborative Learning Developer communities are indispensable in helping web developers transition to blockchain technology. These communities exist in various forms, including online forums, social media groups, and local meetups, and they offer a platform for exchanging knowledge, receiving mentorship, and gaining support. Online forums like Stack Overflow, Reddit, and specialized blockchain development forums are lively with activity, providing a space where beginners can ask questions, share experiences, and get advice from more seasoned blockchain developers. These forums are more than just problem-solving venues; they are also places for sharing the latest industry news, tools, and best practices, keeping everyone updated on the fast-paced developments in the blockchain world. Local meetups and hackathons play a slightly different but equally crucial role. They bring blockchain enthusiasts and professionals together in a more personal and interactive environment. These events offer opportunities for developers to network, collaborate on projects, and learn from each other in real time. They often include workshops, presentations, and coding sessions, which are excellent for hands-on learning and understanding the practical side of blockchain development. Collaborative projects are another fundamental aspect of community support and learning in the blockchain field. , in particular, provide a unique opportunity for practical experience and peer learning. By contributing to these projects, developers can work on real-world applications, applying and testing their skills collaboratively. This helps improve technical skills and provides insight into the team-based development projects common in professional blockchain development. Open-source projects Participation in these collaborative projects also exposes developers to various facets of blockchain development, from coding and testing to documentation and user interface design. The peer review process in open-source collaboration is an invaluable learning tool, offering insights into coding standards, best practices, and innovative solutions. Furthermore, these projects often involve diverse contributors from around the world, bringing in various perspectives and approaches, which enriches the overall learning experience.

The Digital Leap: Addressing the Growing Skills Gap in Blockchain Technology

How can businesses harness the best that an AI-powered future offers? By implementing five key strategies, they can effectively embrace AI-driven communication and unlock its potential to enhance internal operations and customer experiences. Whether it’s instant chat features, AI-powered answers, call transcription and summaries, or multi-lingual content creation, of business owners believe ChatGPT will enhance customer and internal communications. 97% The use cases for AI-driven chatbots are extensive, but scaling AI poses data reliability and security challenges. While many companies are keen to adopt AI across multiple departments, only roughly 50% of projects transition from pilot to implementation, according to a . IT teams need help meeting numerous business needs at once whilst staying ahead of the crowd. Gartner survey The promise of AI is immense. Let's look at the proactive strategies businesses can implement to embrace AI-driven communication successfully. 1. Connect AI algorithms to the business value proposition The AI revolution reminds us of the saying, “data is the new oil.” Unrefined oil is of little use, yet when properly refined, it creates tremendous value in powering planes, cars, and our energy needs. The same applies to the data used in your AI: Refined data creates powerful tools. What can business leaders do to distill high-quality data? Rather than overloading ChatGPT or other AI tools with company data to handle every use case, data teams need to be clear on what constitutes the most pertinent data for their intended AI use case. Data needs to be filtered as part of the development of the AI model. Bad data INPUT means bad AI OUTPUT.  Think about what is the RIGHT data and its relationship to the value proposition you are building. For example, in sales, AI can help agents handle buyer objections. Business leaders need to pair customer product requirements, pricing, and support needs with the appropriate objection-handling responses. Generative AI can provide real-time prompts for agents to handle sales objections and also enable sales agents to store and rate responses. Separate reinforcement learning algorithms will process the data, refine the model's training, and improve future accuracy. 2. Support customers with effective prompting AI NLP (Natural Language Processing) input fields like ChatGPT are freeform; you can ask it anything. The model statistically selects the most likely output based on the surrounding message context. But the specific reasoning needs to be clarified, and depending on the clarity of the prompt, the tool can produce very different results. To increase response accuracy, always provide context, and be specific in tone, structure, and output type. For instance, say you are building a tool for customers to obtain car quotes. A customer asks the chatbot: ‘Can you give me three quotes for an off-road car for a family of four.’ Before providing three quotes, set the model to ask several clarifying questions; what is your budget? Do you have a preferred model? Do you require any special features? The tool can refine the search with user-facing prompts to better understand the customer’s needs and ultimately deliver better results. The idea is to drive specific intent. By prompting the customer, you enable AI to serve the best possible customer experience. 3. Define the level of risk in the use of AI What is the level of risk? Is the use case where you want to implement AI subject to data protection, existing regulation, or legal compliance? For example, AI-powered personalized medicine has great potential for improving patient outcomes and overall efficiency. However, feeding confidential documents, individual health data, proprietary IP, and other protected information into AI models require data security methods to safeguard information. Of the companies in the Gartner survey who stated they had experienced data breaches, 60% reported an internal party compromised the data. Data theft is a real risk in any business today, and the use of AI expands data accessibility. Companies with highly confidential data must weigh the costs and complications of securing their data effectively. In every scenario, ensuring company-wide staff training on the risks of your AI use cases is vital. It is worth noting that not using AI is also a business risk, especially when believe advanced AI will be as important as the creation of the Internet. A recent survey noted that a quarter of generative AI users found answers faster with it than compared with a traditional search engine. 48% of respondents 4. Create a data governance strategy Using generative AI to improve communication in business can be a powerful tool, but it also brings several challenges, especially concerning data privacy, security, and ethical considerations. This is where creating a data governance strategy becomes crucial. Data teams and business leaders must establish governance policies and create data visibility with detailed authorization lineage and consent policies. If using third-party data sources, ensure due diligence on the authorized use of this data. A data governance strategy helps establish guidelines and protocols to ensure compliance with data protection regulations (e.g., GDPR, CCPA), privacy laws, and industry standards. Adhering to industry standards such as SOC 2 compliance, which audits and certifies the quality and reliability of data security and privacy controls, will help ensure the integrity of client data. 5. Keep the human-in-the-loop According to Forbes Advisor, 77% of workers expressed concern about job loss due to AI adoption. However, AI can only function with sufficient human oversight. AI offers immense potential, but human involvement remains crucial, hence the human-in-the-loop (HITL) concept. The HITL serves a variety of vital services. Foremost, humans can review and curate the training data for the chatbot. This process involves filtering out inappropriate or misleading content and ensuring the data is diverse and representative. In this regard, human oversight ensures the ethical and responsible use of AI. Moreover, the HITL can monitor AI responses in real-time. This allows authorized team members to intervene if a chatbot provides inaccurate or harmful responses, or if the conversation goes off-topic or becomes inappropriate. A human's provision of context, empathy, and critical thinking is vital. Human moderation not only helps maintain a high level of quality and ensures that the chatbot adheres to ethical guidelines, but by moderating responses, they can improve the data set that the AI model is working from. Summing up Businesses across every industry are seeing the impact of AI-driven communication. To drive positive results, they must carefully evaluate their value proposition, data sensitivity, the tools’ limitations, and governance strategies to mitigate them. And, always remember, overseeing success is the human-in-the-loop.

5 Key Strategies for Effective AI-Driven Communication in Business Operations

In the rapidly evolving AI landscape, the focus has long been on technological advancements and breakthroughs ( , anyone?). We want AIs that think like humans, and we want them now, damn it. However, this relentless pursuit of innovation often overlooks a critical aspect: . e/acc the environmental impact As Gen. AI tools like chatGPT , their energy consumption and carbon footprint have emerged as pressing issues. Between 2017 and 2021, the electricity used by major cloud computing providers like Meta, Amazon, Microsoft, and Google , . become increasingly integral to our daily lives more than doubled contributing significantly to greenhouse gas emissions This sets the stage for a new study on the matter, titled “ ” (great title), and written by , (Both Hugging Face employees), and (of Carnegie Mellon University & Allen Institute for AI). Power Hungry Processing: Watts Driving the Cost of AI Deployment? Sasha Luccioni Yacine Jernite Emma Strubell The paper offers an in-depth analysis of the energy consumption and carbon emissions associated with AI, specifically during the model (i.e., deployment/public use) phase. The authors compared the environmental costs of both task-specific (fine-tuned for single tasks) and multi-purpose (trained for multiple tasks) AI models. Their methodology included testing 88 models across ten different tasks using various datasets and measuring the energy required and carbon emitted for 1000 inferences. inference In short, the study finds that multi-purpose generative architectures like ChatGPT and MidJourney are significantly more energy-intensive than task-specific systems across various tasks. This discrepancy raises concerns about the growing trend of deploying versatile yet resource-heavy AI systems without fully considering their environmental impact. More below. Key takeaways from the study Multi-purpose Generative AI models like ChatGPT and MidJourney, while versatile, are significantly more energy and carbon-intensive compared to task-specific models. Image-related AI tasks consume more energy than text-based tasks (which seems obvious in hindsight). There is an uneven trade-off between the benefits of multi-purpose AI systems and their environmental costs. The study’s authors urge a reassessment of deploying multi-purpose models for tasks where smaller, task-specific models could suffice. What do we do with that information? Both companies, governments, and consumers have a clear path ahead. They just need to walk it. For companies : Researchers from Google and UC Berkeley have highlighted the concept of the “4Ms” of machine learning, which includes Model, Machine, Mechanization, and Map. Optimizing these factors can significantly reduce the carbon emissions and energy use of ML systems. Optimize AI’s “4Ms” For governments : Current AI regulations often overlook environmental sustainability. However, the EU AI Act now classifies environmental harm as a high-risk area, requiring transparency and disclosure measures from AI model providers. As I’ve often argued, the rest of the world should follow suit. The compliance of data centers, in particular, should be carefully assessed. Meanwhile, governments should encourage research and development of more energy-efficient AI architectures. Develop better policies and regulations : There is no current process for certifying AI model sustainability. However, a rating system similar to the Leadership in Energy and Environmental Design (LEED) in construction could be developed for AI models, considering factors like model architecture and data center location. Support standardized reporting/certifications For consumers for well-defined tasks to reduce energy consumption. For example, if you need a language translation tool, choose one that specializes in translations rather than a multi-purpose model. If you can recycle, you can do that too. Prioritize task-specific AI models : Consider joining or supporting organizations and initiatives that advocate for responsible AI usage and environmental sustainability. Your voice can contribute to collective efforts in holding companies and governments accountable. Advocate for change Too soon to draw conclusions While enriching, the paper does have a few limitations. Primarily, its focus is on a limited set of AI models and tasks, which might not represent the entire spectrum of AI deployments. Additionally, there is a need for more comprehensive data on the energy consumption of different AI models throughout their lifecycle, including training and deployment phases. As researchers often like to say… “ .” more research is needed The study by Luccioni serves as a crucial wake-up call to the AI community, highlighting the need to balance technological advancements with environmental sustainability. et al. It underscores the importance of conscious decision-making in AI deployments, keeping in mind their long-term ecological implications. As we continue to harness the power of AI, it’s imperative to do so with a keen awareness of our environmental responsibilities, fostering a future where innovation and sustainability coexist harmoniously. Good luck out there. Also published . here

AI's Environmental Impact: Balancing Technological Advancements with Sustainability

You can jump to any section of the National Aeronautics Science & Technology Priorities here. Strategic Priorities The U.S. Government will maintain U.S. leadership in aeronautics by fostering a vibrant, secure, dynamic R&D aeronautics community and workforce that includes government, private industry, and academia. To this end, the U.S. government will pursue three priorities: I. Achieving Sustainable Aviation Consistent with the U.S. 2021 Aviation Climate Action Plan, the U.S. Government will achieve its goal of net-zero emissions of greenhouse gases (GHGs) for civil aviation by 2050. It will employ simultaneous lines-of-effort to achieve a future in which GHG emissions will fall to net-zero using cost-effective, efficient approaches. These approaches include: (1) new aircraft and engine technologies to greatly improve environmental performance; (2) operational improvements to substantially reduce energy use and environmental impacts; (3) widespread domestic production and adoption of Sustainable Aviation Fuels (SAF); and (4) exploration and implementation of new and advanced fuels. The development and introduction of new aircraft and engines is critical to increasing energy efficiency and reducing pollutants as demand for air transportation continues to grow. The U.S. Government will coordinate with aviation stakeholders to support the adoption of these new technologies by airlines. This new generation of efficient low-emissions aircraft will directly support the sustainability of aviation, while increasing the competitiveness of U.S. aviation products globally. As demand for aviation services grows, new operational technologies and practices are required to avoid congestion and inefficient operations. Opportunities remain for continued investment in infrastructure and air traffic management tools, and improvements in all phases of flight (surface, takeoff, cruise, and landing operations) to reduce the consumption of fuel and yield efficiencies in regional and nationwide operations. The U.S. Government will accelerate the development, testing, and certification of SAF, with a focus on “drop-in” jet fuels, which are fuels created from renewable resources that mimic the chemistry of petroleum jet fuel and can be used in today’s aircraft and engines. Consistent with the SAF Grand Challenge Roadmap [4], the United States will increase the production of SAF to at least 3 billion gallons per year by 2030, a necessary first step toward the SAF industry providing 35 billion gallons of SAF by 2050. The widespread development of SAF will ensure that key sectors, including aviation, play significant roles in the transition to a net-zero carbon future, ensuring environmental justice, job retention, and economic growth across the United States. Efforts to incentivize the use of SAF are already underway through the Inflation Reduction Act (IRA) (Pub. L 117-169)[5]. The U.S. Government will also phase out 100 low lead aviation gasoline (100LL avgas) known to cause harmful lead emissions. Significant strides have been made in developing, testing, and certifying leadfree replacement fuels. The U.S. Government is committed to continuing to chart a path to lead-free fuel for piston aircraft through the Eliminate Aviation Gasoline Lead Emissions (EAGLE) Initiative. Finally, the U.S. Government will improve the scientific understanding of the climate impacts of nonCO2 aircraft emissions, such as aviation-induced cloudiness. It also will continue to lead development of technologies and science-based decision support tools to help mitigate emissions of conventional pollutants and these non-CO2 climate impacts. [1] Federal Aviation Administration. “The Economic Impact U.S. Civil Aviation: 2020”. https://www.faa.gov/sites/faa.gov/files/2022-08/2022-APL-038%202022_economic%20impact_report.pdf. Released August 2022 [2] International Trade Administration. “Leading Economic Indicators Aerospace Industry”. Accessed February 21, 2023. https://www.trade.gov/leading-economic-indicators-aerospace-industry [3] Federal Aviation Administration. “United States 2021 Aviation Climate Action Plan”. https://www.faa.gov/sites/faa.gov/files/2021-11/Aviation_Climate_Action_Plan.pdf. Released November 2021 p4] Department of Energy, Office of Energy Efficiency and Renewable Energy. “Sustainable Aviation Fuel Grand Challenge Roadmap: Flight Plan for Sustainable Aviation Fuel”. September 2022. https://www.energy.gov/sites/default/files/2022-09/beto-saf-gc-roadmap-report-sept-2022.pdf This was originally published in March 2023 on whitehouse.gov. This content has been broken-up into bite sized sections, each with unique headlines and AI-generated lead images, in line with the provisions of Creative Commons 3.0.

A Green Horizon: U.S. Charts Course for Net-Zero Aviation Emissions by 2050

একটি সবুজ দিগন্ত: 2050 সালের মধ্যে নেট-জিরো এভিয়েশন নির্গমনের জন্য ইউএস চার্ট কোর্স

Ein grüner Horizont: Die USA zeichnen den Kurs für Netto-Null-Luftfahrtemissionen bis 2050 auf

Un horizonte verde: Estados Unidos traza el curso para lograr emisiones netas cero en la aviación para 2050

Un horizon vert : les États-Unis établissent un cap pour atteindre zéro émission nette de l’aviation d’ici 2050

एक हरित क्षितिज: 2050 तक शुद्ध-शून्य विमानन उत्सर्जन के लिए यू.एस. चार्ट कोर्स

グリーンホライズン: 2050 年までに航空機排出量実質ゼロに向けた米国の計画

녹색 지평선: 2050년까지 순 제로 항공 배출을 위한 미국 차트 코스

Um Horizonte Verde: Curso de gráficos dos EUA para emissões líquidas zero da aviação até 2050

Зеленый горизонт: курс США по нулевым авиационным выбросам к 2050 году

Yeşil Ufuk: 2050'ye Kadar Net Sıfır Havacılık Emisyonları için ABD Grafikleri Kursu

Chân trời xanh: Khóa học về biểu đồ của Hoa Kỳ về lượng phát thải hàng không ròng bằng 0 vào năm 2050

绿色地平线：美国到 2050 年实现航空净零排放的路线图

90 Stories To Learn About Mozilla

Stories, News & Mentions about CERTIFYOS

Certifyos’s Stories on HackerNoon