Alation is the leader in enterprise data intelligence solutions, including data search & discovery, data governance, data stewardship, analytics, and digital transformation. Alation’s initial offering dominates the data catalog market. Thanks to its powerful Behavioral Analysis Engine, inbuilt collaboration capabilities, and open interfaces, Alation combines machine learning with human insight to successfully tackle even the most demanding challenges in data and metadata management. Nearly 450 enterprises drive data culture, improve decision-making, and realize business outcomes with Alation, including AbbVie, American Family Insurance, Autozone, Cisco, Draft Kings, Exelon, Fifth Third Bank, Finnair, Munich Re, NASDAQ, Parexel, Pfizer, Salesforce, Virgin Australia, and Vistaprint. Headquartered in Silicon Valley, Alation has been named to Inc. Magazine’s Best Workplaces list three times, is a 2022 UK’s Best Workplaces™ for Women, and recognized as a 2022 UK’s Best Workplaces™ in Tech. The company is backed by leading venture capitalists, including Blackstone, Costanoa, Databricks Ventures, Data Collective, Dell Technologies Capital, Hewlett Packard Enterprise, Icon, ISAI Cap, Riverwood Capital, Salesforce Ventures, Sanabil Investments, Sapphire, Snowflake Ventures, Thoma Bravo, and Union Grove.

Alation

And Why Governance is the Gordian Knot to all Your Business Problems "Data governance is a measure of a company's control over its data" Data governance is a data management concept. It is a measure of the control a company has over its data. This control can be achieved through high-quality data, visibility on data pipelines, actionable rights management, and clear accountability. Data governance encompasses the people, processes, and tools required to create consistent and proper handling of a company's data. By consistent and proper handling of data, I mean ensure availability, usability, consistency, understandability, data integrity, and data security. The most comprehensive governance model — say, for a global bank — will have a robust data-governance council (often with C-suite leaders involved) to drive it; a high degree of automation with metadata recorded in an enterprise dictionary or data catalog; data lineage traced back to the source for many data elements; and a broader domain scope with ongoing prioritization as enterprise needs shift. Good data governance and privacy model is a mix of people, processes, and software. Data Governance has a Direct Business Impact Data governance isn't just that rusty process that companies have to deploy in order to comply with the regulation. Of course, part of it is a legal obligation, and thank god, but clean governance can have high business outcomes. Here are the main goals of data governance: When Did Data Governance Become a Thing? Timeline and key milestones in the space. ‍ For the past twenty years, the challenge around data has been to build an infrastructure to store and consume data efficiently and at scale. Producing data has become cheaper and easier over the years with the emergence of cloud data warehouses and transformation tools like dbt. Access to data has been democratized thanks to BI tools with BI tools like Looker, Tableau, or Metabase. Now, building nice dashboards is the new norm in Ops and Marketing team. This gave rise to a new problem: decentralized, untrustworthy & irrelevant data and dashboards. Even the most data-driven companies still struggle to get value from data - up to 73% of all enterprise data goes unused. ‍ → 1990-2010: the emergence of the 1st regulation on data privacy In the 1970s, the first data protection regulation in the world was vetted in Hessen, Germany. Since then, data regulation has kept increasing. The 1990's mark the first regulations regarding data privacy with the EU directive on data protection. Yet, compliance with regulation really became a worldwide challenge in the second half of the 2010s with the emergence of GDPR, HIPAA, and other regional regulations on personal data privacy. These first regulations drove data governance for large enterprises. This created an urgency to build tools to handle these new requirements. ‍ →  2010 - 2020: 1st tools to comply with the regulation. C-level realizes data governance becomes a strategic advantage to drive business value. With the increasing complexity of data resources/processes on the one hand and the first fines for GDPR infringement on the other, companies started to build regulatory compliance processes. The 1st pieces of software to organize Governance and Privacy were born with companies like Alation and Collibra. The challenge is simple: enforce traceability across the various data infrastructure in the company. Data governance was then a privilege of enterprise-level companies, the only ones able to afford those tools. On-premise data storage makes it expensive to deploy these software. Indeed, companies like Alation and Collibra had to deploy technology specialists on the field to connect the data to their software. The first version of data governance tools aims at collecting and referencing data resources across the organization's departments. There were several forces at play in this period. It became easier to collect data, cheaper to store it, simpler to analyze it. This led to a Cambrian explosion of the number of data resources. As a result, large companies struggled to have visibility over the work done with data. Data was decentralized, untrustworthy & irrelevant. This chaos brought a new strategic dimension to data governance. More than a compliance obligation, data governance became a key lever to bring about business value. ‍ → 2020+: Towards an automated and actionable data governance With the standardization of the cloud data stack, the paradigm changed. It is easier to connect to the data infrastructure and gather metadata. Where it took 6 months to deploy a data governance tool on a multitude of siloed on-premise data centers in 2012, it can take up to 10 minutes in 2021 on the modern data stack (for example: Snowflake, Looker, and DBT). This gave rise to new challenges: automatization and collaboration. Data governance on excel means maintaining manually 100+ fields, on thousands of tables and dashboards. This is impossible. Data governance with a non-automated tool means maintaining 10+ fields on thousands of tables: this is time-consuming. Doing data governance with a fully automated tool means maintaining 1 or 2 fields only on thousands of tables (literally table and column/field description). For that last part of manual work, you want to leverage the community. Prioritize work based on data consumption (high documentation SLA for popular resources) and democratize usage through a friendly UX. Additionally, you want that data governance tool to be integrated into the rest of the data stack. Define something once and find it everywhere: whether this is a table definition, a tag, a KPI, a dashboard, access rights, or data quality results. ‍ Data Governance Challenges Are Not the Same for Everyone Diverse governance's use-cases based on industry needs and company size ‍ There are two main drivers for data governance programs: Level of regulation needed in the industry Data regulation pushes the minimum bar of data governance processes higher. It requires businesses to add controls, reporting, and documentation. This is a need to ensure transparency over sometimes unclear processes. Level of complexity of the data assets Having strong governance becomes increasingly important with the exponential growth of data resources, tools, and people in a company. The level of complexity increases with the scope of business operations (number of lines of business and geographies covered), the velocity of data creation, or the level of automation (decision-making, processes) based on data. ‍ How Do You Set Up Good Data Governance and Privacy Model? Several bricks are needed to enforce data management. Data Architecture (Storage, Modeling, Visualization) Before even talking about data governance, a company needs the basis: a good infrastructure to begin with. Based on business needs and the company's data maturity, the nature of the data architecture can change a lot. Regarding storage, do you go for on-premise or cloud? Data warehouse or a data lake? Regarding modeling: Spark or DBT? in a data warehouse or BI tool? Real-time or batch? Regarding visualization: do you allow anyone to build dashboards or data teams only? Etc. Search and Discovery The first level of any data governance strategy is making sure relevant people can find the relevant datasets to do their analysis or build their AI model. If you don't implement this step, companies end up with a lot of questions on Slack and useless meetings with the engineering teams. The company ends up with a lot of duplicate tables, analyses, and dashboards. It takes valuable time to engineering resources that are needed to perform the next steps. Metadata and Documentation Once you can efficiently find the data, you need to understand it quickly in order to assess if it is going to be useful. For example, you are looking at a dataset called "active_users_revenue_2021".  There is a column "payment." Is this column in € or $? Has it been refreshed this morning, last week, or last year? Does it contain all the data on active users or just the ones in Europe? If I remove a column, will this break important dashboards for the marketing or finance team? etc. Data Quality Now that you have data stored in scalable infrastructure that everyone can find and understand, you need to trust that what is inside is of high quality. This is why so many data observability and reliability tools were born in the last five years. Data observability is the general concept of using automated monitoring, alerting, and triaging to eliminate data downtime. The two main approaches to data quality are: declarative (manually define thresholds and behavior) or ML-driven (detecting sudden changes in distribution). Security and Access Rights Some data might be more private or strategic than others. Let's say you are a bank; you don't want to give access to the transaction logs to anyone in the company. You need to define access rights, and managing them efficiently can quickly become a struggle as the number and type of people working with data grow. Sometimes, you want to give access to someone for a specific mission and for anything else. What happens when one of your employees is in the finance department but moves to marketing? You need to manage these rights thoroughly and efficiently. Compliance and Regulation This one is self-explanatory. You need to list all assets, report on personal information and usage to comply with regulations. For now, only enterprise companies are targeted by regulators; it is just a question of time before smaller companies start receiving fines. ‍ Where Does Data Governance Fit in the Modern Data Stack? Data governance brings trust from the raw data sources to domain expert dashboards. ‍ The typical data flow is the following : You collect data from various sources from your business. It can be product logs, marketing, and website data, payment and sales logs, etc. You extract that information with tools like Fivetran, Stitch, or Airbyte. You then store this data in a data warehouse (Snowflake, Redshift, Bigquery, Firebolt, to name the most popular). The data warehouse is both a place to store and transform your data to refine it. The new trending transformation layer for the past 3 years is DBT. It enables to perform data transformation in SQL within the data warehouse while implementing software engineering good practices. At last, the transformation helps you build your "data mart," the golden standard in terms of refined data. The visualization brick helps domain experts visualize this gold-level data to share insights throughout the whole organization. These steps are happening on different tools with a high level of abstraction. It is hard to keep a bird's eye view of what happens under the hood. This is what data governance is bringing to the table. You can see how the data flows, where the pipeline breaks, where risks lie, where to put your energy as a data manager, etc. Also published at: . https://www.castordoc.com/blog/what-is-data-governance-and-privacy

Why Businesses Need Data Governance

As the Director of Data Engineering at (India’s leading fulfilment platform for digital commerce), I’m surrounded by a huge amount of data. Over 1.2 TB per day, to be exact. Delhivery Delhivery fulfills a million packages a day, 365 days a year. Its 24 automated sort centres, 85+ fulfillment centres, 70 hubs, 3,000+ direct delivery centres, 7,500+ partner centres, 15,000+ vehicles and 40,000+ team members run smoothly thanks to a vast network of IoT devices. There are nearly 60,000 data events and messages coming in and going out of our pipelines per second. With this much data, it’s probably no surprise that data discovery and organization are a big challenge. We finally found our dream data cataloging solution, but it was no simple task. At Delhivery, we started our journey with a data catalog in 2019. Over the next year and a half, we considered and invested in several different types of solutions. We evaluated traditional enterprise-focused data catalogs (e.g. Alation, Collibra, and Waterline), built our own catalog with Atlas and Amundsen, and later adopted the modern SaaS unified data workspace, Atlan. In this blog post, I’m unpacking all of my learnings as a lead engineer on this project in the hopes that it helps other data practitioners and leaders who are facing similar challenges and want to successfully implement a modern data catalog. Why Delhivery desperately needed a data cataloging solution As Delhivery has grown over the past decade, the scale and complexity of its data have grown even faster. Earlier in its history, Delhivery generated 1 TB of data per quarter. Now we’re generating 1.2 TB . per day That data is organized and processed by hundreds of microservices, which means that ownership over our data is distributed across different teams. Delhivery started with a monolithic system, but we took the call to start forking services about 4 years later as the business scaled exponentially. Teams soon started building their own microservices, motivated by a desire to make data-driven decisions. Everyone wanted to find and access specific data, so they’d reach out to several developers and ask for help. Later on, we realized that developers were becoming a bottleneck, and we needed a Google Search–style way for anyone to find data through a common portal. However, finding data wasn’t the only issue. Once teams got hold of the data they wanted, they struggled to understand it. There wasn’t a clear way to organize our data and add important information and business context. This quickly became clear throughout our onboarding process — the typical time to onboard a new team member was 1-2 months, but this process eventually grew to 3-4 months as Delhivery and its data kept growing. By 2019, we realized we desperately needed a data cataloging solution, one where people could navigate through all our data, look for whatever they need, check what a data asset looks like, build a better understanding of other domains or teams within the company, and even add their own info or context about a data asset. Step 1: Evaluating available commercial data catalog solutions in the market We started off our search for a data catalog by evaluating commercial products like , Waterline (now called ) and . We dove deep into their specs and compared them on two main criteria: Alation Lumada Colibra : Did the product have all the features we needed? Features : What was the total cost of ownership (i.e. the purchase price plus other costs like set-up or operations)? Did it fit with our budget? TCO Buying one of these products would have been the simplest fix, but unfortunately, we couldn’t find the right solution. Each one was either missing non-negotiable features (such as seeing a data preview or querying data) or the TCO was just too high for us (due to expensive set-up, licensing and professional service fees). We didn’t want to settle for something that wasn’t quite right, since setting up a data catalog is a huge commitment. So we realized we needed to build and customise our own internal solution. Step 2: Building our first data catalog with Apache Atlas and Amundsen To set up our first data catalog, we kicked off an internal project with two developers. They worked full-time for seven months to develop and customise a data cataloging solution based on open-source software. After researching different options, we decided to use as the backbone of our metadata management framework. Apache Atlas We chose Atlas mainly because of its available features. As a bonus, we also liked that it was widely used, actively in development, backed by Apache, and supported by the entire Apache community. However, as we started implementing Atlas, we realised we had a problem. Atlas is great for developers, but it has a steep learning curve for non-technical team members. We found its interface and language was highly technical, which made it difficult to onboard product managers or operations people who needed to understand our data. In short, Atlas had the features we needed, but it wasn’t right for our end users. To solve this, we decided to bring in . Started at Lyft and advertised as “Google search for data”, Amundsen is a data discovery service that became the search interface for our Atlas backend. Amundsen Step 3: Developing missing features on top of Amundsen and Atlas As we implemented and tested our open-source data catalog, we realized that Atlas and Amundsen still didn’t have all the features we needed. So we started modifying our solution and developing new features on top of it. Here are a few of the features that we built for our internal data catalog: Searching on columns, rather than the default option of searching on tables Better search organisation, based on custom criteria like popularity or relevance Displaying whether columns that appear in search have already been cleaned Improving how columns on a data asset’s detail page are organised to improve relevance Showcasing data quality with Markdown and APIs Our internal Atlas + Amundsen catalog Step 4: Rolling out version 1 of our data catalog to the end users After selecting Atlas and Amundsen, setting them up, and developing our own features on top of them, we finally finished our first data catalog. We rolled it out to the company, but we quickly struggled with getting people to use it. Initially, team members were excited to use the new data catalog. But after logging in a few times, they just didn’t return. It didn’t become a habit, something that they would use continuously. When we asked people what was missing, they all pointed to the user experience. The key problem was that they felt like they weren’t able to grab hold of the catalog and integrate it into their daily work. We had created our catalog with two developers, no UI or UX designers involved. We thought that was fine since the design and layout had already been fully thoughtout by Amundsen. We underestimated the importance of user experience. It plays a crucial role in driving data culture, and our solution couldn’t win without creating an experience that people wanted to use every day. Step 5: Back to the drawing board... After seven months of development, we weren’t much farther than where we started. Even though we had invested so much time and energy in our open-source data catalog, it wasn’t right. Time to go back to the drawing board and figure out our next move… After some thinking and a new roadmap, we realised that fixing our current solution wouldn’t be easy. User experience was key, so we couldn’t succeed without lots of hard work on our interface. We would need a full product team — UI/UX designers, product managers, and engineers — to revise our data catalog so our colleagues would actually use it. At the same time, we started researching commercial options. Yes, we had looked into these products before, but the data catalog market had seen a ton of innovation in the 18 months since our earlier research. So we started our search again, finding and evaluating the latest commercial data cataloguing products. That’s when we found Atlan. Step 6: Choosing Atlan as our unified data workspace is a modern data workspace that excels at data cataloging and discovery, but it is much more than just a traditional data catalog. In our research, we found it to be the best solution for our company. Atlan The Atlan team actually started as a data team themselves, and they built Atlan internally for their work. This has driven their collaboration and user experience–focused approach, making it closer to a . Github or Figma-like experience for data Atlan not only had all the dream features we were looking for, but the were extremely intuitive. Built around a Google-style search, Atlan was designed to be user-friendly for technical and non-technical users alike. Its collaboration-first approach — e.g. the ability to share data assets as links, or its integrations with Slack — was also very promising. When we gave Atlan to team members, they were able to get onboarded and start driving things very quickly. user interface and experience We also liked that Atlan has a fundamentally open philosophy. It was built on open-source (Apache Atlas and Ranger) with an open API backbone. This gave us the ability to build on top of Atlan and customise the product for our end users — bringing together the best of open-source with a polished, off-the-shelf user experience. When we had compared our roadmap with commercial products, we realised that the time we had already sunk into our open-source solution didn’t matter. It would take six or seven people and up to two years to build what Atlan gave us out of the box. We needed a solution on day zero, not in a year or two. As counterintuitive as it seemed, we realised that buying a data cataloging product would be a lower TCO than continuing to build our own. We set aside our internal catalog and said a big yes to Atlan. We also decided to go with Atlan because we realised that we needed a partner who was an expert in data discovery. This would let us rely on their expertise as we rolled out our solution within the company. We were happy to let them be the experts in data cataloging — we had plenty of our own business problems to tackle. The workshop sessions that Atlan conducted during our Proof of Concept also helped us feel sure that we were partnering with the right solution and team. These continuous workshops helped us onboard new team members, dive deep into advanced features, and maintain traction as team members started documenting data and filling our new metadata stores. Atlan’s metadata glossary, which helps users document and understand data assets. Some final thoughts on our build-vs-buy adventure I’ve been on this wild data cataloging journey for the past year and a half. There have been plenty of ups and downs, but I’m excited about where we’ve ended up. I think we’ve finally found the right platform to make our data self-service, discoverable and clear for our data-driven teams. The other day, someone asked me what I’d do differently if I was to start all over again. I realised I had two big learnings. Figure out the end consumers of your data — whether they’re product managers, engineers or operations people — and their daily use cases. What data-driven decisions are they making? How do they find data, and what information do they need about it? How can you remove frustrating bottlenecks like reaching out to developers? Focusing on your users’ needs upfront can save you from shelving a functional but poorly designed solution like us. Remember, if your end users are not developers, then ease of use and intuitiveness is key. First, user experience is key. Instead of waiting until data discovery becomes a problem, it’s much better to start on day one. This lets you scale your process and solution as your company and data scale, rather than having to shoehorn a new product into years of established workflows. Second, start early with data discovery and cataloging. Author: Akash Deep Verma, Director of Data Engineering Image credit: Header photo by CHUTTERSNAP on Unsplash . Also published on: https://humansofdata.atlan.com/2021/05/delhivery-learning-data-catalog/

Build vs Buy: What We Learned by Implementing a Data Catalog

Top 30 Business Intelligence Consulting Companies

The long awaited of this series has arrived. It has taken me a significant amount of time longer to put together than I had anticipated. Now the wait is over, here it is… Part 2 of 3 This list contains companies working on artificial intelligence and machine learning products primarily for business use, non-specific to any industry. Industry specific AI will be the final part of this series. Rise of artificial intelligence is rapidly taking over the world, especially countries like Anguilla; a small island in the Eastern Caribbean with the use of their .ai top level domain for companies within the AI and robotics industry. But in all seriousness, AI is without a doubt accelerating much faster than is probably known and much more established within our daily lives than most would realise. Conducting this research has demonstrated sheer scale of the landscape and progress of the artificial intelligence industry. “Already humans and AI are co-evolving and no one has paid attention to this yet…” “When we search on Google, it’s an AI deciding what we should see. When a dating site matches two people together there’s a machine learning algorithm… that is getting people together, that then have babies. And so in effect you have this machine learning algorithm that is breeding humans.”— Sam Altman, discussing AI . here In the case of the companies listed below, business and AI are co-evolving. With so many websites to go through and dissect there may be some descriptions or names that need changing. Please let me know if that’s the case or if you think there are companies I should add to the list. Shoutout to me , and I’ll get on to it! here Artificial Intelligence for businesses (enterprise intelligence): 🔉 Audio — language transcription and recognition — ? — transcribes insights from phone calls, video footage, and online — discover more customer and employee conversations** **— makes sound searchable — critical insights about your customers conversations — building blocks to add messaging, voice, and video to web+apps Capio Clover Intelligence Deepgram Gridspace MindMeld — advanced AI to power conversational interface Nexidia — turns customer interactions into valuable insights Pop Up Archive TalkIQ Twilio 📊 Business Intelligence & Analytics — optimise and autonomously IT and business operations** ** —behavioural AI for IoT — a suite of intelligent applications for enterprise — a range of products to improve enterprise products** ** — automates repetitive tasks within your business — making complex data science easy for enterprise Arago/HIRO Arimo Ayasdi DataRobot Dataminr — discovers events and breaking information before the news — helps you to answer questions about your business — a smarter Salesforce — adds intelligent decision making to web and mobile apps — helps you index, search, visualise and analyse your data — is a framework for temporal pattern recognition and prediction — to transform raw data into useful information automatically — helps you monitor server bugs Electra by Lore Einstein Fuzzy AI Logz.io NXT AI Paxata Poweredby.ai Sundown UBIX Ruths.ai — helps you do more with your data — a marketplace for analytics — captures, categorises, and extracts key information from all your data — fast data insights using search — solves big data better, faster and at less cost — find hidden value in complex human and business ecosystems — helps to optimise pricing Exchange.ai Owl.ai AnswerRocket iSeek.ai Ecosystem.AI Prix 🍎 Core AI — a common API for many algorithms, functions and models — workbench for neural networks — advanced industry specific ML for the enterprise — advanced machine learning for enterprise — advanced machine learning for enterprise — open source machine learning and deep learning platform — advanced machine learning for enterprise — deep learning as computational system — advanced machine learning for enterprise — advanced machine learning for enterprise — range of financial, ecommerce and digital marketing AI products — open-source deep learning and ETL for enterprise on the JVM — advanced machine learning for enterprise — log Analysis for enterprises Algorithmia Arya CognitiveScale Digital Reasoning Fluid AI H2O.ai Loop AI Labs Nervana Petuum Scaled Inference Sentient Skymind Vicarious Loom Systems PipelineAI — solves the problem of production ML and AI at scale — building AI using neuroscience Ogma 📋 Data Capture — marketplace to automate simple processes — automate the discovery of objects at scale — automatically prepare any data for immediate & compliant use — helps structure data from social media and blog — automatically extract web pages as structured data — extract data from almost any website — training data, image annotation and more for enterprise** ** — tools for operations team to automate business processes Amazon Mechanical Turk CrowdAI Datalogue DataSift Diffbot Import.io Playment WorkFusion 🔬 Data Science — single platform for all predictive use cases — training data, image annotation for machine learning teams — data science platform for prototype, deploy, and run at scale — enterprise data science platform for R&D and production** ** — helps you learn, work, and play with machine learning models — makes data science teams more productive — helps DS teams put machine learning models into production — a platform to build, test, and deploy AI algorithms** **— research engine, capable of discovering complex patterns in data — makes data unification of data silos possible — helps put data into useful structures for analysis — allows data scientists to deploy and update predictive models rapidly — automate the writing of reports, websites, emails, articles and more BigML CrowdFlower Dataiku DataScience Domino Data Lab — platform for collaborating, building and deploying — makes DS accessible to analysts with OpenSource algorithms Exploratory Kaggle RapidMiner Seldon SherlockML Spark Tamr Trifacta Yhat Yseop 💻 Development — detects business incidents — develop more adaptive, trusted and programmable AI models — helps predict project timelines — adds intelligent decision making to web and mobile apps — connecting projects with the right team — augments your coding environment with web available knowledge — deep learning platform for prediction and personalisation — makes developing chatbots for your business easy — make your bot smarter, by helping it self learn — rapidly web and mobile app testing — increase server uptime and predict downtime — project management and chat software that’s easy for teams AnOdot Bonsai Deckard.ai Fuzzy.ai Gigster Kite Layer 6 AI Morph Ozz RainforestQA SignifAI Turtle Improve.ai — automatically optimises app content, design, pricing etc. — gesture recognition for developers — trains deep learning algorithms to learn like human brain — abstracts away the complexity of ML libraries like TensorFlow for more effective management of AI models — helps with decision making for designers, engineers and leaders — helps analyse algorithm Gesture.ai Cognitive Toolkit Bonsai Tangle Imandra 📂 Internal Data — helps you work together, improve productivity, and data indexing** ** — helps employees with tasks and access data using micro apps — better website analytics Alation Cycorp — a range of different AI enterprise products — takes the pain of cluster management away to focus on DS — helps predict project timelines — insight into online trends and other text analytics tools — AI platform for business Databricks Deckard.ai Gavagi IBM Watson Kyndi — helps knowledge workers process vast amounts of information — SaaS AI for risk management and operations — makes your business software smarter oneFactor Probot Sapho Sofia eContext — structure for unstructured data — create intelligent interactions between people, devices, and data — deep insights into customers and internal data —automate decision-making at scale Hayley RelativeInsight Rainbird 🤖 Machine Learning —No-code end-to-end AutoML tool that takes you from data to predictions in minutes* *__ — a range of different AI enterprise products — helps you focus your digital ad on people you want to target — analyse data in the form of images, speech, text and video — now apart of the Uber AI Labs — can do menial task work, saving time for employees — platform to unite siloed data for better recommendations — improves machine learning models 100x faster PI.Exchange — develop more adaptive, trusted and programmable AI models__ Bonsai Cycorp Datacratic deepsense.io Geometric Intelligence HyperScience Nara Logics SigOpt Amazon Machine Learn — ML-as-a-service, amongst other things — import predictive models and scale infinitely to answer existential questions Providence 📡 Sensor (IoT/IIoT) — platform delivers real-time operational insights for industry — learns about shoppers behaviour in your store — helps to unify application development and data science — sensor analytics solution for businesses — machine learning for manufacturing process optimisation — a range of workflow optimisation products for fuel and industry — helps you develop, deploy, and operate industrial apps — helps renewable energy companies utilise their data better — manufacturing analytics — automates data engineering for data science — add a voice assistant to your connected product — platform to manage the development for your IoT applications — a predictive platform for major industries — smart building management for commercial buildings Alluvium Black C3 IoT KONUX Imubit Maana Predix Planet OS Sight Machine Sentenai Snips ThingWorx Uptake Verdigris 📚 Text Analysis/Generation — creates summaries from your text and information in real-time — extract meaning from your text and visuals — text analytics and mining which works without any training** **—automatic knowledge acquisition from text — build profile and understand your users better — scaleable text analytics software — capture, measure, and act on customer feedback — scalable API to automate text classification — interprets your data more useful information — helps you find errors and inaccuracies in documents — free open-source library natural of language processing in Python — automates information extraction, management and analysis — turn text into insights — helps improve how your job ads are written — automate the writing of reports, websites, emails, articles and more Agolo AYLIEN Compreno Cortical.io — advanced language processing fido.ai IntroSpect by Lore Lexalytics Luminoso MonkeyLearn Narrative Science Qeep spaCy Salient Stride Textio Yseop 😎 Vision — add instant text capture functionality to mobile apps and more! — automated drones with vision and diagnosis capabilities — analyses subtle facial expressions to identify human emotions — makes every camera in the world smart — helps cameras to detect analyse the video feed — finds defects in wind turbines — extracts and transforms data from handwritten and typed forms — helps you to organise media libraries — visual analysis for medical and transport industry — image detection for a range of uses and industries — brand and face recognition — makes satellite imagery useful — the first intelligent cloud video platform** ** — computer vision for automative and human detection — matches similar products and helps to suggests combinations — facial recognition — uses facial recognition to help you decided on job candidates — medical data analysis and interpretation — recognises different objects and things — dedicated brand recognition for social networks — satellite image analysis — deep-learning based computer vision platform — planet monitoring and analysis using satellite imagery — satellite image analysis — is an eye and emotion tracking platform — understands and describes video content — image analysis primarily for industrial purposes ABBYY Achron Affectiva Algocian __ __i Angus.a Birds.ai Captricity Clarifai Cortica Deepomatic DeepVision Descartes Labs Flixsense FotoNation GrokStyle Haystack HireVue Lunit Inc. Matroid Netra Orbital Insight Pilot AI Labs Planet Spaceknow Sticky.ai Valossa Vidi CloudSight — high quality understanding of images within seconds — detects foreign and intentionally placed objects for security — a range of intelligent computer vision techniques Irvine Sensors Pilot.ai Originally posted → liamiscool.com

A list of artificial intelligence tools you can use today — for businesses (2/3)

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn;
(2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com;
(3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn;
(4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn;
(5) Fei Wu, Zhejiang University & wufei@zju.edu.cn;
(6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn;
(7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg;
(8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction
Related Work
LLM Negotiation for Sentiment Analysis
Experiments
Ablation Studies
Conclusion and References 5 Ablation Studies In this section, we perform ablation studies on the Twitter dataset to better understand the mechanism behind the negotiation framework. 5.1 Who takes which role matters In the negotiation framework, there are two roles, the generator and the discriminator, which two separate LLMs take. Table 2 shows the performance for setups where GPT-3.5 and GPT-4 take different roles. As can be seen, when GPT-3.5 acts as the generator, and GPT-4 acts as the discriminator (G3.5-D4 for short), the performance (68.8) is better than single GPT-3.5 without negotiation (65.2), but worse than single GPT-4 without negotiation (69.5). In contrast, negotiation-based configurations with GPT-4 acting as the generator (G4-D3.5 and G4-D4) consistently outperforms standalone GPT-4 or GPT-3.5 models without negotiation. These results underscore the pivotal role that the generator plays in influencing the negotiation outcome. Furthermore, we observe G4- D3.5 can beat G4-D4. We attribute such advantage to the hypothesis that utilizing heterogeneous LLMs for distinct roles could optimize the negotiation’s performance. 5.2 Consensus Percentage Table 3 consensus percentage for different setups. As can be seen, when GPT-4 acts as the generator, the negotiation is more likely to reach a consensus, or reach a consensus in fewer turns. The explanation is intuitive: for the twitter task, we can see from table 1 that GPT-4 obtains better performances that GPT-3.5, which means the reasoning process for GPT-4 is more sensible than 3.5, making the decision of the former more likely to be agreed on. 5.3 Effect of the Reasoning Process In the negotiation process, LLMs are asked to articulate the reason process, a strategy akin to CoT(Wei et al., 2022b). We examine the importance for listing reasons in negotiation by removing the reasoning process and asking LLMs to only output decisions. Results are shown in Table 4. As can be seen, for the three setups, single GPT-3.5, where only GPT-3.5 is used without negotiation, single GPT-4, where only GPT-4 is used without negotiation, and GPT-3.5+GPT-4 where negotiation is employed, performances all degrade when the reasoning process is removed. But interestingly, we see a greater degrade (-2.3) for the negotiation than the single model setup (-1.2 for single-GPT-3.5 and -0.9 for single-GPT-4). This is in accord with our expectation as the reasoning process is of greater significance in the negotiation setup. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction Related Work LLM Negotiation for Sentiment Analysis Experiments Ablation Studies Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work LLM Negotiation for Sentiment Analysis LLM Negotiation for Sentiment Analysis Experiments Experiments Ablation Studies Ablation Studies Conclusion and References Conclusion and References 5 Ablation Studies In this section, we perform ablation studies on the Twitter dataset to better understand the mechanism behind the negotiation framework. 5.1 Who takes which role matters In the negotiation framework, there are two roles, the generator and the discriminator, which two separate LLMs take. Table 2 shows the performance for setups where GPT-3.5 and GPT-4 take different roles. As can be seen, when GPT-3.5 acts as the generator, and GPT-4 acts as the discriminator (G3.5-D4 for short), the performance (68.8) is better than single GPT-3.5 without negotiation (65.2), but worse than single GPT-4 without negotiation (69.5). In contrast, negotiation-based configurations with GPT-4 acting as the generator (G4-D3.5 and G4-D4) consistently outperforms standalone GPT-4 or GPT-3.5 models without negotiation. These results underscore the pivotal role that the generator plays in influencing the negotiation outcome. Furthermore, we observe G4- D3.5 can beat G4-D4. We attribute such advantage to the hypothesis that utilizing heterogeneous LLMs for distinct roles could optimize the negotiation’s performance. 5.2 Consensus Percentage Table 3 consensus percentage for different setups. As can be seen, when GPT-4 acts as the generator, the negotiation is more likely to reach a consensus, or reach a consensus in fewer turns. The explanation is intuitive: for the twitter task, we can see from table 1 that GPT-4 obtains better performances that GPT-3.5, which means the reasoning process for GPT-4 is more sensible than 3.5, making the decision of the former more likely to be agreed on. 5.3 Effect of the Reasoning Process In the negotiation process, LLMs are asked to articulate the reason process, a strategy akin to CoT(Wei et al., 2022b). We examine the importance for listing reasons in negotiation by removing the reasoning process and asking LLMs to only output decisions. Results are shown in Table 4. As can be seen, for the three setups, single GPT-3.5, where only GPT-3.5 is used without negotiation, single GPT-4, where only GPT-4 is used without negotiation, and GPT-3.5+GPT-4 where negotiation is employed, performances all degrade when the reasoning process is removed. But interestingly, we see a greater degrade (-2.3) for the negotiation than the single model setup (-1.2 for single-GPT-3.5 and -0.9 for single-GPT-4). This is in accord with our expectation as the reasoning process is of greater significance in the negotiation setup.

Sentiment Analysis through LLM Negotiations: Ablation Studies

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures D DEPENDENCE ON PARAMETERS AND ABLATION STUDIES Most of our parameter explorations were conducted for OMNIGLOT dataset. We chose a 16-channel model trained on a 1-shot-20-way OMNIGLOT task as an example of a model, for which just the logits layer generation was sufficient. We also chose a 4-channel model trained on a 5-shot-20-way OMNIGLOT task for the role of a model, for which generation of all convolutional layers proved to be beneficial. Figures 5 and 6 show comparison of training and test accuracies on OMNIGLOT for different parameter values for these two models. Here we only used two independent runs for each parameter value, which did not allow us to sufficiently reduce the statistical error. Despite of this, in the following, we try to highlight a few notable parameter dependencies. Note here that in some experiments with particularly large feature or model sizes, training progressed beyond the target number of steps and there could also be overfitting for very large models. Number of transformer layers. Increasing the number of transformer layers is seen to be particularly important in the 4-channel model. The 16-channel model also demonstrates the benefit of using 1 vs 2 transformer layers, but the performance appears to degrade when we use 3 transformer layers. Local feature dimension. Particularly low local feature dimension can be seen to hurt the performance in both models, while using higher local feature dimension appears to be advantageous cases except for the 32-dimensional local feature in the 4-channel model. Embedding dimension. Particularly low embedding dimension of 16 can be seen to hurt the performance of both models. Number of transformer heads. Increasing the number of transformer heads leads to performance degradation in the 16-channel model, but does not have a pronounced effect in the 4-channel model. Shared feature dimensions. Removing the shared feature, or using an 8-dimensional feature can be seen to hurt the performance in both cases of the 4- and 16-channel models. Transformer architecture. While the majority of our experiments were conducted with a sequence of transformer encoder layers, we also experimented with an alternative weight generation approach, where both encoder and decoder transformer layers were employed (see Fig. 4). Our experiments with both architectures suggest that the role of the decoder is pronounced, but very different in two models: in the 16-channel model, the presence of the decoder increases the model performance, while in the 4-channel model, it leads to accuracy degradation. Inner transformer embedding sizes. Varying the ν parameter for different components of the transformer model (key/query pair, value and inner fully-connect layer size), we quantify their importance on the model performance. Using very low ν for the value dimension hurts performance of both models. The effect of key/query and inner dimensions can be distinctly seen only in the 4-channel model, where using ν = 1 or ν = 1.5 appears to produce the best results. Weight allocation approach. Our experiments with the “spatial” weight allocation in 4- and 16-channel models showed slightly inferior performance (both accuracies dropping by about 0.2% to 0.4% in both experiments) compared to that obtained with the “output” weight allocation method. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures D DEPENDENCE ON PARAMETERS AND ABLATION STUDIES Most of our parameter explorations were conducted for OMNIGLOT dataset. We chose a 16-channel model trained on a 1-shot-20-way OMNIGLOT task as an example of a model, for which just the logits layer generation was sufficient. We also chose a 4-channel model trained on a 5-shot-20-way OMNIGLOT task for the role of a model, for which generation of all convolutional layers proved to be beneficial. Figures 5 and 6 show comparison of training and test accuracies on OMNIGLOT for different parameter values for these two models. Here we only used two independent runs for each parameter value, which did not allow us to sufficiently reduce the statistical error. Despite of this, in the following, we try to highlight a few notable parameter dependencies. Note here that in some experiments with particularly large feature or model sizes, training progressed beyond the target number of steps and there could also be overfitting for very large models. Number of transformer layers. Increasing the number of transformer layers is seen to be particularly important in the 4-channel model. The 16-channel model also demonstrates the benefit of using 1 vs 2 transformer layers, but the performance appears to degrade when we use 3 transformer layers. Number of transformer layers. Local feature dimension. Particularly low local feature dimension can be seen to hurt the performance in both models, while using higher local feature dimension appears to be advantageous cases except for the 32-dimensional local feature in the 4-channel model. Local feature dimension. Embedding dimension. Particularly low embedding dimension of 16 can be seen to hurt the performance of both models. Embedding dimension. Number of transformer heads. Increasing the number of transformer heads leads to performance degradation in the 16-channel model, but does not have a pronounced effect in the 4-channel model. Number of transformer heads. Shared feature dimensions. Removing the shared feature, or using an 8-dimensional feature can be seen to hurt the performance in both cases of the 4- and 16-channel models. Shared feature dimensions. Transformer architecture. While the majority of our experiments were conducted with a sequence of transformer encoder layers, we also experimented with an alternative weight generation approach, where both encoder and decoder transformer layers were employed (see Fig. 4). Our experiments with both architectures suggest that the role of the decoder is pronounced, but very different in two models: in the 16-channel model, the presence of the decoder increases the model performance, while in the 4-channel model, it leads to accuracy degradation. Transformer architecture. Inner transformer embedding sizes. Varying the ν parameter for different components of the transformer model (key/query pair, value and inner fully-connect layer size), we quantify their importance on the model performance. Using very low ν for the value dimension hurts performance of both models. The effect of key/query and inner dimensions can be distinctly seen only in the 4-channel model, where using ν = 1 or ν = 1.5 appears to produce the best results. Inner transformer embedding sizes. Weight allocation approach. Our experiments with the “spatial” weight allocation in 4- and 16-channel models showed slightly inferior performance (both accuracies dropping by about 0.2% to 0.4% in both experiments) compared to that obtained with the “output” weight allocation method. Weight allocation approach.

HyperTransformer: D Dependence On Parameters and Ablation Studies

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn;
(2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com;
(3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn;
(4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn;
(5) Fei Wu, Zhejiang University & wufei@zju.edu.cn;
(6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn;
(7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg;
(8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction
Related Work
LLM Negotiation for Sentiment Analysis
Experiments
Ablation Studies
Conclusion and References 3 LLM Negotiation for Sentiment Analysis 3.1 Overview In this section, we detail the multi-LLM negotiation framework for sentiment analysis: Two LLMs perform as the answer generator and discriminator. We refer to the interaction between the generator and the discriminator as a negotiation. The negotiation will repeat until a consensus is reached or the maximum number of negotiation turns is exceeded. Illustrations are shown in Figures 1 and 2. 3.2 Reasoning-infused generator The generator is backboned by a large language model. We ask the answer generator based on the ICL paradigm through prompts, aiming to generate a step-by-step reasoning chain and a decision towards the sentiment polarity of the test input. Prompts are composed of three elements: a task description, demonstrations, and a test input.  The task description is a description of the task in natural language (e.g., "Please determine the overall sentiment of test input."); the test input is the textual input in the test set (e.g., "The sky is blue."); demonstrations are from the train set of the task. Each consists of three elements: input, reasoning chains, and sentimental decision. For each test input, we first retrieve K nearest neighbors (input, sentiment decision) from the train set as demonstrations. Then, we transform demonstrations to (input, reasoning process, sentiment decision) triplets by prompting the generator to produce a reasoning chain. After concatenating the task description, demonstrations, and the test input, we forward the prompt to the generator, which will respond with a step-by-step reasoning chain and a sentimental decision. 3.3 Explanation-deriving discriminator The discriminator is backboned by another LLM. After finishing the answer generating process, the answer discriminator is used to judge whether the decision made by the generator is correct and provide a reasonable explanation. To accomplish this goal, we first construct prompts for the answer discriminator. The prompt is composed of four elements: a task description, demonstrations, a test input, and the response from the answer generator. The task description is a piece of text that describes the task in natural language (e.g., "Please determine whether the decision is correct."). Each demonstration is composed of six elements: (input text, a reasoning chain, sentiment decision, discriminator attitude, discriminator explanations, discriminator decision) and constructed by prompting the answer discriminator to provide explanations of why the sentiment decision is correct for the input text. Then we ask the discriminator with the construct prompt. The answer discriminator will respond with a text string, containing an attitude (i.e., yes, no) that denotes whether the discriminator agrees with the generator, explanations that explain why the discriminator agrees/disagrees with the generator, and a discriminator decision that determines the sentiment of the test input. Why Two LLMs but Not One? There are two reasons for using two different LLMs separately for the generator and the discriminator rather than using a single LLM to act as two roles: (1) If an LLM makes a mistake as a generator due to incorrect reasoning, it is more likely that it will also make the same mistake as the discriminator as since generator and the discriminator from the same model are very likely to make similar rationales; (2) by using two separate models, we are able to take the advantage of the complementary abilities of the two models. 3.4 Role-flipped Negotiation After two LLMs end with a negotiation, we ask them flip roles and initiate a new negotiation, where the second LLM acts as the generator, and the first LLM acts as the discriminator. We refer the interaction of two LLMs with flipped roles as role-flipped negotiation. Likewise, the role flipped negotiation is ended until a consensus is reached or the maximum number of negotiation turns is exceeded. When both negotiations result in an agreement and their decisions are the same, we can choose either decision as the final one since they are the same. If one of the negotiations fails to reach a consensus while the other reaches a decision, we choose the decision from the negotiation  that reached a consensus as the final decision. However, if both negotiations reach a consensus but their decisions do not align, we will require the assistance of an additional Language Model (LLM), as will be explained in more detail below." Introducing a third LLM If the decision from the two negotiations do not align, we introduce a third LLM and conduct the negotiation and role-flipped negotiation with each of the two aforementioned LLMs. Subsequently, we will get 6 negotiation results and vote on these results: the decision that appears most frequently is taken as the sentiment polarity of the input test. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction Related Work LLM Negotiation for Sentiment Analysis Experiments Ablation Studies Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work LLM Negotiation for Sentiment Analysis LLM Negotiation for Sentiment Analysis Experiments Experiments Ablation Studies Ablation Studies Conclusion and References Conclusion and References 3 LLM Negotiation for Sentiment Analysis 3.1 Overview In this section, we detail the multi-LLM negotiation framework for sentiment analysis: Two LLMs perform as the answer generator and discriminator. We refer to the interaction between the generator and the discriminator as a negotiation. The negotiation will repeat until a consensus is reached or the maximum number of negotiation turns is exceeded. Illustrations are shown in Figures 1 and 2. 3.2 Reasoning-infused generator The generator is backboned by a large language model. We ask the answer generator based on the ICL paradigm through prompts, aiming to generate a step-by-step reasoning chain and a decision towards the sentiment polarity of the test input. Prompts are composed of three elements: a task description, demonstrations, and a test input.  The task description is a description of the task in natural language (e.g., "Please determine the overall sentiment of test input."); the test input is the textual input in the test set (e.g., "The sky is blue."); demonstrations are from the train set of the task. Each consists of three elements: input, reasoning chains, and sentimental decision. For each test input, we first retrieve K nearest neighbors (input, sentiment decision) from the train set as demonstrations. Then, we transform demonstrations to (input, reasoning process, sentiment decision) triplets by prompting the generator to produce a reasoning chain. After concatenating the task description, demonstrations, and the test input, we forward the prompt to the generator, which will respond with a step-by-step reasoning chain and a sentimental decision. 3.3 Explanation-deriving discriminator The discriminator is backboned by another LLM. After finishing the answer generating process, the answer discriminator is used to judge whether the decision made by the generator is correct and provide a reasonable explanation. To accomplish this goal, we first construct prompts for the answer discriminator. The prompt is composed of four elements: a task description, demonstrations, a test input, and the response from the answer generator. The task description is a piece of text that describes the task in natural language (e.g., "Please determine whether the decision is correct."). Each demonstration is composed of six elements: (input text, a reasoning chain, sentiment decision, discriminator attitude, discriminator explanations, discriminator decision) and constructed by prompting the answer discriminator to provide explanations of why the sentiment decision is correct for the input text. Then we ask the discriminator with the construct prompt. The answer discriminator will respond with a text string, containing an attitude (i.e., yes, no) that denotes whether the discriminator agrees with the generator, explanations that explain why the discriminator agrees/disagrees with the generator, and a discriminator decision that determines the sentiment of the test input. Why Two LLMs but Not One? There are two reasons for using two different LLMs separately for the generator and the discriminator rather than using a single LLM to act as two roles: (1) If an LLM makes a mistake as a generator due to incorrect reasoning, it is more likely that it will also make the same mistake as the discriminator as since generator and the discriminator from the same model are very likely to make similar rationales; (2) by using two separate models, we are able to take the advantage of the complementary abilities of the two models. Why Two LLMs but Not One? 3.4 Role-flipped Negotiation After two LLMs end with a negotiation, we ask them flip roles and initiate a new negotiation, where the second LLM acts as the generator, and the first LLM acts as the discriminator. We refer the interaction of two LLMs with flipped roles as role-flipped negotiation. Likewise, the role flipped negotiation is ended until a consensus is reached or the maximum number of negotiation turns is exceeded. When both negotiations result in an agreement and their decisions are the same, we can choose either decision as the final one since they are the same. If one of the negotiations fails to reach a consensus while the other reaches a decision, we choose the decision from the negotiation  that reached a consensus as the final decision. However, if both negotiations reach a consensus but their decisions do not align, we will require the assistance of an additional Language Model (LLM), as will be explained in more detail below." Introducing a third LLM If the decision from the two negotiations do not align, we introduce a third LLM and conduct the negotiation and role-flipped negotiation with each of the two aforementioned LLMs. Subsequently, we will get 6 negotiation results and vote on these results: the decision that appears most frequently is taken as the sentiment polarity of the input test. Introducing a third LLM

Sentiment Analysis through LLM Negotiations: LLM Negotiation for Sentiment Analysis

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn;
(2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com;
(3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn;
(4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn;
(5) Fei Wu, Zhejiang University & wufei@zju.edu.cn;
(6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn;
(7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg;
(8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction
Related Work
LLM Negotiation for Sentiment Analysis
Experiments
Ablation Studies
Conclusion and References 2 Related Work 2.1 Sentiment Analysis Sentiment analysis (Pang and Lee, 2008; Go et al., 2009; Maas et al., 2011a; Zhang and Liu, 2012; Baccianella et al., 2010; Medhat et al., 2014; Bakshi et al., 2016; Zhang et al., 2018) is a task that aims to determine the overall sentiment polarity (e.g., positive, negative, neutral) of a given text. Earlier work often formalized the task as a two-step problem: (1) extract features using RNNs (Socher et al., 2013; Qian et al., 2016; Peled and Reichart, 2017; Wang et al., 2016b; Guggilla et al., 2016; Vo and Zhang, 2015), CNNs (Kalchbrenner et al., 2014; Wang et al., 2016a; Guan et al., 2016; Yu and Jiang, 2016; Mishra et al., 2017), pretrained language models (Lin et al., 2021; Sun et al., 2021; Phan and Ogunbona, 2020; Dai et al., 2021), etc; and (2) feed extracted features into a classifier for obtaining a pre-defined sentimental label. In recent years, in-context learning (ICL) has achieved great success and changed the paradigm of NLP tasks. Many works adapt ICL to the sentiment analysis task: Qin et al. (2023b); Sun et al.(2023a) propose a series of strategies to improve ChatGPT’s performance on the sentiment analysis task; Fei et al. (2023) propose a threehop reasoning framework, which induces the implicit aspect, opinion, and finally the sentiment polarity for the implicit sentiment analysis task;ƒ Zhang et al. (2023d) find that LLMs can achieve satisfactory performance on the binary sentiment classification task, but they underperform to the supervised baseline on more complex tasks (e.g., fine-grained sentiment analysis) that require deeper understanding or structured sentiment information. 2.2 Large Language Models and In-context Learning Large language models (LLMs) (Wang et al., 2022a; Zhang et al., 2023b) are models trained on massive unlabeled text corpora with self supervised learning techniques. Based on the model architecture, LLMs can be categorized into three types: (1) encoder-only models, which contain a text encoder and generate the input representations, such as BERT (Devlin et al., 2018) and its variants (Lan et al., 2019; Liu et al., 2019; Sun et al., 2020; Clark et al., 2020; Feng et al., 2020; Joshi et al., 2020; Sun et al., 2020, 2021); (2) decoder-only models, which have a decoder and generate text conditioned on the input text like GPT-series models (Radford et al., 2019; Brown et al., 2020; Keskar et al., 2019; Radford et al., 2019; Chowdhery et al., 2022; Ouyang et al., 2022; Zhang et al., 2022a; Scao et al., 2022; Zeng et al., 2022b; Touvron et al., 2023a; Peng et al., 2023; OpenAI, 2023); and (3) encoder-decoder models, which have a pair of encoder-decoder and generate text conditioned on the input representation, such as T5 (Raffel et al., 2020) and its variants (Lewis et al., 2019; Xue et al., 2020). Starting with GPT-3 (Brown et al., 2020), LLMs have shown emerging capabilities (Wei et al., 2022a) and completed NLP tasks through incontext learning (ICL), where LLMs generate labelintensive text conditioned on a few annotated examples without gradient updates. Many studies in the literature propose strategies for improving ICL performances on NLP tasks. Li and Liang (2021); Chevalier et al. (2023); Mu et al. (2023) optimize prompts in the continuous space. Liu et al. (2021a); Wan et al. (2023); Zhang et al. (2023a) search through the train set to retrieve k nearest neighbors of a test input as demonstrations. Zhang et al. (2022b); Sun et al. (2023b); Yao et al (2023) decompose a task into a few subtasks and solve them step-by-step towards the final answer conditioned on LLM-generated reasoning chains. Sun et al. (2023a); Wang et al. (2023) propose to verify LLMs’ results by conducting a new round of prompting; Liu et al. (2021b); Feng et al. (2023) use LLMs to generate natural language knowledge statements and integrate external knowledge statements into prompts. 2.3 The LLM collaboration The LLM collaboration involves multiple LLMs working together to solve a given task. Specifically, the task is decomposed to several intermediate tasks, and each LLM is assigned to complete one intermediate task independently. The given task is solved after integrating or summarizing these intermediate results. The LLM collaboration approach can exploit the capabilities of LLMs, improve performances on complex tasks and enable to build complicated systems. Shinn et al. (2023); Sun et al. (2023a); Gero et al. (2023); Wang and Li (2023); Chen et al. (2023b) construct auxiliary tasks (e.g., reflection, verification tasks) and revise the response to the original task referring to the result of the auxiliary task. Talebirad and Nadiri (2023); Hong et al. (2023); Qian et al. (2023) assign characterize profiles (e.g., project manager, software engineer) to LLMs and gain performance boosts on character-specific tasks through behavior animations. Li et al. (2022); Zeng et al. (2022a); Chen et al. (2023a); Du et al. (2023); Liang et al. (2023) use a debate strategy in which multiple different LLMs propose their own responses to the given task and debate over multiple turns until getting a common final answer. Besides, Shen et al. (2023); Gao et al. (2023); Ge et al. (2023); Zhang et al. (2023c); Hao et al. (2023) employone LLM as the task controller, which devises a  plan for the given task, selects expert models for implementation and summarizes the responses of intermediate planned tasks. Other LLMs serve as task executors, completing intermediate tasks in their areas of expertise. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaofei Sun, Zhejiang University & xiaofei_sun@zju.edu.cn; (2) Xiaoya Li, Shannon.AI, Bytedance & xiaoya_li@shannonai.com; (3) Shengyu Zhang, Zhejiang University & sy_zhang@zju.edu.cn; (4) Shuhe Wang, Peking University & wangshuhe@stu.pku.edu.cn; (5) Fei Wu, Zhejiang University & wufei@zju.edu.cn; (6) Jiwei Li, Zhejiang University &  jiwei_li@zju.edu.cn; (7) Tianwei Zhang, Nanyang Technological University & tianwei.zhang@ntu.edu.sg; (8) Guoyin Wang, Bytedance & guoyin.wang@bytedance.com. Table of Links Abstract and Introduction Related Work LLM Negotiation for Sentiment Analysis Experiments Ablation Studies Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work LLM Negotiation for Sentiment Analysis LLM Negotiation for Sentiment Analysis Experiments Experiments Ablation Studies Ablation Studies Conclusion and References Conclusion and References 2 Related Work 2.1 Sentiment Analysis Sentiment analysis (Pang and Lee, 2008; Go et al., 2009; Maas et al., 2011a; Zhang and Liu, 2012; Baccianella et al., 2010; Medhat et al., 2014; Bakshi et al., 2016; Zhang et al., 2018) is a task that aims to determine the overall sentiment polarity (e.g., positive, negative, neutral) of a given text. Earlier work often formalized the task as a two-step problem: (1) extract features using RNNs (Socher et al., 2013; Qian et al., 2016; Peled and Reichart, 2017; Wang et al., 2016b; Guggilla et al., 2016; Vo and Zhang, 2015), CNNs (Kalchbrenner et al., 2014; Wang et al., 2016a; Guan et al., 2016; Yu and Jiang, 2016; Mishra et al., 2017), pretrained language models (Lin et al., 2021; Sun et al., 2021; Phan and Ogunbona, 2020; Dai et al., 2021), etc; and (2) feed extracted features into a classifier for obtaining a pre-defined sentimental label. In recent years, in-context learning (ICL) has achieved great success and changed the paradigm of NLP tasks. Many works adapt ICL to the sentiment analysis task: Qin et al. (2023b); Sun et al.(2023a) propose a series of strategies to improve ChatGPT’s performance on the sentiment analysis task; Fei et al. (2023) propose a threehop reasoning framework, which induces the implicit aspect, opinion, and finally the sentiment polarity for the implicit sentiment analysis task;ƒ Zhang et al. (2023d) find that LLMs can achieve satisfactory performance on the binary sentiment classification task, but they underperform to the supervised baseline on more complex tasks (e.g., fine-grained sentiment analysis) that require deeper understanding or structured sentiment information. 2.2 Large Language Models and In-context Learning Large language models (LLMs) (Wang et al., 2022a; Zhang et al., 2023b) are models trained on massive unlabeled text corpora with self supervised learning techniques. Based on the model architecture, LLMs can be categorized into three types: (1) encoder-only models, which contain a text encoder and generate the input representations, such as BERT (Devlin et al., 2018) and its variants (Lan et al., 2019; Liu et al., 2019; Sun et al., 2020; Clark et al., 2020; Feng et al., 2020; Joshi et al., 2020; Sun et al., 2020, 2021); (2) decoder-only models, which have a decoder and generate text conditioned on the input text like GPT-series models (Radford et al., 2019; Brown et al., 2020; Keskar et al., 2019; Radford et al., 2019; Chowdhery et al., 2022; Ouyang et al., 2022; Zhang et al., 2022a; Scao et al., 2022; Zeng et al., 2022b; Touvron et al., 2023a; Peng et al., 2023; OpenAI, 2023); and (3) encoder-decoder models, which have a pair of encoder-decoder and generate text conditioned on the input representation, such as T5 (Raffel et al., 2020) and its variants (Lewis et al., 2019; Xue et al., 2020). Starting with GPT-3 (Brown et al., 2020), LLMs have shown emerging capabilities (Wei et al., 2022a) and completed NLP tasks through incontext learning (ICL), where LLMs generate labelintensive text conditioned on a few annotated examples without gradient updates. Many studies in the literature propose strategies for improving ICL performances on NLP tasks. Li and Liang (2021); Chevalier et al. (2023); Mu et al. (2023) optimize prompts in the continuous space. Liu et al. (2021a); Wan et al. (2023); Zhang et al. (2023a) search through the train set to retrieve k nearest neighbors of a test input as demonstrations. Zhang et al. (2022b); Sun et al. (2023b); Yao et al (2023) decompose a task into a few subtasks and solve them step-by-step towards the final answer conditioned on LLM-generated reasoning chains. Sun et al. (2023a); Wang et al. (2023) propose to verify LLMs’ results by conducting a new round of prompting; Liu et al. (2021b); Feng et al. (2023) use LLMs to generate natural language knowledge statements and integrate external knowledge statements into prompts. 2.3 The LLM collaboration The LLM collaboration involves multiple LLMs working together to solve a given task. Specifically, the task is decomposed to several intermediate tasks, and each LLM is assigned to complete one intermediate task independently. The given task is solved after integrating or summarizing these intermediate results. The LLM collaboration approach can exploit the capabilities of LLMs, improve performances on complex tasks and enable to build complicated systems. Shinn et al. (2023); Sun et al. (2023a); Gero et al. (2023); Wang and Li (2023); Chen et al. (2023b) construct auxiliary tasks (e.g., reflection, verification tasks) and revise the response to the original task referring to the result of the auxiliary task. Talebirad and Nadiri (2023); Hong et al. (2023); Qian et al. (2023) assign characterize profiles (e.g., project manager, software engineer) to LLMs and gain performance boosts on character-specific tasks through behavior animations. Li et al. (2022); Zeng et al. (2022a); Chen et al. (2023a); Du et al. (2023); Liang et al. (2023) use a debate strategy in which multiple different LLMs propose their own responses to the given task and debate over multiple turns until getting a common final answer. Besides, Shen et al. (2023); Gao et al. (2023); Ge et al. (2023); Zhang et al. (2023c); Hao et al. (2023) employone LLM as the task controller, which devises a  plan for the given task, selects expert models for implementation and summarizes the responses of intermediate planned tasks. Other LLMs serve as task executors, completing intermediate tasks in their areas of expertise.

ALATION

alation.com

501-1K employees

Since 2012

COMPANY RANKING

ALATION

EVERGREEN INDEX #789

Representing this company? Click here to claim and customize this page!

Alation WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

ALATION

alation.com

501-1K employees

Since 2012

COMPANY RANKING

ALATION

EVERGREEN INDEX #789

RELATED COMPANIESRANK

STORIES ABOUT ALATION

Representing this company? Click here to claim and customize this page!

Alation WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!