Ness

ness

Hands-on with Apache Iceberg on Your Laptop: Deep Dive with Apache Spark, Nessie, Minio and more!

Many of us have made changes to data based on requirements that later evolve. By the time we realize it, it becomes impossible to roll back. Change isn't just a constant; it's an essential part of managing data that demands a sophisticated approach. Enter the Git-like functionality of Dremio's data catalog Nessie. Just as Git has become fundamental to software development, data engineers need similar tools to work in parallel, compare data versions, promote changes to production and roll back data when needed. Nessie provides data engineers with a Git-like version control system for managing data versions, branches, merges, and commits. This can be very helpful when multiple data engineers are working with and transforming data at the same time. Nessie allows each engineer to work in separate branches while maintaining a single source of truth in the form of the main branch. This functionality empowers data engineering teams to maintain data quality collaboratively in the face of unrelenting change. This article provides a step-by-step guide demonstrating how Nessie, Dremio and MinIO work together to enhance data quality and collaboration in your data engineering workflows. Whether you're a data engineer, ML engineer, or just a modern data lake enthusiast, this blog equips you with the knowledge and tools needed to effectively enhance your data versioning practices. Understanding the Basics: Nessie Workloads Nessie allows for Git-like workloads that let you test, develop and push to production. Let's break down some key concepts: Just like in Git, branches in Nessie allow you to work on different data engineering tasks concurrently. For example, you might have a branch for feature development, data cleaning and data transformation. Each branch can have its own set of data changes. Branches: In Nessie, a commit represents a snapshot of the data at a specific point in time. When you make changes to your data, you create a new commit, which records those changes. Commits are linked to a specific branch. When you need to revert your data to a stable or known-good state, Nessie makes it easy to select a specific commit and roll back to that data version. This ensures data quality and consistency. Commits: : Nessie allows you to merge the changes from one branch into another. This is similar to merging code changes in Git. When you merge a branch, the data changes made in that branch become part of the target branch. Merges The Components is a distributed analytics engine that operates as an open-source platform, offering an intuitive self-service interface for data exploration, transformation and collaborative work. Its design is grounded in a high-speed columnar memory format. Dremio Apache Arrow, We’ve already explored how to deploy Dremio with Kubernetes, and also how to . For additional information about Dremio, consult . query Iceberg tables on MinIO with Dremio Dremio Resources is high-performance object storage. Renowned for its exceptional and , MinIO serves as a pivotal component in building and maintaining modern data lake infrastructures. MinIO empowers architects to efficiently manage and store massive volumes of data on-prem, on bare metal, on the edge, or on any of the public clouds. MinIO speed scalability is an open table format suited for managing large volumes of data in data lakes. Unique features like time travel, dynamic schema evolution, and Partition Evolution make it a game-changer, allowing query engines to work concurrently on the same data safely and efficiently. See for more information about Iceberg features. Apache Iceberg MinIO The Definitive Guide to Lakehouse Architecture with Iceberg and Getting Started You’ll need the and for this tutorial. The easiest way to get both if you don’t already have them is to install Docker Desktop. Docker Engine Docker Compose This part of the tutorial is based on Dremio’s . You can get the repo for this project . blog post here To begin, open a terminal and navigate to the folder where you cloned/downloaded the repo, and run the command below to start up Dremio. docker-compose up dremio Wait a few moments and then navigate to to access Dremio. Update the requested fields and then click Next. http://localhost:9047 Next, run the following command to start MinIO. docker-compose up minio The final command below will start up Nessie. docker-compose up docker-compose up nessie Navigate to to log in to MinIO with the username and password of . You’ll be prompted to create a bucket. http://localhost:9001 minioadmin:minioadmin Make one named . iceberg-datalake Then head back over to Dremio at and click on Add Source and select Nessie. http://localhost:9047 Set the Name to nessie Set the endpoint URL to http://nessie:19120/api/v2 Set the authentication to none Don’t click Save yet. Instead, in the navigation panel on the left, click Storage. MinIO is S3-API compatible object storage and can use the same connection pathways as AWS S3. For your access key, set minioadmin For your secret key, set minioadmin Set root path to /iceberg-datalake Scroll down for the next set of instructions. Click on the button under to create and configure the following properties. Add Property Connection Properties to fs.s3a.path.style.access true to fs.s3a.endpoint minio:9000 to dremio.s3.compat true Uncheck Encrypt connection Then click Save. You should now see Nessie Catalogs in your data sources. Create a Data Source In Dremio navigate to SQL Runner on the left-hand side. Make sure that in the upper right area of the text editor is set to our Nessie source. Otherwise, you will have to reference the context like instead of just to run this query. Copy and paste the SQL below and run. Context nessie.SalesData SalesData CREATE TABLE SalesData (
    id INT,
    product_name VARCHAR,
    sales_amount DECIMAL,
    transaction_date DATE
) PARTITION BY (transaction_date); Run the query below to insert data into the table you just created. INSERT INTO SalesData (id, product_name, sales_amount, transaction_date)
VALUES
    (1, 'ProductA', 1500.00, '2023-10-15'),
    (2, 'ProductB', 2000.00, '2023-10-15'),
    (3, 'ProductA', 1200.00, '2023-10-16'),
    (4, 'ProductC', 1800.00, '2023-10-16'),
    (5, 'ProductB', 2200.00, '2023-10-17'); Navigate back to MinIO to see that your data lake has been populated with the Iceberg tables. Branching and Merging with Nessie Return to Dremio at . Begin by querying a table on the main branch using the syntax: http://localhost:9047 AT BRANCH SELECT * FROM  nessie.SalesData AT BRANCH main; Create an ETL (Extract Transform and Load) branch to allow you to play around with and transform data without impacting production. CREATE BRANCH etl_06092023 in nessie Within the ETL branch, insert new data into the table: USE BRANCH etl_06092023 in nessie;
INSERT INTO nessie.SalesData (id, product_name, sales_amount, transaction_date) VALUES
(6, 'ProductC', 1400.00, '2023-10-18'); Confirm the immediate availability of the new data within the ETL branch: SELECT * FROM nessie.SalesData AT BRANCH etl_06092023; Note the isolation of changes from users on the main branch: SELECT * FROM nessie.SalesData AT BRANCH main; Merge the changes from the ETL branch back into the main branch: MERGE BRANCH etl_06092023 INTO main in nessie; Select the main branch again to see that the changes have indeed been merged. SELECT * FROM nessie.SalesData AT BRANCH main This branching strategy enables data engineers to independently handle numerous transactions across multiple tables. When they are ready, data engineers can merge these transactions into a single, comprehensive multi-table transaction within the main branch. Conclusion This blog post delved into the power of Git-like version control in data engineering, emphasizing how seamlessly manages data versions, branches and merges. This step-by-step guide demonstrates how Nessie, in collaboration with Dremio and MinIO, as the object storage foundation, enhances data quality and collaboration in data engineering workflows. Nessie Let us know what your data lakehouse looks like at or on . hello@minio.io our Slack channel Also appears . here

Data Lake Mysteries Revealed: Nessie, Dremio, and MinIO Make Waves

ডেটা লেক রহস্য প্রকাশিত হয়েছে: নেসি, ড্রেমিও এবং মিনিও মেক ওয়েভস

Data-Lake-Rätsel aufgedeckt: Nessie, Dremio und MinIO sorgen für Aufsehen

Misterios del lago de datos revelados: Nessie, Dremio y MinIO hacen olas

Les mystères du Data Lake révélés : Nessie, Dremio et MinIO font des vagues

डेटा लेक रहस्य का खुलासा: नेस्सी, ड्रेमियो और मिनिओ मेक वेव्स

データレイクの謎が明らかに: Nessie、Dremio、MinIO が波紋を起こす

데이터 레이크의 미스터리 공개: Nessie, Dremio 및 MinIO가 물결을 일으키다

Mistérios do Data Lake revelados: Nessie, Dremio e MinIO fazem ondas

Тайны озера данных раскрыты: Несси, Дремио и MinIO вызывают фурор

Data Lake'in Gizemleri Ortaya Çıktı: Nessie, Dremio ve MinIO Dalga Yaratıyor

Những bí ẩn về hồ dữ liệu được tiết lộ: Nessie, Dremio và MinIO tạo nên làn sóng

数据湖谜团揭开：Nessie、Dremio 和 MinIO 掀起波澜

FTC v. Amazon Court Filing, retrieved on Sep 26, 2023, is part of . You can jump to any part in this filing . This is part 60 of 80. HackerNoon’s Legal PDF Series here COUNT IV: UNFAIR METHOD OF COMPETITION (15 U.S.C. § 45(a)) 457. Plaintiff FTC re-alleges and incorporates by reference the allegations in paragraphs 1-456 above. 458. Amazon has engaged in an unfair method of competition, called Project Nessie, that (redacted) 459. Amazon designed and used its Project Nessie pricing system (redacted) 460. Amazon’s Project Nessie pricing system (redacted) 461 (Redacted) 462. Amazon’s use of its Project Nessie pricing system is an unfair method of competition in violation of Section 5(a) of the FTC Act, 15 U.S.C. § 45(a). 463. There is no valid and cognizable justification for Amazon’s use of Project Nessie. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case 2:23-cv-01495 retrieved on October 2, 2023, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. ftc.gov

How Amazon's Project Nessie Supports Its Anti-Competitive Tactics

Winter(Physics) is Coming It now looks like Large Language Models running on the GPT technology may be nearing a plateau in their development. This is from popular Physicist Dr. Sabine Hossenfelder’s latest YouTube video. https://www.youtube.com/watch?v=AqwSZEQkknU&embedable=true Dr. Sabine cites a couple of sources, including Bill Gates, who said last year that GPT 5 won’t be much better than GPT 4. And Prof. Gary Marcus says he has been drumming this particular beat for years. This goes against all the Ray Kurzweilian hype by Sam Altman and Ilya Sutskever as seen in the video. But if merely beautiful mathematics won’t spur the progress of Physics, as Dr. Sabine always reminds us, then merely wishful group-thinking won’t give us a superhuman artificial general intelligence (AGI). Why isn’t throwing as much data, computing power, and human supervision as can be afforded, going to birth us a superhuman AGI? Dr. Sabine has a ready answer. The problem, she explains, is the Decoupling of Scales. The Decoupling of Scales The world is broken up into scales. In the physical world, there is the scale of land animals like us humans. Then below that we have microbes, then molecules, atoms, and subatomic particles. In the opposite direction, we have aircraft and skyscrapers, mountains, planets, stars, solar systems, galaxies, etc. In the world of data, we have high-resolution data e.g. categories like “objects that can be touched”, “animals”, “cats”. Then low-resolution data to further expand the above categories e.g. “cat species”, “cat patterns”, “cat behavior at 8:00 am on Monday,” etc. There are other taxonomies, however. Some data is online and some is offline in books, papers, and people’s minds. We may have all the knowledge we desire at the high-resolution scale of human language creation on the internet. But that doesn’t equate to the much bigger, low-resolution data source that is human language creation in the world. Thus, Sam Altman’s AI is like Aristotle in the image below. Hoping to discover the Standard Model of Physics by looking at his hand. But without something like the Large Hadron Collider to poke subatomic particles and see the strange new data hiding within, Aristotle is at a loss on how to proceed. A Large Hadron Collider for AI GPT architecture, unveiled to the world only 2 years ago with the phenomenal ChatGPT, is likely nearing the end of its “improvement-ness”. Perhaps it is time to seriously consider passing the baton, and the VC funding, to something else. I am rooting for thermodynamic computing, currently being explored by companies like Extropic, which works by leveraging thermal noise to create random samples of data as in Stable Diffusion technology, then denoising using statistical analysis. In fact, in any neural network, the initial weights are usually initialized with random noise. During training, backpropagation shapes the final weights from this random noise - writes Laszlo Fazekas. The advantage lies in the chips operating at scales smaller than those of traditional digital computing, where quantum effects result in abundant thermal noise. Thermal noise has always been plenty, anyway. My kitchen stove is full of thermal noise. The real problem has always been how to understand it better through statistical chunking of its data, without throwing away data during regression analysis. My guess? Some sort of massive cloud computational AI training leveraging smartphones, bitcoin nodes, TV sets, radios, name it. Currently, allowing the AI some “thinking time'“ so that it runs longer inferences on the same data, is how AI companies are squeezing out some more juice. For example with ChatGPT’s o1 model. But maybe even training time should be extended. Maybe to get great AI, like a child, it needs decades of training from data collected from all over. Collected and slowly siphoned into a condensed AI model in a few servers. Sam Altman might need those trillions of government dollars for such a plan. Alas, they will likely not be easy to get. The D.O.G.E might not bankroll him (if xAI is suffering the same troubles). Meanwhile, Bitcoin is still growing As the above graphic shows, very few people are using Bitcoin. Yet it is already at $90k. Data from Bitbo shows that in fact, only 400,000 people are using Bitcoin daily. And only 106 million own Bitcoin. Contrast this with the 314 million people who have used AI tools as per data from Statista. ChatGPT alone has millions of users per day. Probably more than the entire Bitcoin blockchain combined! What’s more? AI amassed the bulk of these users in less than a year. This feels wrong. Bitcoin took years to grow. People sweated, lost, DCA’d, had faith. Maybe that’s it. ChatGPT has grown too fast. Now it has slowed down and everybody is sour. Sorry, ya’ll, there is still work to do. AGI will not be so easy to grasp, and GPT scientists should learn to hodl like Bitcoiners. Winter will come and go, but right now, it is time to pay the piper. Winter(Physics) is Coming It now looks like Large Language Models running on the GPT technology may be nearing a plateau in their development. This is from popular Physicist Dr. Sabine Hossenfelder’s latest YouTube video. https://www.youtube.com/watch?v=AqwSZEQkknU&embedable=true https://www.youtube.com/watch?v=AqwSZEQkknU&embedable=true Dr. Sabine cites a couple of sources, including Bill Gates, who said last year that GPT 5 won’t be much better than GPT 4. And Prof. Gary Marcus says he has been drumming this particular beat for years. This goes against all the Ray Kurzweilian hype by Sam Altman and Ilya Sutskever as seen in the video. said last year says But if merely beautiful mathematics won’t spur the progress of Physics, as Dr. Sabine always reminds us, then merely wishful group-thinking won’t give us a superhuman artificial general intelligence (AGI). Why isn’t throwing as much data, computing power, and human supervision as can be afforded, going to birth us a superhuman AGI? Dr. Sabine has a ready answer. The problem, she explains, is the Decoupling of Scales . Decoupling of Scales The Decoupling of Scales The world is broken up into scales. In the physical world, there is the scale of land animals like us humans. Then below that we have microbes, then molecules, atoms, and subatomic particles. In the opposite direction, we have aircraft and skyscrapers, mountains, planets, stars, solar systems, galaxies, etc. In the world of data, we have high-resolution data e.g. categories like “objects that can be touched”, “animals”, “cats”. Then low-resolution data to further expand the above categories e.g. “cat species”, “cat patterns”, “cat behavior at 8:00 am on Monday,” etc. There are other taxonomies, however. Some data is online and some is offline in books, papers, and people’s minds. We may have all the knowledge we desire at the high-resolution scale of human language creation on the internet . But that doesn’t equate to the much bigger, low-resolution data source that is human language creation in the world . human language creation on the internet human language creation in the world Thus, Sam Altman’s AI is like Aristotle in the image below. Hoping to discover the Standard Model of Physics by looking at his hand. But without something like the Large Hadron Collider to poke subatomic particles and see the strange new data hiding within, Aristotle is at a loss on how to proceed. A Large Hadron Collider for AI GPT architecture, unveiled to the world only 2 years ago with the phenomenal ChatGPT, is likely nearing the end of its “improvement-ness”. Perhaps it is time to seriously consider passing the baton, and the VC funding, to something else. I am rooting for thermodynamic computing , currently being explored by companies like Extropic, which works by leveraging thermal noise to create random samples of data as in Stable Diffusion technology, then denoising using statistical analysis. thermodynamic computing In fact, in any neural network, the initial weights are usually initialized with random noise. During training, backpropagation shapes the final weights from this random noise - writes Laszlo Fazekas. in any neural network , the initial weights are usually initialized with random noise. During training, backpropagation shapes the final weights from this random noise - neural network backpropagation The advantage lies in the chips operating at scales smaller than those of traditional digital computing, where quantum effects result in abundant thermal noise. Thermal noise has always been plenty, anyway. My kitchen stove is full of thermal noise. The real problem has always been how to understand it better through statistical chunking of its data, without throwing away data during regression analysis. My guess? Some sort of massive cloud computational AI training leveraging smartphones, bitcoin nodes, TV sets, radios, name it. Currently, allowing the AI some “thinking time'“ so that it runs longer inferences on the same data, is how AI companies are squeezing out some more juice. For example with ChatGPT’s o1 model. But maybe even training time should be extended. Maybe to get great AI, like a child, it needs decades of training from data collected from all over. Collected and slowly siphoned into a condensed AI model in a few servers. Sam Altman might need those trillions of government dollars for such a plan. Alas, they will likely not be easy to get. The D.O.G.E might not bankroll him (if xAI is suffering the same troubles). Meanwhile, Bitcoin is still growing As the above graphic shows, very few people are using Bitcoin. Yet it is already at $90k . already at $90k Data from Bitbo shows that in fact, only 400,000 people are using Bitcoin daily. And only 106 million own Bitcoin. Bitbo Contrast this with the 314 million people who have used AI tools as per data from Statista . ChatGPT alone has millions of users per day . Probably more than the entire Bitcoin blockchain combined! data from Statista has millions of users per day What’s more? AI amassed the bulk of these users in less than a year. This feels wrong. Bitcoin took years to grow. People sweated, lost, DCA’d, had faith. Maybe that’s it. ChatGPT has grown too fast. Now it has slowed down and everybody is sour. Sorry, ya’ll, there is still work to do. AGI will not be so easy to grasp, and GPT scientists should learn to hodl like Bitcoiners. Winter will come and go, but right now, it is time to pay the piper.

The AI Winter is Near, Courtesy of Physics

Der KI-Winter steht vor der Tür, mit freundlicher Genehmigung der Physik

El invierno de la IA se acerca, cortesía de la física

به لطف فیزیک، زمستان هوش مصنوعی نزدیک است

Tekoälytalvi on lähellä, fysiikan luvalla

एआई शीतकाल निकट है, भौतिकी के सौजन्य से

物理学のおかげでAIの冬が近づいている

AI қысы жақын, физиканың рұқсатымен

Зима искусственного интеллекта уже близко. Предоставлено Physics

Iyo AI Chando Yava Pedyo, Nekuda kweFizikisi

Dimri i AI është afër, me mirësjellje të fizikës

AI குளிர்காலம் அருகில் உள்ளது, இயற்பியல் உபயம்

AI qishi yaqin, fizika izni bilan

Financial institutions are investing heavily in AI, with North American banks leading the way in adopting gen AI for enhanced customer service and operational efficiency. The financial and banking services sector has entered a new phase in its digital transformation journey: the Artificial Intelligence (AI) era. While the use of AI in financial services is not new, and AI-powered applications such as trading algorithms and autonomous rule-based credit decision systems have been around for a while— the recent influx of capital from financial institutions to build future-proof AI infrastructure emphasizes the technology’s growing significance.According to a recentIDC report, the financial sector is projected to be the largest spender on AI solutions between 2024 and 2028, with banking at the forefront. The financial services industry is anticipated to account for over 20% of all AI expenditures, representing the highest growth rate among major industries. Today, AI's impact is evident across an array of financial applications, from automated knowledge management to personalized banking services. Specifically, North American banks are heavily investing in generative AI (genAI) to drive innovation and improve operational transparency. GenAI is helping enhance banking and chatbot capabilities, offering experiences that are distinct from traditional automated services. GenAI agents are now available 24/7 in multiple languages, to assist customers with banking queries and account management. To support these efforts, they are acquiring crucial hardware like AI chips and data processing infrastructure, while simultaneously channeling resources to hire human talent. Finance and AI: A Synergy in the Making Financial institutions are among the most data-rich organizations, handling billions of customer transactions and interactions across both digital and physical platforms. This wealth of data is essential to train A.I. systems, which is why banks have been pouring significant resources into maintaining the technology. The goal is to refine existing customer processes and explore high-impact use cases for inculcating A.I., scaling innovative prototypes into robust solutions. For instance, Paris-based Amundi, managing €2 trillion ($2.1 trillion) in assets, utilizes genAI tools to customize investment portfolios for its more than 100 million clients. By engaging clients in discussions about their risk preferences through its chatbot, the asset management firm crafts portfolios that meet individual needs while also gaining real-time insights into client sentiment. “Younger demographics, who will make up an increasingly large percentage of overall target customers for the finance industry - are very comfortable interacting with chatbots and often prefer this option over traditional options like call centers,” Ranjit Tinaikar, CEO of Ness Digital Engineering, told me. “AI can drastically improve the risk insights in pricing consumer loans. The banking industry is tapping into these trends to future-proof the way they deliver customer service and optimize experiences” Renowned North American investment bank and wealth management firm Morgan Stanley, recently introduced its new internal genAI model designed for its financial advisors and support staff. Powered by OpenAI’s GPT-4, the ‘AI @ Morgan Stanley Assistant’ helps thousands of wealth managers to swiftly retrieve information from an extensive internal knowledge base. It can also summarize client meeting notes and automatically draft follow-up emails. Despite its promise, the use of AI for handling sensitive financial decisions does have its drawbacks. AI models’ decision-making abilities are often hindered by unreliable data or unexpected infrastructural errors, which in turn impacts their effectiveness. Challenges and Limitations of AI in Financial Services A.I.'s ability to disrupt industry dynamics appears closely tied to the nature of customer data and the challenges it addresses. In banking service areas like wealth management or lending, where data is less abundant and unstable, A.I.'s impact has been more incremental than transformational. The finance industry’s experience with AI also raises questions about whether AI will democratize or consolidate power within the sector. “Most banking corporations don’t have the requisite organizational expertise and governance to deploy A.I. solutions. GenA.I. is only as smart as the data used to train the large language models (LLMs),” Tinaikar added. “Access to this data, which is often fragmented across internal and external systems, is a major challenge to create reliable financial A.I. models.” It is crucial to ensure that A.I. operates within a secure, ethical, and compliant framework to prevent potentially sensitive data misuse. In this context, larger companies with vast resources for technology and data investments hold a notable edge over their smaller rivals. For small and medium-sized businesses (SMBs), the real challenge to creating smarter and reliable A.I. systems is their inability to afford A.I. experts and invest in security management architectures. “You do need a “human-in-loop” for sensitive decisions around A.I.-generated pricing and risk recommendations,” said Tinaikar. “In the interim, SMB corporates have to develop their self-governance to protect their consumers, investors, and other key stakeholders.” Why Human Judgment Remains Crucial in AI-Enhanced Finance GenA.I. represents a transformative shift, enabling banks to fully harness the power of their customer data. By analyzing structured and unstructured data from diverse sources and understanding the relationships between them, it offers enhanced visibility and insights into both risks and opportunities within and beyond the bank. But, as AI makes data analysis more efficient and cost-effective, the role of human judgment becomes even more crucial. AI systems currently lack the nuance and creativity of human insight, making them unreliable for sensitive use cases. Although AI is an incredibly powerful tool, financial managers and institutions stand to gain the most from combining AI's power with unique human abilities. “The difference lies, of course, in the type of emotional judgment applied by human experts,” explained Tinaikar. “As AI introduces the potential for a disruptive shift where machines could replace human judgment, we need to develop regulatory and technological solutions to ensure that humans remain involved in mission-critical tasks where machine-simulated judgment may not be reliable.” Financial institutions are investing heavily in AI, with North American banks leading the way in adopting gen AI for enhanced customer service and operational efficiency. Financial institutions are investing heavily in AI, with North American banks leading the way in adopting gen AI for enhanced customer service and operational efficiency. The financial and banking services sector has entered a new phase in its digital transformation journey: the Artificial Intelligence (AI) era. While the use of AI in financial services is not new, and AI-powered applications such as trading algorithms and autonomous rule-based credit decision systems have been around for a while— the recent influx of capital from financial institutions to build future-proof AI infrastructure emphasizes the technology’s growing significance. According to a recent IDC report , the financial sector is projected to be the largest spender on AI solutions between 2024 and 2028, with banking at the forefront. The financial services industry is anticipated to account for over 20% of all AI expenditures, representing the highest growth rate among major industries. IDC report IDC report Today, AI's impact is evident across an array of financial applications, from automated knowledge management to personalized banking services. Specifically, North American banks are heavily investing in generative AI ( genAI ) to drive innovation and improve operational transparency. GenAI is helping enhance banking and chatbot capabilities, offering experiences that are distinct from traditional automated services. GenAI agents are now available 24/7 in multiple languages, to assist customers with banking queries and account management. genAI To support these efforts, they are acquiring crucial hardware like AI chips and data processing infrastructure, while simultaneously channeling resources to hire human talent. Finance and AI: A Synergy in the Making Finance and AI: A Synergy in the Making Financial institutions are among the most data-rich organizations, handling billions of customer transactions and interactions across both digital and physical platforms. This wealth of data is essential to train A.I. systems, which is why banks have been pouring significant resources into maintaining the technology. The goal is to refine existing customer processes and explore high-impact use cases for inculcating A.I., scaling innovative prototypes into robust solutions. For instance, Paris-based Amundi , managing €2 trillion ($2.1 trillion) in assets, utilizes genAI tools to customize investment portfolios for its more than 100 million clients. By engaging clients in discussions about their risk preferences through its chatbot, the asset management firm crafts portfolios that meet individual needs while also gaining real-time insights into client sentiment. Amundi Amundi “Younger demographics, who will make up an increasingly large percentage of overall target customers for the finance industry - are very comfortable interacting with chatbots and often prefer this option over traditional options like call centers,” Ranjit Tinaikar, CEO of Ness Digital Engineering, told me. “AI can drastically improve the risk insights in pricing consumer loans. The banking industry is tapping into these trends to future-proof the way they deliver customer service and optimize experiences” Renowned North American investment bank and wealth management firm Morgan Stanley, recently introduced its new internal genAI model designed for its financial advisors and support staff. Powered by OpenAI’s GPT-4, the ‘ AI @ Morgan Stanley Assistant ’ helps thousands of wealth managers to swiftly retrieve information from an extensive internal knowledge base. It can also summarize client meeting notes and automatically draft follow-up emails. AI @ Morgan Stanley Assistant AI @ Morgan Stanley Assistant Despite its promise, the use of AI for handling sensitive financial decisions does have its drawbacks. AI models’ decision-making abilities are often hindered by unreliable data or unexpected infrastructural errors, which in turn impacts their effectiveness. Challenges and Limitations of AI in Financial Services Challenges and Limitations of AI in Financial Services A.I.'s ability to disrupt industry dynamics appears closely tied to the nature of customer data and the challenges it addresses. In banking service areas like wealth management or lending, where data is less abundant and unstable, A.I.'s impact has been more incremental than transformational. The finance industry’s experience with AI also raises questions about whether AI will democratize or consolidate power within the sector. “Most banking corporations don’t have the requisite organizational expertise and governance to deploy A.I. solutions. GenA.I. is only as smart as the data used to train the large language models (LLMs),” Tinaikar added. “Access to this data, which is often fragmented across internal and external systems, is a major challenge to create reliable financial A.I. models.” It is crucial to ensure that A.I. operates within a secure, ethical, and compliant framework to prevent potentially sensitive data misuse. In this context, larger companies with vast resources for technology and data investments hold a notable edge over their smaller rivals. For small and medium-sized businesses (SMBs), the real challenge to creating smarter and reliable A.I. systems is their inability to afford A.I. experts and invest in security management architectures. “You do need a “human-in-loop” for sensitive decisions around A.I.-generated pricing and risk recommendations,” said Tinaikar. “In the interim, SMB corporates have to develop their self-governance to protect their consumers, investors, and other key stakeholders.” Why Human Judgment Remains Crucial in AI-Enhanced Finance GenA.I. represents a transformative shift, enabling banks to fully harness the power of their customer data. By analyzing structured and unstructured data from diverse sources and understanding the relationships between them, it offers enhanced visibility and insights into both risks and opportunities within and beyond the bank. But, as AI makes data analysis more efficient and cost-effective, the role of human judgment becomes even more crucial. AI systems currently lack the nuance and creativity of human insight, making them unreliable for sensitive use cases. Although AI is an incredibly powerful tool, financial managers and institutions stand to gain the most from combining AI's power with unique human abilities. “The difference lies, of course, in the type of emotional judgment applied by human experts,” explained Tinaikar. “As AI introduces the potential for a disruptive shift where machines could replace human judgment, we need to develop regulatory and technological solutions to ensure that humans remain involved in mission-critical tasks where machine-simulated judgment may not be reliable.”

AI is Taking Over Your Bank, and You Might Not Even Know It

Holistic Evaluation of Text-to-Image Models: Author contributions, Acknowledgments and References

A Brief Guide to the Governance of Apache Iceberg Tables

The Ultimate Directory of Apache Iceberg Resources

Revolutionizing Virtual Try-On: Key Findings and Future Directions

Latest Advances in Stable Diffusion Technology

Stories, News & Mentions about NESS

Ness’s Stories on HackerNoon