Get the most recent info and news about Docuten on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Soluciones sencillas de factura electrónica, firma digital y voto electrónico. Creadores de Hazteunafacturae.com y signedoc.com, también realizamos integraciones con ERP.

Docuten

docuten

HackerNoon Reporter: Please tell us briefly about your  background. I studied Computer Engineering at the University of A Coruña in Spain and spent several years working at different digital signature and electronic invoicing companies like TB Solutions, SERES, and Invinet before starting my own company. What's your startup called? And in a sentence or two, what does it do? Docuten transforms clients into paperless companies by digitally transforming administrative processes through digital signature, and issuing and receiving electronic invoices. By relying on Docuten as their sole provider, companies can reduce costs by 50% and implementation can be carried out 2.5 times faster than working with several different providers. What is the origin story? After working at a number of digital signature and e-invoicing companies, I saw a need in the market to simplify and streamline digital administrative processes that were often made overly complex. I partnered with my colleague Ángel Aparicio and launched Docuten in 2009 (formerly under the name Enxendra). Over the years, our venture has grown. We now have offices in A Coruña and Madrid, and a rapidly expanding team of more than 35 employees. Our product suite has also evolved to include digital signature, and e-invoicing issuance and reception. What do you love about your team, and why are you the ones to solve this problem? The diversity! Docuten is a super diverse, multi-talented team of individuals from all over that bring different perspectives and approaches to our work. The majority of us have years of experience in our area of expertise and are eager to give our clients the best service possible. Why do clients trust us to improve their administrative processes? There are some big players in the digital signature and e-invoicing markets, but we offer bundled solutions for end-to-end administration digitalisation with first-rate support and customer service. If you weren’t building your startup, what would you be doing? The truth is I can't imagine following another path in life. At the moment, how do you measure success? What are your core metrics? I measure success by two interconnected metrics: client and employee satisfaction. Companies with content employees thrive, and clients can feel that. The goal is to ensure that your clients are happy with the service you are providing, and to have a team that is excited to be at work every day. Our Annual Recurring Revenue growth is demonstrative of a high level of client satisfaction:  between 2019 and 2020 Docuten had a 38% growth in ARR. Last year, we brought in 75% more clients that bill over 3000€ annually. With that type of upsurge, we are predicting even more growth in 2021. In fact, this year Docuten has a 72% increase in billing as compared to the same period in 2020; as of July 2021, our ARR is 1.585.945€. What’s most exciting about your traction to date? What’s been most exciting is expanding our client base into different sectors and getting to learn more about (and address the challenges of) diverse industries. We currently have a strong foothold in the industrial sector as well as in construction, but have clients across the board: in fashion, renting, hospitality, food, etc. Docuten also serves a variety of different departments like HR, legal, sales, and finance. Our growing knowledge about these distinct industries and business areas has proven to be invaluable in providing the best service possible to our clients. What technologies are you currently most excited about, and most worried about? And why? In the world of administrative digitalisation, I think OCR technology is pretty exciting. Docuten offers various means for our clients to receive invoices from vendors. Unfortunately, some vendors are less enthusiastic about having to send invoices electronically. That is where our OCR technology comes in. Vendors can easily send invoices in PDF format as an email attachment to the client’s Docuten mailbox. Our OCR reader receives the information from the PDF invoice, structures it, and indexes it directly in the client’s systems. Our OCR technology has a 99.6% accuracy! Alternatively, some advances (especially at the intersection of medicine and technology) bring certain socio-cultural implications that could prove controversial, which is worrisome. What drew you to get published on HackerNoon? What do you like most about our platform? It is important to keep your finger on the pulse of the newest technologies and innovations out there. HackerNoon is a great way to connect to tech professionals all over the world and learn about exciting up-and-coming startups. We are honored to have Docuten nominated as one of HackerNoon’s Startups of the Year alongside other leading companies. What advice would you give to the 21-year-old version of yourself? Believe in yourself and never give up. It might sound cliché but it’s so important to have faith in your ideas and abilities. Also, surround yourself with great people. Docuten wouldn’t be where it is today without our amazing team. What is something surprising you've learned this year that your contemporaries would benefit from knowing? Remote work is the future. I’m obviously not the only one who has come to this conclusion after the events of last year, but it’s worth highlighting. Our entire team has been working remotely since March of 2020. More recently, as restrictions have been lifted, some employees have returned to the office​. Due to the nature of our work, as well as the fact that Docuten has offices in various locations and remote work policies were in place prior to the pandemic, it has been a smooth transition to a digital workplace. Docuten’s product suite has also enabled the majority of our clients to continue working during this period since transactions that were traditionally done on paper and/or in-person can now be handled remotely and electronically. Our services have been crucial for many of the companies we work with. Docuten was nominated as in Startups of the Year hosted by HackerNoon. Go vote for them! one of the best startups in A Coruña

"Companies With Content Employees Thrive, And Clients Can Feel That", Brais Méndez, Docuten CEO

240 Stories To Learn About Startups Of The Year

206 Stories To Learn About Digital Transformation

Let's learn about via these 63 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Company Culture /Learn Repo 1. Ex-Twitter Employee Rejected at New Job For Asking 'Political Questions' Think it's easy finding a job after Twitter? Think again. 2. Why 2022 Will Be the Year of the DAOs DAO developers believe that they can eliminate human manipulation or error from how companies work. This way, decisions are geared towards profitability alone. 3. Developers Created a Culture of Shame and a Toxic Workplace Open up Twitter or any dev forum really, have a little scroll, and it will not take you long at all to find developers being unpleasant to each other. 4. "The 4-Day Work Week" and other Insights on Bootstrapped Business with GooseChase CEO Andrew Cross Inspired by scavenger hunts, GooseChase is an online platform that enables you to create exceptional experiences for your communities. Meet CEO Andrew Cross. 5. Beyond Fame and Parties: What top Engineers Really Want From a Startup 8 things that make engineers today say, “this place is for me!” 6. Ageism in Tech: How It Feels To Be The Oldest in the Room The phone interview started with the founder explaining the format, followed by the standard “tell me about yourself.” Before the interview, I had done research to know he wasn’t yet 30. It seemed to be his first startup and publicly available information about this venture was minimal. I was excited that I got the phone interview because I knew my previous marketing experience was a perfect fit. I kept my story short by including only relevant startup marketing projects. 7. MashRadar Reviewed HackerNoon - Everything You Need to Know Hacker Noon is a tech-focused news website that features content from over 10,000 contributors. 8. Fun at Work Matters: Building a Company Culture Having non-productive fun with colleagues at work is a great way of relaxing during work and getting to know the team. 9. Remote Working and Why I Wouldn't Relocate for a Job Again Here is a list of reasons why I won't relocate just because of a job offer. 10. Productivity Hacks: How to Keep Your Employees Motivated Employee wellbeing is a challenge in this world as i89% of employees have worked while feeling unwell. 11. TekPon is Offering "Free" Marketing Services Of $1.5m To Newly Launched Software Companies A tech start-up is offering free marketing services of $1.5m to newly launched software companies 12. How Digital Transformation Impacts Your Company Culture The successful implementation of digital transformation in the enterprise depends largely on the willingness of the corporate culture to change. 13. Why Working in a Startup Is More Rewarding Than a Career in Big-Tech Five reasons of why working in a startup might be more rewarding than position in a big-name company for specialists, who are focused on career achievements. 14. How to Protect Myself Against Sexism in the Workplace In the era of #metoo and Harvey Weinstein, people are becoming more and more aware of harassment and discrimination in the workplace. Society should thank the brave women who stood up and created a better culture for us, but sexism still happens. It can range from unwanted physical advances to unfair pay to getting passed up for a promotion. The work isn’t done yet. We need to continue to fight to make things equal and safe for women in the workplace. To contribute fighting sexism in society, we need to start at our own workplaces. Here are some tips on how to protect yourself from sexism in the workplace. 15. Pandemic Remote Work != Remote Work Have you switched to remote work in the past couple of weeks/days? Well, chances are high you have, or you will. I want to share my words of caution and tips to make pandemic remote work better. 16. 4 Routes to Remarkable Company Culture Everything I learnt about how to do company culture right I learnt in a tiny, cramped office at an up and coming publishing startup in Farringdon Road, London. 17. How to Prevent Burnout and Keep Your Organization on Track As a result of significant events against me, I burned out in 2016. Here are the keys to overcoming (and potentially protecting staff/colleagues from) burnout. 18. What it Really Means to Hire Slow, Fire Fast A look at what it truly means to practically incorporate hire slow, fire fast within your organization. 19. The 10x value isn't all originating from the engineer, though There's been a lot of discussion on Twitter recently about so-called 10x engineers. 20. How to Preserve Your Remote Company Culture Promoting socializing, resisting surveillance, and hosting idea sessions can help to preserve your remote company culture. 21. 6 Questions Devs Should Ask Interviewers To Discern Culture If you want to learn about growing a digital product without targeted advertising, subscribe to my newsletter Engineering Growth 22. The One Objective to Establish a Winning Startup Culture: Get Better Everyday Last week I had a conversation with a couple of friends about the issues they've been facing at their startup. These friends, like many others, have been affected by CoViD-19 related business impact, as well as the Work From Home situation. As you can guess, the biggest issues have been around communication, specially when it involves some sort of negotiation about what product features to prioritise, and how much effort to put into the same. 23. "Companies With Content Employees Thrive, And Clients Can Feel That", Brais Méndez, Docuten CEO B​rais Méndez, CEO of Docuten, discusses the company's beginnings, what the future holds, and exciting innovations in digital signature and e-invoicing. 24. Why Outsourcing Core Business Functions Can Be Dangerous Outsourcing a core business function, no matter how big and strong a company is, can be a very serious decision. 25. Why It’s a Good Idea for Developers to Work with Startups Instead of for Them What if I told you, it’s possible to get the best of both worlds – to capture the advantages of working with a startup while mitigating the downsides? 26. Twitter Employees Fear Elon Musk's Takeover Will Ruin Company Culture Despite being a longtime lightning rod for controversy, even Twitter employees were caught off-guard by recent events. 27. Does Hiring for Culture Affect Diversity? Can your company hire for both diversity and culture? Learn how to prioritize and balance both through talent intelligence and psychological data. 28. How to Overcome Employee Resistance to Change Company Culture (An Infographic) Focusing on cultural transformation can bring big rewards, including an 85% net profit increase in just five years. But employees sometimes resist this change. 29. Managing Workplace Emotions During Organizational Transformation For a long time, organizations have been analyzed as systems and remained mainly indifferent to how various organization members feel about information. 30. Do Away With Command and Control: How to Change Your Work Culture Despite articles and studies claiming it dead, command and control remain in around nine out of ten organizations. 31. What Whiplash Teaches Us About the "Sink or Swim" Method in Engineering I love movies - Particularly ones that can be referenced to bring a point home in a professional discussion. Such is the case with the movie “Whiplash” and the debate over the “sink or swim” methodology in engineer onboarding. 32. "They're The Ones Signing The Check..." Signing the check
The good and the bad 33. How Cultural Sector Can Help The Business Industry Working with people from other fields may help artists and thinkers contribute to the corporate world in useful ways that the current paradigm does not allow. 34. How Companies Should Act When Employees Work Two Full-Time Jobs The pandemic has changed the rules of the game, influencing how we take care of our health, our leisure, and especially how we work. 35. Demystifying Culture - What Does Culture Really Mean? Annie is a software engineering leader on Square’s platform & infrastructure engineering organization. Prior to Square, she worked at a number of startups across a spectrum of industries from consumer products to enterprise solutions, as well as a wide variety of teams from sales to engineering. 36. The Importance Of Staying Humble And Strong Company Culture: Stephen Goldberg, HarperDB CEO Harper DB is a startups of the year nominee from Denver, Colorado. Stephen Goldberg, CEO, shares the origin story and what makes their company culture relevant. 37. How the Noonies is Like the Dundies [or at least I told my team that] Managing people, meh. Building teams, now that's fun. It is really interesting to think about how individuals can make others better. 38. Why Cybersecurity Must Start with Company Culture Cyberattacks, though frequently thought of as ‘sophisticated’, are often anything but. They are actually surprisingly straightforward and mostly financially motivated. We often see the media reporting on large-scale cyberattacks that have managed to breach complex security protocols, at a substantial financial cost to the business involved. 39. Deglorifying CEOs Bright CEOs belong in the past. Web3 is talent-centric and doesn't need a "charismatic leader" to become successful. Well, are we right, or are we right? 40. Omnipresent Requests That All Employees Return to the Office: The Best April Fools Prank Ever? What would you do if your employer suddenly announced that everyone needed to work from the office? Just kidding, it's just a really good joke. 41. Startup Interview with Dmitry Bubnov, CEO Of ENBISYS ENBISYS CEO shares his tips on running software development company and how his team manages to stay abreast of technology development. 42. 5 Ways to Future-Proof Your Brand Your company’s brand is an essential yet intangible asset that narrates how your audience views your organization. It’s no longer sufficient to merely develop your branding. To be sustainable, you must also future-proof your brand, ensuring that you evolve with industry trends and stay relevant to a digital audience. 43. Improve Team Dynamics is Key to Better Workplace Efficiency Understanding group dynamics and how team-building, when done correctly, can dramatically improve workplace efficiency. 44. HackerNoon and the Time We Talked About Pints? HackerNoon is trying to settle a debate: do the international (non-UK) folks order a “pint” of beer, or just generally connect a “pint” with drinking? 45. Proper Communication Helps Make Digital Transformation Smooth The digital transformation process of the business. The role of communication during the fundamental change. Agility as the locomotive of transformation. 46. 7 Ways to Improve Internal Communication to Foster Better Corporate Culture Here are some of the best practices for developing an internal communication strategy that makes sense for your organization. 47. How to Help Your Employees Maintain Work-life Balance in a Hybrid Landscape Companies need to adapt to the changes necessary in a hybrid workspace to set employees up for success. 48. Subtle Culture Problems That Often Go Unnoticed When Peter Drucker said in 2006 that “Culture eats strategy for breakfast,” he did not mean that strategy is not important. Rather an effective strategy is crucial to organizational success. A good strategy is the key to successfully implementing the vision. But a strategy by itself cannot push any organization to achieve tremendous growth and success. What gives life to that strategy are its people. 49. Building Remote Teams: Lessons From The Gitlab Remote Playbook DevOps giant GitLab are one of the leading fully remote tech companies. They have produced a ‘Remote Playbook’ to guide companies looking to adopt remote workin 50. The Unbundled Work Era: The Time Is Now The internet and rise of micro-entrepreneurship 51. The Disregarded Reality of Burnout Crisis in the Tech Industry A self-help blog on how to survive the industry pressure and revolutionize the toxic work culture into something that might become sustainable eventually. 52. 5 Simple Strategies To Build Trust in Remote Teams Before we attempt to answer the challenge of how to build trust over Zoom, Loom, Slack. Let’s try to analyze what built trust 53. Treating Your Internet Friends with Respect Starts Right Here at HackerNoon HQ TL;DR: All Hacker Noon full time employees and long-term part-time contractors are now eligible to enter our fully-funded Mental Health initiative. Plus a few tips on how to make Remote Work work. 54. How to Set Up a Team's Systems and Culture for Strong Code Reviews All software companies must take an intentional and structured approach to code reviews. Here are some tips on how to do that. 55. Office Inferno: Why Aren't We Happy At Work? I just finished watching “The Office”. Funny on the surface, but very satirical and even dark once you go a layer below. 56. Implement These 21 Employee Wellness Program Ideas For Better Retention Employee wellness programs have become a staple in many companies as a way to attract top talent, keep them happy and productive, and decrease employee turnover. 57. Some Tips On Holding a Productive Virtual Offsite Offsites are a big part of remote teams. They allow everyone to socialize, connect more deeply with coworkers, and help build shared experiences and empathy. Even if video calls are a great tool to share information, they can be tiring. It's too easy to miss non-verbal cues. Chance encounters over coffee never happen, and we don’t always experience the same personal connections that come from small-talk. That’s the downside of remote work, and that's why meeting a few times a year is a great way to fill the gap! 58. 5 Things To Ask About the Engineering Team to Avoid Taking a Job From Hell Anyone with enough job experience knows that some jobs can leave you completely exhausted at the end of the day — while others make you excited, proud, and willing to do more. Sometimes people experience these opposite effects from jobs that are in fact quite similar in terms of responsibilities. So what can account for this difference? 59. Secrets of High-Performing Teams: Part II A robust DevOps team facilitates faster development of new products. 60. Ash Coleman, Head of DEI at Credit Karma, talks about Diversity, Equity, and Inclusion at Work Federico Toledo, wraps up season three of Quality Sense with an illuminating interview with a leader in the DEI space. 61. Team Building Activities In The Workplace Help To Establish Trust Let’s take a look at some benefits of team building activities and games at the workplace. 62. Everyone Is Doing Remote Work but Not Everyone Is Doing It Well with Michael Brooks On this episode of the HackerNoon podcast, Amy Tom and Michael Brooks get into everything freelance. They dive into https://golance.com, Michael’s freelance marketplace company. How it came about and Michael’s insights on running it. They talk about wild experiences like getting kicked out of Belarus and taking a shot of Brandy first thing every morning before work in Serbia. It’s a wild ride and you definitely should tune in! Brandy or coffee in hand, your choice ;) 🍻 63. Improving Communication With Global Cultures Using Contextual Communication If we wish to communicate effectively, we must understand how culture affects communication. Here's a communication design to avoid conflicts and uncertainty. Thank you for checking out the 63 most read stories about Company Culture on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

63 Stories To Learn About Company Culture

If you are here, that means you must have seen my PR from the Google MLKit Flutter package, which was merged recently. If you have not, you can check it out here #605. Google ML Kit for Flutter The Google Machine Language Kit for Flutter is a set of Flutter plugins that enable Flutter apps to use Google’s standalone ML Kit. Presently, 13 plugins are available, most commonly face detection, text recognition, barcode scanning, selfie segmentation, and so on. So, let’s talk about the new plugin Document Scanner. Digitizing physical documents, which allows users to convert physical documents into digital formats, has become a very common user journey in mobile apps. The Document Scanner provides a solution with high-quality, orderly UI flow across Android Apps. You can apply filters, remove stains and shadows, and crop documents. Read more #here. Some of the key capabilities are: High-quality and consistent user interface for digitizing physical documents.
Automatic capture with document detection.
No camera permission is needed from your app. Note: Document Scanner is not yet support for iOS and is currently on Beta mode Using the package on your flutter app is much easy as you think. you just have to add the plugin yo your pubspec.yaml as followed: dependencies:
  google_mlkit_document_scanner: ^0.2.0 Or you can run this command on your terminal to add the package flutter pub add google_mlkit_document_scanner We cannot go ahead with using the package in our app. Create an instance of the DocumentOption class. DocumentScannerOptions options = DocumentScannerOptions(
      mode: ScannerMode.filter, // to control the feature sets in the flow
      isGalleryImport: false, // importing from the photo gallery
      pageLimit: 1, // setting a limit to the number of pages scanned
 ); Mode is to change the scanner mode, which can be ScannerMode*.full, ScannerMode.filter or ScannerMode.base*


IsGalleryImport if you want to allow gallery import set to true


pageLimit to set the number of pages scanned


DocumentFormat to be retrieved after the scanned, which can be in **jpeg or pdf
Create an instance of the DocumentScanner class and pass the options. DocumentScanner documentScanner = DocumentScanner(
    options: options
  ); Create a List to store the path of the scanned documents. List<String>? documents; Don’t forget to dispose. @override
  void dispose() {
    documentScanner.close();
    super.dispose();
  } To start the scan, all you have to do is call the scanDocument method like so. documents = await documentScanner.scanDocument(); Easy-peasy, now the documents list contains the paths to the document scanned. You can get all code sources from below https://github.com/bensonarafat/document_scanner?source=post_page-----e3d978ef2957--------------------------------&embedable=true If you have any questions or issues, please add them to the comment section. I’m open to discussing any Flutter project opportunities, big or small. Also, feel free to follow me on Twitter @bensonarafat for updates and stay connected. Adios! 👋 If you are here, that means you must have seen my PR from the Google MLKit Flutter package, which was merged recently. If you have not, you can check it out here #605 . #605 #605 Google ML Kit for Flutter Google ML Kit for Flutter The Google Machine Language Kit for Flutter is a set of Flutter plugins that enable Flutter apps to use Google’s standalone ML Kit. Presently, 13 plugins are available, most commonly face detection , text recognition , barcode scanning , selfie segmentation, and so on. Google Machine Language Kit for Flutter Google Machine Language Kit for Flutter face detection face detection text recognition text recognition barcode scanning barcode scanning selfie segmentation , selfie segmentation So, let’s talk about the new plugin Document Scanner. Document Scanner. Document Scanner. Document Scanner. Digitizing physical documents, which allows users to convert physical documents into digital formats, has become a very common user journey in mobile apps. The Document Scanner provides a solution with high-quality, orderly UI flow across Android Apps. You can apply filters, remove stains and shadows, and crop documents. Read more #here. #here. #here. Some of the key capabilities are: High-quality and consistent user interface for digitizing physical documents. Automatic capture with document detection. No camera permission is needed from your app. High-quality and consistent user interface for digitizing physical documents. Automatic capture with document detection. No camera permission is needed from your app. Note: Document Scanner is not yet support for iOS and is currently on Beta mode Note: Document Scanner is not yet support for iOS and is currently on Beta mode Note: Document Scanner is not yet support for iOS and is currently on Beta mode Using the package on your flutter app is much easy as you think. you just have to add the plugin yo your pubspec.yaml as followed: pubspec.yaml dependencies:
  google_mlkit_document_scanner: ^0.2.0 dependencies:
  google_mlkit_document_scanner: ^0.2.0 Or you can run this command on your terminal to add the package flutter pub add google_mlkit_document_scanner flutter pub add google_mlkit_document_scanner We cannot go ahead with using the package in our app. Create an instance of the DocumentOption class. DocumentScannerOptions options = DocumentScannerOptions(
      mode: ScannerMode.filter, // to control the feature sets in the flow
      isGalleryImport: false, // importing from the photo gallery
      pageLimit: 1, // setting a limit to the number of pages scanned
 ); DocumentScannerOptions options = DocumentScannerOptions(
      mode: ScannerMode.filter, // to control the feature sets in the flow
      isGalleryImport: false, // importing from the photo gallery
      pageLimit: 1, // setting a limit to the number of pages scanned
 ); Mode is to change the scanner mode, which can be ScannerMode*.full, ScannerMode.filter or ScannerMode.base* IsGalleryImport if you want to allow gallery import set to true pageLimit to set the number of pages scanned DocumentFormat to be retrieved after the scanned, which can be in **jpeg or pdf
Create an instance of the DocumentScanner class and pass the options. Mode is to change the scanner mode, which can be ScannerMode*.full, ScannerMode.filter or ScannerMode.base* Mode is to change the scanner mode, which can be ScannerMode *. full , ScannerMode . filter or ScannerMode . base * Mode ScannerMode ScannerMode full ScannerMode filter ScannerMode base IsGalleryImport if you want to allow gallery import set to true IsGalleryImport if you want to allow gallery import set to true IsGalleryImport pageLimit to set the number of pages scanned pageLimit to set the number of pages scanned pageLimit DocumentFormat to be retrieved after the scanned, which can be in **jpeg or pdf Create an instance of the DocumentScanner class and pass the options. DocumentFormat to be retrieved after the scanned, which can be in ** jpeg or pdf DocumentFormat jpeg or pdf Create an instance of the DocumentScanner class and pass the options. DocumentScanner documentScanner = DocumentScanner(
    options: options
  ); DocumentScanner documentScanner = DocumentScanner(
    options: options
  ); Create a List to store the path of the scanned documents. List<String>? documents; List<String>? documents; Don’t forget to dispose. @override
  void dispose() {
    documentScanner.close();
    super.dispose();
  } @override
  void dispose() {
    documentScanner.close();
    super.dispose();
  } To start the scan, all you have to do is call the scanDocument method like so. scanDocument scanDocument documents = await documentScanner.scanDocument(); documents = await documentScanner.scanDocument(); Easy-peasy, now the documents list contains the paths to the document scanned. documents documents You can get all code sources from below https://github.com/bensonarafat/document_scanner?source=post_page-----e3d978ef2957--------------------------------&embedable=true https://github.com/bensonarafat/document_scanner?source=post_page-----e3d978ef2957--------------------------------&embedable=true If you have any questions or issues, please add them to the comment section. I’m open to discussing any Flutter project opportunities, big or small. Also, feel free to follow me on Twitter @bensonarafat for updates and stay connected. Adios! 👋

 Using the New Google ML Kit Document Scanner for Flutter Apps

is an open-source search and analytics engine based on Apache Lucene. When building applications on change data capture (CDC) data using Elasticsearch, you’ll want to architect the system to handle frequent updates or modifications to the existing documents in an index. Elasticsearch In this blog, we’ll walk through the different options available for updates including full updates, partial updates, and scripted updates. We’ll also discuss what happens under the hood in when modifying a document and how frequent updates impact CPU utilization in the system. Elasticsearch Example Application with Frequent Updates To better understand use cases that have , let’s look at a search application for a video streaming service like Netflix. When a user searches for a show, ie “political thriller”, they are returned a set of relevant results based on keywords and other metadata. frequent updates Let’s look at an example document in Elasticsearch of the show “House of Cards”: {
	"name": "House of Cards",
	"description": "Frank Underwood is a Democrat appointed as the Secretary of State. Along with his wife, he sets out on a quest to seek revenge from the people who betrayed him while successfully rising to supremacy.",
	"genres": ["drama", "thriller"],
	"views": 100,
} The search can be configured in Elasticsearch to use and as full-text search fields. The field, which stores the number of views per title, can be used to boost content, ranking more popular shows higher. The field is incremented every time a user watches an episode of a show or a movie. name description views views When using this search configuration in an application the scale of , the number of updates performed can easily cross millions per minute as determined by the . According to the Report, users watched ~100 billion hours of content from January to July. Assuming an average watch time of 15 minutes per episode or a movie, the number of views per minute reaches 1.3 million on average. With the search configuration specified above, each view would require an update in the millions scale. Netflix Netflix Engagement Report Many search and analytics applications can experience frequent updates, especially when built on CDC data. Performing updates in Elasticsearch Let’s delve into a general example of how to perform an update in Elasticsearch with the code below: -
from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID you want to update
index_name = 'movies'
document_id = 'your_document_id'

# Retrieve the current document to get the current 'views' value
try:
    current_doc = es.get(index=index_name, id=document_id)
    current_views = current_doc['_source']['views']
except Exception as e:
    print(f"Error retrieving current document: {e}")
    current_views = 0  # Set a default value if there's an error

# Define the update body to increment 'views' by 1
update_body = {
    "doc": {
        "views": current_views + 1  # Increment 'views' by 1
    }
}

# Perform the update
try:
    es.update(index=index_name, id=document_id, body=update_body)
    print("Document updated successfully!")
except Exception as e:
    print(f"Error updating document: {e}") Full Updates versus Partial Updates in Elasticsearch When performing an update in Elasticsearch, you can use the to replace an existing document or the to make a partial update to a document. index API update API The index API retrieves the entire document, makes changes to the document, and then reindexes the document. With the update API, you simply send the fields you wish to modify, instead of the entire document. This still results in the document being reindexed but minimizes the amount of data sent over the network. The update API is especially useful in cases where the document size is large and sending the entire document over the network will be time-consuming. Let’s see how both the index API and the update API work using Python code. Full updates using the index API in Elasticsearch from elasticsearch import Elasticsearch

# Connect to Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID
index_name = "your_index"
document_id = "1"

# Retrieve the existing document
existing_document = es.get(index=index_name, id=document_id)

# Make your changes to the document
existing_document["_source"]["field1"] = "new_value1"
existing_document["_source"]["field2"] = "new_value2"

# Call the index API to perform the full update
es.index(index=index_name, id=document_id, body=existing_document["_source"]) As you can see in the code above, the index API requires two separate calls to Elasticsearch which can result in slower performance and higher load on your cluster. Partial updates using the update API in Elasticsearch Partial updates internally use the , but have been configured to only require a single network call for better performance. reindex API from elasticsearch import Elasticsearch

# Connect to Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID
index_name = "your_index"
document_id = "1"

# Specify the fields to be updated
update_fields = {
    "field1": "new_value1",
    "field2": "new_value2"
}

# Use the update API to perform a partial update
es.update(index=index_name, id=document_id, body={"doc": update_fields}) You can use the update API in Elasticsearch to update the view count but, by itself, the update API cannot be used to increment the view count based on the previous value. That is because we need the older view count to set the new view count value. Let’s see how we can fix this using a powerful scripting language, Painless. Partial updates using Painless scripts in Elasticsearch is a scripting language designed for Elasticsearch and can be used for query and aggregation calculations, complex conditionals, data transformations and more. Painless also enables the use of scripts in update queries to modify documents based on complex logic. Painless In the example below, we use a Painless script to perform an update in a single API call and increment the new view count based on the value of the old view count. from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID you want to update
index_name = 'movies'
document_id = 'your_document_id'

# Define the Painless script for the update
update_script = {
    "script": {
        "lang": "painless",
        "source": "ctx._source.views += 1"  # Increment 'views' by 1
    }
}

# Perform the update using the Painless script
try:
    es.update(index=index_name, id=document_id, body=update_script)
    print("Document updated successfully!")
except Exception as e:
    print(f"Error updating document: {e}") The Painless script is pretty intuitive to understand, it is simply incrementing the view count by 1 for every document. Updating a Nested Object in Elasticsearch in Elasticsearch are a data structure that allows for the indexing of arrays of objects as separate documents within a single parent document. Nested objects are useful when dealing with complex data that naturally form a nested structure, like objects within objects. In a typical Elasticsearch document, arrays of objects are flattened, but using the nested data type allows each object in the array to be indexed and queried independently. Nested objects Painless scripts can also be used to update nested objects in Elasticsearch. from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID for the example
index_name = 'your_index'
document_id = 'your_document_id'

# Specify the nested field and the updated value
nested_field = "nested_field_name"
updated_value = "new_value"

# Define the Painless script for the update
update_script = {
    "script": {
        "lang": "painless",
        "source": "ctx._source.nested_field_name = params.updated_value",
        "params": {
            "updated_value": updated_value
        }
    }
}

# Perform the update using the Update API and the Painless script
try:
    es.update(index=index_name, id=document_id, body=update_script)
    print("Nested object updated successfully!")
except Exception as e:
    print(f"Error updating nested object: {e}") Adding a New Field in Elasticsearch Adding a new field to a document in Elasticsearch can be accomplished through an index operation. You can partially update an existing document with the new field using the Update API. When dynamic mapping on the index is enabled, introducing a new field is straightforward. Simply index a document containing that field and Elasticsearch will automatically figure out the suitable mapping and add the new field to the mapping. With dynamic mapping on the index disabled, you will need to use the update mapping API. You can see an example below of how to update the index mapping by adding a “category” field to the movies index. PUT /movies/_mapping
{
  "properties": {
    "category": {
      "type": "keyword"
    }
  }
} Updates in Elasticsearch Under the Hood While the code is simple, Elasticsearch internally is doing a lot of heavy lifting to perform these updates because data is stored in immutable segments. As a result, Elasticsearch cannot simply make an in-place update to a document. The only way to perform an update is to reindex the entire document, regardless of which API is used. Elasticsearch uses Apache Lucene under the hood. A Lucene index is composed of one or more segments. A segment is a self-contained, immutable index structure that represents a subset of the overall index. When documents are added or updated, new Lucene segments are created and older documents are marked for soft deletion. Over time, as new documents are added or existing ones are updated, multiple segments may accumulate. To optimize the index structure, Lucene periodically merges smaller segments into larger ones. Updates are essentially inserts in Elasticsearch Since each update operation is a reindex operation, all updates are essentially inserts with soft deletes. There are cost implications for treating an update as an insert operation. On one hand, the soft deletion of data means that old data is still being retained for some time, bloating the storage and memory of the index. Performing soft deletes, reindexing and garbage collection operations also take a heavy toll on the CPU, a toll that is exacerbated by repeating these operations on all replicas. Updates can get more tricky as your product grows and your data changes over time. To keep Elasticsearch performant, you will need to update the shards, analyzers, and tokenizers in your cluster, requiring a reindexing of the entire cluster. For production applications, this will require setting up a new cluster and migrating all of the data over. Migrating clusters is both time-intensive and error prone so it's not an operation to take lightly. Updates in Elasticsearch The simplicity of the update operations in Elasticsearch can mask the heavy operational tasks happening under the hood of the system. Elasticsearch treats each update as an insert, requiring the full document to be recreated and reindexed. For applications with frequent updates, this can quickly become expensive as we saw in the Netflix example where millions of updates happen every minute. We recommend either batching updates using the , which adds latency to your workload, or looking at alternative solutions when faced with frequent updates in Elasticsearch. Bulk API Rockset, a search and analytics database built in the cloud, is a mutable alternative to Elasticsearch. Being built on , a key-value store popularized for its mutability, Rockset can make in-place updates to documents. This results in only the value of individual fields being updated and reindexed rather than the entire document. RocksDB If you’d like to compare the performance of Elasticsearch and Rockset for update-heavy workloads, you can start a with $300 in credits. free trial of Rockset

A Simple Guide for Updating Documents in Elasticsearch

ইলাস্টিকসার্চে ডকুমেন্ট আপডেট করার জন্য একটি সহজ গাইড

Eine einfache Anleitung zum Aktualisieren von Dokumenten in Elasticsearch

Una guía sencilla para actualizar documentos en Elasticsearch

Un guide simple pour mettre à jour des documents dans Elasticsearch

इलास्टिकसर्च में दस्तावेज़ों को अपडेट करने के लिए एक सरल गाइड

Elasticsearch でドキュメントを更新するための簡単なガイド

Elasticsearch에서 문서 업데이트를 위한 간단한 가이드

Um guia simples para atualização de documentos no Elasticsearch

Простое руководство по обновлению документов в Elasticsearch

Elasticsearch'te Belgeleri Güncellemek İçin Basit Bir Kılavuz

Hướng dẫn đơn giản để cập nhật tài liệu trong Elaticsearch

Elasticsearch 中更新文档的简单指南

It's true that most software developers are familiar with code-signing certificates, but some newcomers may not fully understand them or how they differ from email-signing documents. While both certificates and documents involve digital signatures, they serve different purposes, and it's crucial to understand the distinction between them. Digital signatures are commonly used in various contexts, with document and code signing being the most prevalent. If you've explored ways to secure your data, you've likely encountered different software and certificates, such as email signing and document signing. If you find yourself unsure about which certificate is suitable for your needs, rest assured that you're not alone. In this article, we'll delve into the differences between these two types of certificates. But before we do that, let's start by covering the basics of document signing and code signing certificates. What is a Code Signing Certificate? Software, scripts, and executables are signed by code-signing certificates to ensure their identity and integrity. The software publisher can digitally sign code with a code signing certificate (for example, you can digitally sign software you download and install). As a result, it provides two important benefits: . Software publisher information is confirmed and displayed Which software would you download if two similar programs appeared on the market, one issued by Apple, and the other by AppleHackingzone? By signing the final version of the code, a developer ensures that the software users download has not been tampered with. Two very significant things are accomplished by code signing. It guarantees the authorship and the developer's identity, preventing a random attacker from passing off malware as a or software patch. It verifies the software again to ensure it wasn't damaged, corrupted, or tampered with during installation. The code must seem exactly as it did when the developer signed it, and this is ensured by comparing the user-generated hash to the developer-generated value. Microsoft Apple The public can still trust a relatively unknown software company if the developers utilize a code signing certificate they obtained from a certificate authority (CA). This is because code signing extends the CA's trust in the developer. Malware assaults have grown in frequency as a result of people spending more time online as a result of the pandemic. Code signing certificates are one of the strategies used by hackers to mask malware as legitimate software, increasing the number of victims. By compromising the victim's private key and certificate, this is accomplished. To avoid this important step, users must sign their code using a highly secure certificate. Benefits of the Code Signing Certificate Digital signatures are applied to programs, files, updates, or executables to verify their authenticity and integrity after installation and execution. Using code signing, Microsoft developers, programmers, and software engineers can prove, for example, that your Windows 10 update came from Microsoft, not a hacker. By downloading a code-signed file, you can ensure you are downloading it from the intended author/publisher instead of some hacker trying to steal your data and information. In essence, it informs you that the code hasn't been altered by a bad actor, allowing you to install and execute it confidently on your computer. What is an Email Signing and Document? Email signing and document certificates are used for digitally signing emails and documents. You may have seen these documents enabled in organizations and different document types like Microsoft Office and Adobe. The certificate shows the identity of the sender while defining the document. The validity and identity of the certificate holder are guaranteed by digital certificates. You include a signature when you send or receive an email or fill out an online form, which can only be unlocked by the recipient's server using their public key. This guarantees that the data is transmitted securely and without being tampered with or changed by anyone. This encryption key creates an unbreakable connection between your web browser and the receiving server that is strictly necessary for safe data submission and to verify the sender's identity. Why consider Email Signing and Documentation? Sign and encrypt emails. Sign Microsoft Office documents electronically Verify your identity with online services How are Code-and Email-Signed Documents Used? In a nutshell, we can say that both are somewhat similar. If we look closely, they are both used to secure data for software packages, emails, or documents. Code-signing certificates and email-signing documents are digital Both have rigorous vetting processes to ensure the authority Both are used to secure users from being attacked Both provide certain periods of validity like one, two, and three years Code Signing Certificate vs. Email Signing & Document - Core Differences Core Differences Code Signing Certificate Email Signing and Documentation Security Secure software, application and device drivers Secure emails, Adobe and Microsoft Office Users Developers and software publishers Organizations, banks, and universities Number of Signings Unlimited Limited or varied depending on the organization Conclusion Code signing certificates and email and document signing are a complex world, and it is important to have the right solution in place to protect your organization from every type of digital fraud. As a result, you must choose the right signature provider to safeguard online communication and experiences.

Code Signing Certificate vs. Email and Document Signing Certificate

In previous articles, I explained how we could . But we started this exploration into MongoDB, covering a lot of the basics, and I wanted to start looking into more interesting aspects of how we use these things together. As I was putting video content together about all of these topics, one thing that caught my attention was the generic methods vs. methods that operated on BsonDocument, and I was curious if the performance was any different. So, to start, I figured we’d look at C# MongoDB insert benchmarks and see if any interesting patterns stand out. use C# and MongoDB to get records inserted Considerations For These C# MongoDB Benchmarks I was chatting with on Twitter the other day about benchmarks that he often posts on social media. It turns out there was another discussion floating around the Twitter-verse, where sync vs. async calls to databases were being debated. The proposal was that for very fast DB queries, the async variations are indeed slower. David Callan Of course, this got the gears turning. I have been writing about and creating videos for MongoDB in C# all week and hinting at performance benchmarks coming. This coupled with my conversation with Dave AND the conversation I saw on Twitter made me think that I had to give this more attention. I speculated it might have something to do with Task vs ValueTask, but I didn’t have a lot of grounds for that. However, I originally wanted to look into these benchmarks because I was curious if using BsonDocument compared to a dedicated DTO (whether it was a struct, class, or record variation) would have different performance characteristics. That meant that I would want to ensure I covered a matrix across: Synchronous Asynchronous Asynchronous with ValueTask Struct Class Record Struct Record Class BsonDocument And with that, I set off to go write some super simple benchmarks! Let’s check them out. The C# MongoDB Insert Benchmarking Code As with all benchmarking we do in C#, is our go-to tool! Make sure you start by installing the BenchmarkDotNet NuGet package. We’ll use this to ensure consistent setup, warmup, runs, and reporting for our MongoDB benchmarks. BenchmarkDotNet Since we’re trying to reduce as many possible external factors as we can for these benchmarks, we’re going to use Testcontainers to run an instance of MongoDB in a Docker container. Of course, interacting with anything outside of our code directly can lead to inconsistencies and more room for error to show up in our results. However, this should help minimize things. You’ll want to get the Testcontainers.MongoDB NuGet package for this as well. You can , but we’ll start with what our entry point looks like: find all the relevant code on GitHub using BenchmarkDotNet.Running;

using System.Reflection;

BenchmarkRunner.Run(
    Assembly.GetExecutingAssembly(),
    args: args); Nice and simple just to kick off the benchmarks. And the benchmarks are the most important part here: using BenchmarkDotNet.Attributes;

using MongoDB.Bson;
using MongoDB.Driver;

using Testcontainers.MongoDb;

[MemoryDiagnoser]
//[ShortRunJob]
[MediumRunJob]
public class InsertBenchmarks
{
    private MongoDbContainer? _container;
    private MongoClient? _mongoClient;
    private IMongoCollection<BsonDocument>? _collection;
    private IMongoCollection<RecordStructDto>? _collectionRecordStruct;
    private IMongoCollection<RecordClassDto>? _collectionRecordClass;
    private IMongoCollection<StructDto>? _collectionStruct;
    private IMongoCollection<ClassDto>? _collectionClass;

    [GlobalSetup]
    public async Task SetupAsync()
    {
        _container = new MongoDbBuilder()
            .WithImage("mongo:latest")
            .Build();
        await _container.StartAsync();
        
        _mongoClient = new MongoClient(_container.GetConnectionString());
        var database = _mongoClient.GetDatabase("test");
        _collection = database.GetCollection<BsonDocument>("test");
        _collectionRecordStruct = database.GetCollection<RecordStructDto>("test");
        _collectionRecordClass = database.GetCollection<RecordClassDto>("test");
        _collectionStruct = database.GetCollection<StructDto>("test");
        _collectionClass = database.GetCollection<ClassDto>("test");
    }

    [GlobalCleanup]
    public async Task CleanupAsync()
    {
        await _container!.StopAsync();
    }

    [Benchmark]
    public async Task InsertOneAsync_BsonDocument()
    {
        await _collection!.InsertOneAsync(new BsonDocument()
        {
            ["Name"] = "Nick Cosentino",
        });
    }

    [Benchmark]
    public async ValueTask InsertOneAsyncValueTask_BsonDocument()
    {
        await _collection!.InsertOneAsync(new BsonDocument()
        {
            ["Name"] = "Nick Cosentino",
        });
    }

    [Benchmark]
    public void InsertOne_BsonDocument()
    {
        _collection!.InsertOne(new BsonDocument()
        {
            ["Name"] = "Nick Cosentino",
        });
    }

    [Benchmark]
    public async Task InsertOneAsync_RecordStruct()
    {
        await _collectionRecordStruct!.InsertOneAsync(new RecordStructDto("Nick Cosentino"));
    }

    [Benchmark]
    public async ValueTask InsertOneAsyncValueTask_RecordStruct()
    {
        await _collectionRecordStruct!.InsertOneAsync(new RecordStructDto("Nick Cosentino"));
    }

    [Benchmark]
    public void InsertOne_RecordStruct()
    {
        _collectionRecordStruct!.InsertOne(new RecordStructDto("Nick Cosentino"));
    }

    [Benchmark]
    public async Task InsertOneAsync_RecordClass()
    {
        await _collectionRecordClass!.InsertOneAsync(new RecordClassDto("Nick Cosentino"));
    }

    [Benchmark]
    public async ValueTask InsertOneAsyncValueTask_RecordClass()
    {
        await _collectionRecordClass!.InsertOneAsync(new RecordClassDto("Nick Cosentino"));
    }

    [Benchmark]
    public void InsertOne_RecordClass()
    {
        _collectionRecordClass!.InsertOne(new RecordClassDto("Nick Cosentino"));
    }

    [Benchmark]
    public async Task InsertOneAsync_Struct()
    {
        await _collectionStruct!.InsertOneAsync(new StructDto() { Name = "Nick Cosentino" });
    }

    [Benchmark]
    public async ValueTask InsertOneAsyncValueTask_Struct()
    {
        await _collectionStruct!.InsertOneAsync(new StructDto() { Name = "Nick Cosentino" });
    }

    [Benchmark]
    public void InsertOne_Struct()
    {
        _collectionStruct!.InsertOne(new StructDto() { Name = "Nick Cosentino" });
    }

    [Benchmark]
    public async Task InsertOneAsync_Class()
    {
        await _collectionClass!.InsertOneAsync(new ClassDto() { Name = "Nick Cosentino" });
    }

    [Benchmark]
    public async ValueTask InsertOneAsyncValueTask_Class()
    {
        await _collectionClass!.InsertOneAsync(new ClassDto() { Name = "Nick Cosentino" });
    }

    [Benchmark]
    public void InsertOne_Class()
    {
        _collectionClass!.InsertOne(new ClassDto() { Name = "Nick Cosentino" });
    }

    private record struct RecordStructDto(string Name);

    private record class RecordClassDto(string Name);

    private struct StructDto
    {
        public string Name { get; set; }
    }

    private class ClassDto
    {
        public string Name { get; set; }
    }
} The benchmark code has as much as possible that we’re not interested in exercising pulled out into the GlobalSetup and GlobalCleanup marked methods. Results For C# MongoDB Insert Benchmarks Cue the drumroll! It’s time to look at our MongoDB benchmark results and do a little bit of an analysis: Here are my takeaways from the benchmark data above: All async variations used ~5KB more than the normal versions of the methods that we had to use. There didn’t seem to be any difference in allocated memory for async vs async value task BUT Gen0 and Gen1 didn’t have any value for *some* of the ValueTask benchmarks — However, not for all of them. It almost looks like ValueTask combined with a struct data type for the insert results in Gen0 and Gen1 with no value, but plain old BsonDocument is an exception to this. The fastest and lowest memory footprint seems to be InsertOne_RecordClass, although the InsertOne_BsonDocument is only a few microseconds off from this. Async versions of the benchmarks seem slower than their normal versions across the board, as well. This seems to be very much aligned with some of Twitter's opening thoughts on Async operations! So, some hypotheses proved/disproved: Async *is* overall worse for very fast DB operations ValueTask doesn’t stand out as a consistent performance optimization in these situations For single items, there’s no big difference in the memory footprint we’re seeing between any of these variations of the data types It’ll be a good exercise to follow up with benchmarking, inserting many items into MongoDB from C#. I think we may start to see some of these variations stand out in different ways once we’re working with collections of items — But this is still a hypothesis that needs to be proven! Wrapping Up C# MongoDB Insert Benchmarks This was a simple investigation of insert benchmarks for MongoDB using C#. Overall, there were some surprises for me, but I still think there’s more investigation to be done when we’re working with multiple records at a time. I truly was a little bit surprised to see async be worse across the board since I figured that perhaps any type of IO would mask the performance impact of async overhead. But this was a fun experiment to try out, and there is more to come! If you found this useful and you’re looking for more learning opportunities, consider and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube Also published . here

Benchmarks for Inserting Documents Using C# for MongoDB

In my recent blogs, I’ve covered many of the basics of working with MongoDB from C#. I highly suggest you check out since it will be a stepping stone before we get into deleting. In this article, I’ll cover how to delete documents from MongoDB in C# and the code examples will assume you have some basic MongoDB filtering knowledge already how you can perform filtering of documents from MongoDB in C# If you want to incorporate MongoDB into your C# application and want to understand how to delete documents, read on! Basics of MongoDB in C# To get started with MongoDB in C#, you’ll need to install the MongoDB driver for C#. This driver provides a high-level API for interacting with MongoDB from your C# code, and it’s . Once you have the driver installed, you can establish a connection to your MongoDB database and start working with documents. as simple as getting the NuGet package installed If you’re using something like MongoDB Atlas and the desktop tool Compass, these should walk you through how you can get your connection string sorted out. However, if you’re using other hosting providers or hosting locally, you’ll need to follow relevant instructions for how to get your connection string set up so that you can connect properly. Unfortunately, I can’t document this for every possibility 🙂 In MongoDB, data is stored in collections, which are analogous to tables in a relational database. Maybe not structurally or how they’re implemented, but conceptually, this is the easiest way to think about them. Each document within a collection is a JSON-like object that can have different fields and values. In SQL databases, we’re used to thinking about rows representing our data, and a single row spans multiple columns in a table. However, in a document database like MongoDB, the “row” is an entire document, which can be hierarchical data. To delete a document from MongoDB, you need to specify the collection and the document you want to delete. Before We Delete Documents From MongoDB In C#… Before getting ahead of ourselves here with deleting documents, it’s important to make sure we understand how filtering works. This is because to delete the correct documents, we need to make sure that we can identify WHICH documents we need to delete! In production, you really need to get this right. If you’re not filtering properly, you could be risking dropping all sorts of incorrect documents from your database… and I don’t even want to think about the fun times ahead for recovering that. If you need a primer on how to filter, you can read . And if that’s not straightforward or aligned with your learning style, you may find of better value: this article on filtering MongoDB records from C# this video on using C# to filter MongoDB records https://youtu.be/2zXvDW2YFcg?embedable=true DeleteOne and DeleteOneAsync in MongoDB The method deletes a single document that matches the specified filter criteria. It’s important to note here that there’s nothing that forces you to write a filter that will match a single document, and if you do intend to potentially match multiple documents, we’ll see more about that in the next section DeleteOne The MongoDB C# driver allows you to write a filter that’s empty to match EVERY document and call . But what would you expect to happen in this case? You’re going to match every document with the filter but the method call will still only delete one document. Is it the document you expect? Probably by definition, no, if you wrote a filter that matches multiple. Try to pay special attention to this! DeleteOne Here’s a simple example of calling the method, and an important callout that there is an async version as well: DeleteOne var filter = Builders<MyDocument>.Filter.Eq("fieldName", "fieldValue");
var result = collection.DeleteOne(filter);

// async version
// var result = await collection.DeleteOneAsync(filter); The result that is returned has a matched and deleted count that we can investigate. It’s worth noting in all of my experience so far that even if your filter matches MANY items, if you call it will at most show you a matched count of one still. As a result, this does not seem to be a reliable way to tell if your filter would have (or did) match multiple items and still only delete one. DeleteOne DeleteMany and DeleteManyAsync in MongoDB Much like the previous example, and behave similarly except that they are intended to be used with and support handling the deletion of multiple documents. will not throw an exception if the filter matched multiple documents, but if your intention is truly to be able to support multiple document deletion, these are the methods to use instead. DeleteMany DeleteManyAsync DeleteOne The code example below shows the sync and async variations: var filter = Builders<MyDocument>.Filter.Gte("fieldValue", minValue);
var result = collection.DeleteOne(filter);

// async version
//var result = await collection.DeleteOneAsync(filter); Like , the result that we have to work with has the matched and deleted counts. However, unlike , the match count will indicate how many items truly did match the filter and not be limited to one at most. Again, use these method variations if you need to support multiple documents matching the filter. DeleteOne DeleteOne FindOneAndDelete and FindOneAndDeleteAsync Methods and the async variation are very much like method variations. These methods will perform a delete of up to one document matching the filter, but the interesting difference is the return value. The return type provides us access to the document that matched the filter and was deleted from MongoDB. FindOneAndDelete DeleteOne Here’s a code example of : FindOneAndDelete var filter = Builders<MyDocument>.Filter.Gte("fieldValue", minValue);
var result = collection.FindOneAndDelete(filter);

// async version
//var result = await collection.FindOneAndDeleteAsync(filter); The result that comes back from this method call is the document that matched the filter and was deleted. This can be especially useful if you need to work with the record information and want to save yourself an extra query and logic to validate the response. If you want to see this method along with the ones from the earlier section in action, you can check out : this video on how to delete documents from MongoDB in C# https://youtu.be/wK2TlqTY3nw?embedable=true Wrapping Up How To Delete Documents From MongoDB In C# And now you have all of the basics covered on how to delete documents from MongoDB in C#! In this article, I covered several variations that are all very similar and only need a filter definition to work with: DeleteOne: deletes up to one single document based on the filter DeleteMany: deletes all of the matching documents based on the filter FindOneAndDelete: deletes up to one single document based on the filter and returns the matching document that was deleted If you found this useful and you’re looking for more learning opportunities, consider , and check out my ! Meet other like-minded software engineers, and ! subscribing to my free weekly software engineering newsletter free videos on YouTube join my Discord community Also published here

Using C# To Delete Documents From MongoDB

Recently I’ve been publishing content on working with MongoDB from C#, and this article continues on that path. If you haven’t already covered , I’d highly suggest you get a basic understanding before continuing. In this article, I’ll cover how to update MongoDB documents in C#, but the code examples will assume you know how to filter what you’re interested in! how you can perform filtering of documents from MongoDB in C# If you’re looking to incorporate MongoDB into your C# application and want to understand how to update documents, read on! Basics of MongoDB in C# To get started with MongoDB in C#, you’ll need to install the MongoDB driver for C#. This driver provides a high-level API for interacting with MongoDB from your C# code, and it’s . Once you have the driver installed, you can establish a connection to your MongoDB database and start working with documents. as simple as getting the NuGet package installed If you’re using something like MongoDB Atlas and the desktop tool Compass, these should walk you through how you can get your connection string sorted out. However, if you’re using other hosting providers or hosting locally, you’ll need to follow relevant instructions for how to get your connection string set up so that you can connect properly. Unfortunately, I can’t document this for every possibility 🙂 In MongoDB, data is stored in collections, which are analogous to tables in a relational database. Maybe not structurally or how they’re implemented, but conceptually, this is the easiest way to think about them. Each document within a collection is a JSON-like object that can have different fields and values. In SQL databases, we’re used to thinking about rows to update, and a single row spans multiple columns in a table. However, in a document database like MongoDB, the “row” is an entire document, which can be hierarchical data. To update a document in MongoDB, you need to specify the collection and the document you want to update. Updating Documents in MongoDB Using C# Before getting ahead of ourselves here with updating, it’s important to make sure we understand how filtering works. This is because to update the correct documents, we need to make sure that we can identify WHICH documents we need to update! If your filter is wrong, you’ll be in for a world of pain — or maybe not pain but a high amount of discomfort as you’re debugging database issues. If you need a primer on how to filter, you can read . And if that’s not straightforward or aligned with your learning style, you may find of better value: this article on filtering MongoDB records from C# this video on using C# to filter MongoDB records https://youtu.be/2zXvDW2YFcg?embedable=true UpdateOne and UpdateOneAsync in MongoDB The method updates a single document that matches the specified filter criteria. You can use the $set operator to update specific fields within the document, or in C# specifically, by calling the Set() method on the update builder. UpdateOne It’s important to note here that there’s nothing that forces you to write a filter that will match a single document. In fact, you can write a filter that’s just empty to match EVERY document and call . By doing that, you’re going to match every document but the method call will still only update one document. Is it the document you expect? Probably by definition, no, if you wrote a filter that matches multiple. Try to pay special attention to this! UpdateOne Here’s a simple example of calling the method, and an important callout that there is an async version as well: UpdateOne var filter = Builders<MyDocument>.Filter.Eq("fieldName", "fieldValue");
var update = Builders<MyDocument>.Update.Set("fieldName", "newValue");

var result = collection.UpdateOne(filter, update);

// async version
// var result = await collection.UpdateOneAsync(filter, update); The result that is returned has a matched and modified count that we can investigate. It’s worth noting in all of my experience so far that even if your filter matches MANY items, if you call it will at most show you a matched count of one still. As a result, this does not seem to be a reliable way to tell if your filter would have (or did) match multiple items and still only updated one. UpdateOne UpdateMany and UpdateManyAsync in MongoDB Much like the previous example, and have *almost* the same behavior — except for one tiny detail, which you probably figured out already. The “many” part. UpdateMany UpdateManyAsync will allow you to take that same approach with your filter and update all of the records that match the filter in the way that you’ve defined your update definition. If you truly expect your filter to be able to match multiple, I would advise using this — otherwise, makes more sense. UpdateMany UpdateOne The code example below shows the sync and async variations: var filter = Builders<MyDocument>.Filter.Gte("fieldValue", minValue);
var update = Builders<MyDocument>.Update.Set("fieldName", "newValue");

var result = collection.UpdateMany(filter, update);

// async version
//var result = await collection.UpdateManyAsync(filter, update); Like , the result that we have to work with has the matched and updated counts. However, unlike , the match count will indicate how many items truly did match the filter and not be limited to one at most. UpdateOne UpdateOne FindOneAndUpdate and FindOneAndUpdateAsync Methods and the async variation are very much like method variations. These methods will perform an update on up to one document matching the filter, but the interesting difference is the return value. The return to a snapshot of the document that matched the filter BEFORE it was updated. FindOneAndUpdate UpdateOne type provides us access Here’s a code example of : FindOneAndUpdate var filter = Builders<MyDocument>.Filter.Gte("fieldValue", minValue);
var update = Builders<MyDocument>.Update.Set("fieldName", "newValue");

var result = collection.FindOneAndUpdate(filter, update);

// async version
//var result = await collection.FindOneAndUpdateAsync(filter, update); The result that comes back from this method call would provide the matching document where would still be whatever value existed before it was set to “newValue”. This can be useful in eliminating a full round-trip to the database if you want to know ahead of time which document is going to be updated. fieldName Check out : this video on these MongoDB document update features in C# https://youtu.be/fnPRmoSlTHE?embedable=true Wrapping Up How To Update MongoDB Documents in C# And now you have all of the basics covered on how to update MongoDB documents in C#! In this article, I covered several variations that are all very similar given that they require a proper filter and a proper update definition to be written: UpdateOne: updates up to one single document based on the filter UpdateMany: updates all of the matching documents based on the filter FindOneAndUpdate: updates up to one single document based on the filter and returns the matching document BEFORE it was updated If you found this useful and you’re looking for more learning opportunities, consider and check out my ! Meet other like-minded software engineers and ! subscribing to my free weekly software engineering newsletter free videos on YouTube join my Discord community Also published . here

Using C# To Update MongoDB Documents

Designed for developers familiar with Azure and generative AI, the guide walks you through the process of harnessing the power of the prebuilt model for document intelligence. Introduction The Microsoft team has recently launched an Open AI challenge in which a developer can learn how to build Azure AI solutions and apps. Prerequisite Experience working with Azure and Azure portals. An understanding of generative AI. Experience in one high-level programming language like C# or Python Getting Started The article demonstrates the use of a Prebuilt Model — Invoice in the document intelligence service. What is a Prebuilt Model — Invoice? Use a predefined model to identify and collect standard elements from particular types of documents. In this instance, we employ the built-in invoice model to process and extract information from an invoice. The guide will cover the following features. The steps to configure the document intelligence service on the Azure Portal. A console application in C# to process a sample document. Reference: Sample document Step 1: Navigate to the Azure Portal Search for Azure Document Intelligence and fill out the following details Step 2: Choose Network Step 3: Assign Identity Enable system-assigned identity to grant the resource access to other existing resources. For this demonstration, we don't require identity. Step 4: Add Tags For this exercise, tag names are not required. But in a production environment, it should be added as it’s a best practice. Step 5: Review & Create Post validation checks by Azure Cloud and proceed with creating the resource. Make sure you review the details entered in the previous steps. Let’s switch to code Please find below the sample document highlighting the fields that will be processed as part of this exercise using C# code. The expected output should be as follows, along with the confidence score Vendor Name: 'CONTOSO LTD.', with confidence 0.93.
Customer Name: 'MICROSOFT CORPORATION', with confidence 0.915.
Invoice Total: '$110', with confidence 0.97. Step 1: Create a console application To test image generation, create a console application in Visual Studio or Visual Studio Code. dotnet new console Step 2: Read the configuration Read the configuration from appsettings.json file IConfiguration config = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();
string? endpoint = config["AzureEndpoint"];
string? apiKey = config["AzureKey"];
string? documentUrl = config["DocumentUrl"]; Step 3: Create a client var cred = new AzureKeyCredential(apiKey);
var client = new DocumentAnalysisClient(new Uri(endpoint), cred); Step 4: Analyze the Invoice Install the NuGet package using the following command dotnet add package Azure.AI.FormRecognizer and add the following code AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, "prebuilt-invoice", fileUri);
await operation.WaitForCompletionAsync();

AnalyzeResult result = operation.Value; Step 5: Display Invoice Information The final step is to display the invoice information to the user using the following code snippet. To keep things straightforward, we’re not displaying every key-value pair returned by the service here. For a comprehensive list of all fields supported and their respective types, please refer to the . Invoice concept page foreach (AnalyzedDocument invoice in result.Documents)
{
    if (invoice.Fields.TryGetValue("VendorName", out DocumentField? vendorNameField))
    {
        if (vendorNameField.FieldType == DocumentFieldType.String)
        {
            string vendorName = vendorNameField.Value.AsString();
            Console.WriteLine($"Vendor Name: '{vendorName}', with confidence {vendorNameField.Confidence}.");
        }
    }


    if (invoice.Fields.TryGetValue("CustomerName", out DocumentField? customerNameField))
    {
        if (customerNameField.FieldType == DocumentFieldType.String)
        {
            string customerName = customerNameField.Value.AsString();
            Console.WriteLine($"Customer Name: '{customerName}', with confidence {customerNameField.Confidence}.");
        }
    }

    if (invoice.Fields.TryGetValue("InvoiceTotal", out DocumentField? invoiceTotalField))
    {
        if (invoiceTotalField.FieldType == DocumentFieldType.Currency)
        {
            CurrencyValue invoiceTotal = invoiceTotalField.Value.AsCurrency();
            Console.WriteLine($"Invoice Total: '{invoiceTotal.Symbol}{invoiceTotal.Amount}', with confidence {invoiceTotalField.Confidence}.");
        }
    }
}
Console.WriteLine("\nAnalysis complete.\n"); Step 6: Run the code Make sure the document URL is added to the appsettings.json file. The URL for the example document is provided below. https://github.com/MicrosoftLearning/mslearn-ai-document-intelligence/blob/main/Labfiles/01-prebuild-models/sample-invoice/sample-invoice.pdf?raw=true Complete Program.cs file // See https://aka.ms/new-console-template for more information


// Build a config object and retrieve user settings.
using Azure;
using Azure.AI.FormRecognizer.DocumentAnalysis;
using Microsoft.Extensions.Configuration;

IConfiguration config = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();
string? endpoint = config["AzureEndpoint"];
string? apiKey = config["AzureKey"];
string? documentUrl = config["DocumentUrl"];

Uri fileUri = new Uri(uriString: documentUrl);

Console.WriteLine("\nConnecting to Forms Recognizer at: {0}", endpoint);
Console.WriteLine("Analyzing invoice at: {0}\n", fileUri.ToString());

// Create the client

var cred = new AzureKeyCredential(apiKey);
var client = new DocumentAnalysisClient(new Uri(endpoint), cred);

// Analyze the invoice
AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, "prebuilt-invoice", fileUri);
await operation.WaitForCompletionAsync();

AnalyzeResult result = operation.Value;

// Display invoice information to the user
foreach (AnalyzedDocument invoice in result.Documents)
{
    if (invoice.Fields.TryGetValue("VendorName", out DocumentField? vendorNameField))
    {
        if (vendorNameField.FieldType == DocumentFieldType.String)
        {
            string vendorName = vendorNameField.Value.AsString();
            Console.WriteLine($"Vendor Name: '{vendorName}', with confidence {vendorNameField.Confidence}.");
        }
    }


    if (invoice.Fields.TryGetValue("CustomerName", out DocumentField? customerNameField))
    {
        if (customerNameField.FieldType == DocumentFieldType.String)
        {
            string customerName = customerNameField.Value.AsString();
            Console.WriteLine($"Customer Name: '{customerName}', with confidence {customerNameField.Confidence}.");
        }
    }

    if (invoice.Fields.TryGetValue("InvoiceTotal", out DocumentField? invoiceTotalField))
    {
        if (invoiceTotalField.FieldType == DocumentFieldType.Currency)
        {
            CurrencyValue invoiceTotal = invoiceTotalField.Value.AsCurrency();
            Console.WriteLine($"Invoice Total: '{invoiceTotal.Symbol}{invoiceTotal.Amount}', with confidence {invoiceTotalField.Confidence}.");
        }
    }
}
Console.WriteLine("\nAnalysis complete.\n"); Complete code on GitHub Make sure to give it a star on GitHub and provide feedback on how to improve the tool further..!! To access the document intelligence sample, move to the directory samples > Azure.OpenAI.DocumentIntelligence GitHub - ssukhpinder/AzureOpenAI: Sample for testing Azure Open AI GPT3 Turbo Model C# Programming🚀 Thank you for being a part of the C# community! Before you leave: If you’ve made it this far, please show your appreciation with a clap and follow the author! 👏️️ Follow us: | | | | | X LinkedIn Dev.to Hashnode Newsletter Tumblr Visit our other platforms: | | | | GitHub Instagram Tiktok Quora Daily.dev More content at C# Programming Also published . here

Taking the Azure Open AI Challenge, Day 5: Azure Document Intelligence

Finding the right AI model to build a workflow around is hard. With so many models available across different platforms, it’s impossible to know where to start or how to find the one that best fits your specific needs. That’s the problem I set out to solve with AIModels.fyi, the only search engine for discovering and comparing AI models from across the web. The site has a simple premise: describe your problem in natural language, and get back a list of AI models that can potentially solve it. For example, you might search for “I need a model that can help upscale my images without losing clarity” or “make my cat look like it’s singing”. The goal is to , not just ML engineers, to find useful AI tools for their projects. make it easy for anyone Under the hood, the search works by taking the user’s query and comparing it to a database of over 240,000 AI models from 7,500+ creators across major platforms like Hugging Face, Replicate, Cerebrium, and DeepInfra. When a user types in a search, the query gets embedded into a vector using OpenAI’s text-embedding-ada-002 model. This query vector is then compared to pre-computed vector representations of the model names, descriptions, and use cases in the database. The most similar models by cosine distance are returned as the search results. Here’s a simplified version of the core search logic I’ve been using for about six months: const fetchData = async (query) => {
  // Embed the search query
  const embeddingResponse = await openAi.createEmbedding({
    model: "text-embedding-ada-002",
    input: query,
  });
  
  const embedding = embeddingResponse.data.data[0].embedding;

// Find the most similar models in the vector database 
  const { data: modelsData, error } = await supabase.rpc("search_models", {
    query_embedding: embedding,
    similarity_threshold: 0.75,
    match_count: 10,
  });
  return modelsData;
}; This vector search approach works reasonably well for surfacing relevant models when there’s a direct overlap between the query terms and the model descriptions. However, it struggles with more abstract, complex, or niche queries where the user’s language doesn’t exactly match the technical terminology in the database. For example, I recently wanted to find a model that could animate a portrait image using an audio voice clip to create a talking head effect. I knew there were some good models out there for this, but my searches for things like “talking portrait from image and audio” weren’t turning up the results I expected. The problem was that my query terms didn’t have enough literal matches with the specific words used in the model descriptions, even though the semantic intent was similar. This got me thinking about ways to improve the search experience to better handle those kinds of abstract or complex queries. How could I make the system smarter at understanding the user’s true intent and mapping it to relevant models in the database? That’s when I came across the idea of Hypothetical Document Embeddings (HyDE) from after reading about it in . this research paper this Medium post Meet Mr. HyDE — The query expander! The key insight behind HyDE is to use a large language model (LLM) to expand the user’s short query into a more detailed hypothetical “ideal” document before embedding it for search. So, instead of directly matching the query embedding to the database embeddings, it first translates the query into a richer representation in the same semantic space as the real documents. Here’s how the process works: The user enters a natural language query expressing their needs, like “I want to convert an image into a 3D model.” This generated pseudo-document is then embedded into a dense vector using a model like OpenAI’s text-embedding-ada-002. The resulting embedding is used to query the vector database and find the most similar real model documents. The query is fed into an instruction-tuned LLM like GPT-3 to generate a hypothetical document. The prompt asks the model to imagine an ideal document for answering the query. For example: Query: I want to convert an image into a 3D model Hypothetical Document: There are several AI models available that can convert a 2D image into a 3D model. The process is known as single-view reconstruction or 3D estimation. It typically involves using deep learning to predict the depth map and 3D shape from the 2D input image. Some popular models for this task include Pix2Vox, Pixel2Mesh, AtlasNet, and OccNet. These models are trained on large datasets of 3D objects and can handle complex shapes. To use them, you provide an input image and the model will output a 3D mesh or voxel representation that you can render or edit further in 3D modeling tools… The LLM step helps bridge the gap between the user’s simple query and the more technical language used in the model descriptions. It expands on the initial query to infer additional relevant terms, concepts, and context that can help retrieve better matches. At the same time, using real document embeddings as the final search index keeps the results grounded and prevents hallucination. I implemented a basic version of HyDE on top of the existing AIModels search, and the initial results are promising. In some sample queries I tested, HyDE surfaced relevant models that the original keyword-matching approach missed. For the talking head portrait query, it was able to generate a hypothetical document with key terms like “lip sync”, “facial animation”, and “audio-driven animation”, which helped retrieve some strong model matches that the plain search had overlooked. two to three times more How to implement HyDE in your own search system Let me show you how HyDE can be added in on top of an existing semantic search system to dramatically improve its quality. Let’s start with a simplified code sketch based on my current search flow. Here’s the full file — you can glance through it in totality before we look at the individual parts: const fetchData = async (query) => {
  // Generate a hypothetical document embedding for the query 
  const hypotheticalDocument = await generateHypotheticalDocument(query);
  
  const embeddingResponse = await openAi.createEmbedding({
    model: "text-embedding-ada-002",
    input: hypotheticalDocument,
  });
  
  const embedding = embeddingResponse.data.data[0].embedding;

// Search the real model embeddings using the hypothetical document embedding
  const { data: modelsData, error } = await supabase.rpc("search_models", {
    query_embedding: embedding,
    similarity_threshold: 0.75,
    match_count: 10,
  });
  return modelsData;
};
const generateHypotheticalDocument = async (query) => {
  // Use GPT-3 to expand the query into a hypothetical ideal document
  const prompt = `The user entered the following search query: 
  "${query}"
  
  Please generate a detailed hypothetical document that would be highly relevant to answering this query. The document should use more technical language and expand on the key aspects of the query.
  Hypothetical Document:`;
  const response = await openAi.createCompletion({
    model: "text-davinci-002",
    prompt: prompt,
    max_tokens: 200,
    n: 1,
    stop: null,
    temperature: 0.5,
  });
  const hypotheticalDocument = response.data.choices[0].text.trim();
  return hypotheticalDocument;
} Let’s see how it works. We’ll start with the function, which is the main entry point for the search process. First up: fetchData const hypotheticalDocument = await generateHypotheticalDocument(query); This line calls the function (which we'll explain in more detail later) to generate a hypothetical ideal document based on the user's search query. generateHypotheticalDocument The hypothetical document is a key part of the HyDE approach, as it aims to capture the user’s search intent in a more detailed and technical way than the original query. const embeddingResponse = await openAi.createEmbedding({
  model: "text-embedding-ada-002",
  input: hypotheticalDocument,
});
const embedding = embeddingResponse.data.data[0].embedding; These lines use OpenAI’s model to generate an embedding vector for the hypothetical document. I think it's the best embedding option out there so far. text-embedding-ada-002 Embeddings are a way to represent text as a dense vector of numbers, where similar texts have similar vectors. This allows us to perform semantic similarity searches. By embedding the hypothetical document instead of the original query, we aim to get a vector representation that better captures the expanded search intent. Next up: const { data: modelsData, error } = await supabase.rpc("search_models", {
  query_embedding: embedding,
  similarity_threshold: 0.75,
  match_count: 10,
}); This part performs the actual similarity search in the model database. It is called a remote procedure ( ) on the Supabase database, passing in the embedding of the hypothetical document as the . search_models query_embedding The and parameters control how closely the results need to match and how many to return. similarity_threshold match_count The search looks for model embeddings that are most similar to the query embedding based on cosine similarity. Now let’s look at the function: generateHypotheticalDocument const prompt = `The user entered the following search query: 
"${query}"

Please generate a detailed hypothetical document that would be highly relevant to answering this query. The document should use more technical language and expand on the key aspects of the query.
Hypothetical Document:`; This part constructs the prompt that will be sent to the language model to generate the hypothetical document. The prompt includes the user’s original query and instructions for the model to generate a detailed, technical document that expands on the query’s key aspects. Crafting an effective prompt is crucial for guiding the language model to generate useful hypothetical documents. Now for the next part: const response = await openAi.createCompletion({
  model: "text-davinci-002",
  prompt: prompt,
  max_tokens: 200,
  n: 1,
  stop: null,
  temperature: 0.5,
}); These lines use OpenAI’s model to generate the hypothetical document based on the prompt. text-davinci-002 The parameter limits the length of the generated document while controls the randomness (higher values make the output more diverse but potentially less focused). Adjusting these parameters can help tune the quality and variety of the generated documents. max_tokens temperature Next, we have: const hypotheticalDocument = response.data.choices[0].text.trim(); This line extracts the generated hypothetical document text from the GPT-3 API response. The generated text is then returned to be used in the embedding step. Then we can just retreive the data as before, this time using the hypothetical document as the query, but getting a lot more relevant results due to the expanded search term and all it’s extra, juicy context! Recap — what did we just build? I hope the code and my explanation demonstrate how the HyDE approach can be implemented on top of an existing semantic search system. It’s not that hard! The key idea is this: By inserting the hypothetical document generation and embedding steps into the search pipeline, we aim to improve the search results by better capturing the user’s intent. To recap, the key parts were: Using a language model to expand the user’s query into a more detailed, technical hypothetical document Embedding the hypothetical document to get a vector representation that aligns with the model database Performing a similarity search between the hypothetical document embedding and the model embeddings to find the most relevant results Of course, this is just a simplified implementation (simplementation?), and there are many details and optimizations to consider in a real implementation. But it illustrates the core flow and components of the HyDE approach. Taking it further Since this is just an initial prototype there’s still a lot of room for experimentation and refinement. Some of the key open questions I’m thinking about: What’s the optimal prompt structure and output length for generating useful hypothetical documents? I’d like to try different prompt templates and output sizes to see how they impact result quality. How can the hypothetical documents be made more interpretable and transparent to users? It could be confusing if users don’t understand why certain terms are being added. I’m considering showing parts of the generated text as a “query interpretation” to provide more insight into the process. To what extent does the result quality depend on the specific LLM and embedding models used? I’m curious to benchmark different model combinations (e.g. GPT-4 vs GPT-3, vs OpenAI embeddings, etc.) to find the best recipe. Contriever How can this approach be efficiently scaled to work over millions of database entries? Inference on large LLMs like GPT-3 is expensive and slow. I’ll need to look into caching, optimizing, or distilling the models to make this viable at scale. I’m excited about the potential for language models to help make search interfaces like AIModels more intuitive and powerful for a wider audience. Bridging the semantic gap between how users naturally express their needs and how AI models are technically described feels like an important piece of the puzzle. Techniques like HyDE point a way toward search experiences that deeply understand the intent and meet users where they are. My long-term goal is to keep refining this interface until anyone can easily find relevant AI models for their needs, no matter their level of technical expertise. I’ll be sharing more details as I continue experimenting with HyDE and other approaches. If you’re curious to learn more about the technical side, I recommend checking out the original paper. And if you have ideas or feedback on the AIModels search experience, I’d love to hear from you! Let me know what you think in the comments. Also published . here

Improve Semantic Search and Enhance AI Model Discovery with Hypothetical Document Embeddings (HyDE)

Why do we even need any documents in the development process? What about this statement that the ? code documents itself Let's consider the most common scenario: the system's code (be it a program, project, or product) is being written over a long period, and the team gradually changes during this process, taking certain knowledge about the system with them as developers leave. What can we do in such a case? The simplest answer is to that captures all the implementation details to ensure the system meets the original requirements. write a specification However, such a document is very difficult to write in advance, and during the development process, some implementation details may change (adapting to the market/new requests for mechanics, etc.). So, what can we devise to improve the ? bus factor Let's try to follow a flow that could be one of the possible solutions to address the problem mentioned above. Requirements & RFC First, we need to describe the initial design based on requirements from stakeholders and document it. After that, this document can be shared with other teams and their feedback requested: ask to implement certain features, comment on the initial design, fix a certain interface, etc. Such a document can be called an . RFC , or "Request for Comments," is a document distributed among interested parties—including developers, architects, and other teams—to collect feedback, comments, and suggestions. It is less detailed than a specification and includes only the initial problem, task, and solution domain. Being more flexible, it allows for actively accepting changes in the design ensuring a deep understanding of the task and facilitating quality and thoughtful decision-making. RFC Design Commitment & ADR Okay, we've and . What's next? defined technical requirements gathered requirements from other teams At this stage, it's necessary to and all the main functions it will perform. For this purpose, we write an . finalize the system design ADR , or "Architecture Decision Record," is a document that records important architectural decisions made during the software development process. Each describes a specific high-level architectural decision, its context, alternatives considered, the decision made, and the motivation for choosing these specific details over others. ADR ADR Such a document allows every team member (and other teams as well) the principles and values that underpin the design. If a new developer joins the team years later and asks, "Why did you do it this way?", they can be shown this document, which will answer all their questions. to understand Specification Now it's time to write the code and its specifications. At this stage, we thoroughly work through each feature, simultaneously compiling all the information and implementation details into a special document. This document should reflect the current low-level requirements for the system. : during the software lifecycle, such a specification can change significantly, and that's okay. However, it's very important to still stay within the original design and architecture to prevent the codebase from becoming something unmanageable. An important point Test plan Why is it needed? It is critical for a test plan to be built that was written according to the specification (we write code and tests for this code so that they pass), but on that includes critical scenarios that . It is also very convenient that you can submit such a test plan for review to other teams (for integrations or just for additional testing), making it clear how the system will behave in different situations. not on the basis of code the basis of a design must be processed correctly What does it include? All possible system operation scenarios Happy path Sad path Error handling All possible invariants that must be maintained during system operation Acceptance tests to check the system state at the start (should consider the environment, e.g., data on the network) We've finalized the , written the code and , and compiled the . Sounds pretty solid already! But what else could we add? design specification test plan Deployment Plan Such a plan might be needed to some extent to and create conditions in which any team member can deploy the system and verify it's state. improve the bus factor Why can't we do without it? We can, but in the real world, large teams where many people are responsible for different parts of the system, and the deployment process might be to DevOps. What's wrong with that, since we've written tests, put them in CI, and checked for vulnerabilities, do we need anything else? Maybe not, but often tests do not consider the current state of the system and test not quite what we would like. fully delegated What deployment plan : might contain A complete description of actions that need to be performed for the deployment to take place Deployment parameters: Environment variables Initial state Parameters for launch A plan for if something goes wrong at any step A backup plan, if possible The necessary state to which the system must be brought in case continuing the deployment is not possible Actions that need to be performed after deployment Notify other teams Enable necessary integrations, if needed Nothing complicated, right? Having such a document for a specific update can significantly and on specific individuals. Isn't that what we want? improve the bus factor prevent reliance Conclusion In the software development process, it's important not just to write code, but also to create documentation that ensures understanding and consistency throughout all stages of development. The documentation , but experience has shown that documentation is very important to maintain the quality, stability and future scalability of the system, especially when the team changes during development, and also when the project evolves and adapts to new requirements. may be the code itself Documentation includes (Request for Comments), (Architecture Records), , , , and more. This will guarantee the , simplify the process of into the project, and . RFC ADR Specification Test Plans Deployment Plans retention of knowledge in the team integrating new employees increase the overall reliability and resistance of the system to changes

What Do You Know About Document-Driven Development?

ডকুমেন্ট-চালিত উন্নয়ন সম্পর্কে আপনি কি জানেন?

Was wissen Sie über dokumentengesteuerte Entwicklung?

¿Qué sabe sobre el desarrollo basado en documentos?

Que savez-vous du développement basé sur les documents ?

दस्तावेज़-संचालित विकास के बारे में आप क्या जानते हैं?

ドキュメント駆動開発について何を知っていますか?

문서 중심 개발에 대해 무엇을 알고 있습니까?

O que você sabe sobre desenvolvimento orientado a documentos?

Что вы знаете о разработке, управляемой документами?

Belge Odaklı Geliştirme Hakkında Ne Biliyorsunuz?

Bạn biết gì về phát triển dựa trên tài liệu?

您对文档驱动开发了解多少？

Perhaps you’ve noticed that photos appearing on the news depicting war are being questioned much more frequently than before. In many instances, they’ve been identified as fakes by expert inspection, the consensus of a community of people, or tools such as fake photodetectors that analyze them. Whereas 2023 may be looked upon as the year where it became standard practice to question whether photos and, in many cases, videos and documents are real, 2024 is poised to be the year where it becomes standard practice to run every questionable photo, video, and document through AI systems to determine their authenticity. After years of experts warning about a potential scenario where deepfakes will have the power to skew reality, it only makes sense to ask a couple of questions: How did we so rapidly get into this situation? What can we do about it? GenAI Implications With the explosion of generative AI text tools such as ChatGPT in 2023, it was easy to lose track of text-to-photo offering, , making tremendous headway in the ability to create fake images. OpenAI’s Dall-E At the same time, many similar frameworks emerged from several vendors and open source, including , , , and , amongst many others. If your goal is to create fake photos, you no longer have to look very far. Stable Diffusion Midjourney Adobe Firefly Google Many of these frameworks have already undergone significant improvements in 2023 alone, and the outcome is very clear. The ability to generate fake or AI-altered photos is now in the hands of the masses, and the ability to generate fake videos and documents is also here and rapidly improving. So now that the proverbial cat has been let out of the bag, there are a number of efforts circulating to help resolve the issue of fakes, ranging from government policies and controls to standards to help identify fake photos, videos, and documents, to anti-AI frameworks you can begin using today. While government policies and controls may represent a valiant effort to save us from this impending doom, open-source frameworks tend to elude corporate-enforced controls, because they are not owned by any particular organization. Moreover, the effort would require governments from virtually all nations to act in tandem to help solve the problem that effectively has spread worldwide. Standards to identify fake photos are also a valiant effort to get the corporations developing generative AI to digitally watermark the photos and videos created, so, they can be identified as fakes. However, these corporations amount to the good actors of the potential information warfare that can ensue. What about the bad actors leveraging open source or finding ways to circumvent or remove the watermarking? The answer, if you haven’t guessed, is there is no way to get bad actors to play nice. That leaves anti-AI frameworks, or fake detectors, as the best available and most proactive tool, particularly for businesses who need to protect important data against the onslaught of fakes. Tools that identify what is real and what is fake are the best immediate defense, albeit there are a few caveats everyone should be aware of. Static tools will become obsolete very fast as a result of the advancement of generative AI technology. There needs to be constant improvement on the side of the detectors to keep up with the constantly improving fakes. While some may argue that this is a losing battle, it is the only battle for those who want to guard against fraud and potential reputation damage. The other caveat is that a detection framework needs to be analyzed rapidly and at scale to counter incoming threats. Thanks to GPU technology and cloud scale, this is all possible today. Ultimately, while a few choices are floating around how to thwart fakes, the best defense is a tangible strategy rather than hoping the problem goes away or that others resolve the problem for you. Those who take early action will benefit in the long term. Luckily, there are proactive steps you can take to protect yourself and your business against the growing threat of fakes.

The 2024 Inundation of Fake Photos, Videos and Documents: And Why 2023 Was Just the Beginning

Why is an SSH tunnel into AWS EC2 needed? Before I tell you why it's needed, I'd like to share why I had to do it. The answer is simple: to locally test things in our ML Codebase. Now, coming to why it's needed: Amazon DocumentDB is a managed database service that is designed to be secure. This simply means that the database is hosted privately onto something called Amazon Virtual Private Cloud (Amazon VPC). In simple terms, I like to think of it as Amazon's own private internet. So DocumentDB can be directly accessed by any AWS service within the same VPC or any other with the required permissions. SSH tunneling is needed when we want to access DocumentDB resources from outside the cluster's VPC, here on our local machine. To access DocumentDB from your local machine, you typically need to go through a bastion host (EC2 instance) using SSH. This extra layer of security ensures that your database connection is not directly exposed to the internet, reducing the risk of unauthorized access. What is SSH Tunneling? SSH tunneling, also known as "port forwarding," is a technique used to secure and encrypt communication between two computer systems over an unsecured network, such as the Internet. It involves creating a secure channel (tunnel) through which data can be transferred between a local and a remote machine. In simple terms, it's establishing a VPN. In our context: The local machine is the one running your Python script. The remote machine is an EC2 instance in your AWS environment. The SSH tunnel allows secure communication between your local machine and the EC2 instance, providing a secure pathway for data to travel. Once the tunnel is established, you can use it to connect to DocumentDB securely, as if it were running on your local machine. Code Before diving into the code, you will need a few important constants. I suggest storing them as environment variables for security purposes. # SSH tunnel configuration
SSH_HOST=ec2-x-x-x-x.region.compute.amazonaws.com
SSH_USER=ec2-user
SSH_KEY_PATH=path to ec2-host-key-pair.pem file
LOCAL_BIND_PORT=3000 # any port of your choice

# MongoDB server configuration
MONGO_HOST=replica_db_name.*.*.docdb.amazonaws.com
MONGO_PORT=27017
MONGO_USERNAME=your_monogdb_username
MONGO_PASSWORD=your_monogdb_password

MONGO_DB_NAME=YOUR_DB_NAME
MONGO_COLLECTION_NAME=YOUR_DEFAULT_COLLECTION_NAME

# db parameters dict
DB_PARAMS = {
    "host": '127.0.0.1',
    "port": LOCAL_BIND_PORT,
    "username": MONGO_USERNAME,
    "password": MONGO_PASSWORD,
} : is the public IP for your EC2 instance running in the same VPC as your DocumentDB. SSH_HOST : path to your key-pair.pem file. This is used to authenticate your SSH connection to the EC2 instances. SSH_KEY_PATH NOTE: Whitelist your IP Address in your EC2 Security Groups before running the code. from pymongo import MongoClient
from sshtunnel import SSHTunnelForwarder


tunnel = SSHTunnelForwarder(
            (SSH_HOST, 22),
            ssh_username=SSH_USER,
            ssh_pkey=SSH_KEY_PATH,
            remote_bind_address=(MONGO_HOST, MONGO_PORT),
            local_bind_address=('127.0.0.1', LOCAL_BIND_PORT)
        )

# start the tunnel
tunnel.start()

# get mongo client
client = MongoClient(
                directConnection=True,
                **DB_PARAMS
        )

# do something
db = client[MONGO_DB_NAME]
collection = db[MONGO_COLLECTION_NAME]

documents = list(collection.find(some_query))
print(documents)

# stop the tunnel and close the client
client.close()
tunnel.stop()

client=None How does this work? Here is a simple picture to describe it: The figure presents a simplified overview of SSH tunneling. The secure connection over the untrusted network is established between an SSH client and an SSH server. This SSH connection is encrypted, protects confidentiality and integrity, and authenticates communicating parties. The SSH connection is used by the application (our Python code) to connect to the application server (Mongo/DocDB Server). With tunneling enabled, the application contacts a port (= 3000) on the local host ('127.0.0.1') that the SSH client listens on. The SSH client then forwards the application over its encrypted tunnel to the server (EC2 Instance). The server then connects to the actual application server (DocumentDB) - usually on the same machine or in the same data center as the SSH server. The application communication is thus secured without having to modify the application or end-user workflows. References - by ssh.com What is SSH Tunneling? AWS Docs: , , Link 1 Link 2 Link 3 StackOverflow Post Also published . here

SSH Tunneling to AWS EC2 and Connecting to DocumentDB with Python

This paper is available on arxiv under CC 4.0 license. Authors: (1) Michael Günther, michael.guenther; (2) Jackmin Ong, jackmin.ong; (3) Isabelle Mohr, isabelle.mohr; (4) Alaeddine Abdessalem, alaeddine.abdessalem; (5) Tanguy Abel, tanguy.abel; (6) Mohammad Kalim Akram, kalim.akram; (7) Susana Guzman, susana.guzman; (8) Georgios Mastrapas, georgios.mastrapas; (9) Saba Sturua, saba.sturua; (10) Bo Wang, bo.wang; (11) Maximilian Werk, maximilian.werk; (12) Nan Wang, nan.wang; (13) Han Xiao, han.xiao}@jina.ai. Table of Links Abstract & Introduction Related Work Training Process Overview Backbone Pre-training Fine-Tuning for Embeddings Evaluation Conclusion & References Appendix A Appendix

JINA EMBEDDINGS 2: 8192-Token General-Purpose Text Embeddings for Long Documents: Appendix

This paper is available on arxiv under CC 4.0 license. Authors: (1) Michael Günther, michael.guenther; (2) Jackmin Ong, jackmin.ong; (3) Isabelle Mohr, isabelle.mohr; (4) Alaeddine Abdessalem, alaeddine.abdessalem; (5) Tanguy Abel, tanguy.abel; (6) Mohammad Kalim Akram, kalim.akram; (7) Susana Guzman, susana.guzman; (8) Georgios Mastrapas, georgios.mastrapas; (9) Saba Sturua, saba.sturua; (10) Bo Wang, bo.wang; (11) Maximilian Werk, maximilian.werk; (12) Nan Wang, nan.wang; (13) Han Xiao, han.xiao}@jina.ai. Table of Links Abstract & Introduction Related Work Training Process Overview Backbone Pre-training Fine-Tuning for Embeddings Evaluation Conclusion & References Appendix 6 Evaluation To evaluate the efficacy of our approach, we initiate with a comprehensive analysis of our pre-trained backbone models, as outlined in Section 6.1. This is followed by an in-depth assessment of our embedding models in Section 6.2. Furthermore, we have conducted experiments to delve into the effects of encoding extended sequence lengths on the performance of the embeddings, presented in Section 6.2.2. 6.1 Evaluation of Jina BERT Following previous work [Liu et al., 2019b], we evaluate our pretrained models on the GLUE bench mark [Wang et al., 2018]. General Language Understanding Evaluation (GLUE) is a collection of nine datasets for evaluating natural language understanding systems. Six tasks are framed as either single-sentence classification or sentence-pair classification tasks. The GLUE organizers provide training, development, and test data splits, as well as a submission server and leaderboard.[5] The test split does not contain labels, and the submission server allows participants to evaluate and compare their systems against the private labels of the test split. For the Jina BERT training described in Section 4, we fine-tune the pre-trained models on the corresponding single-task training data using several hyperparameter settings and, for each task, pick the best fine-tuning hyperparameters on the development set. Following the methodology of [Phang et al., 2018], for RTE, STS, and MRPC, we fine-tune starting from the MNLI single-task model, rather than the baseline pretrained Jina BERT models. As in the BERT paper [Devlin et al., 2019], our finetuning procedure relies on representing the input sequence and using the final hidden vector C ∈ R H corresponding to the first input token ([CLS]) as the aggregate representation. We train for 10 epochs with batch sizes {16, 32} and learning rates {1e−5, 2e−5, 3e−5}. For each task, the best fine-tuned model on the development set is used for the test set. In Table 2, we report the results of the best performing models on the test sets after submission to the GLUE benchmark server. Furthermore, we evaluate Jina BERT models on documents of long text sequences by computing the accuracy of the MLM task with varying sequence lengths. The accuracy of masked language modeling is computed on 50, 000 samples from the C4 validation set where, for each chosen sequence length, each sample document is tokenized and truncated to fit the sequence length. We compare Jina BERT to RoBERTa and BERT models in Figure 2. It essentially shows that, even though Jina BERT models were trained on a 512 sequence length, the MLM accuracy does not drop when we extrapolate to an 8192 sequence length. For other BERT and RoBERTa models, since they use absolute positional embeddings that are trained on a 512 sequence length, it’s not possible to compute the MLM accuracy beyond 512. The figure demonstrates ALiBi’s effectiveness in maintaining MLM performance during inference for long documents. 6.2 Evaluation of Jina Embeddings v2 To comprehensively evaluate our embedding models, we employ the Massive Text Embedding Benchmark (MTEB) [Muennighoff et al., 2023]. Our choice of MTEB is motivated by its unparalleled breadth, distinguishing it among embedding benchmarks. Rather than focusing on a single task and dataset, MTEB covers an expansive set of 8 tasks, encompassing a rich collection of 58 datasets across 112 languages. This expansive benchmark allows us to scrutinize our model’s adaptability across diverse applications and languages and benchmark it against other topperforming models. However, a limitation of the MTEB benchmark is its omission of very long texts, which are essential for evaluating our model’s prowess in handling 8192 sequence lengths. Consequently, we introduce new retrieval and clustering tasks featuring extended documents, and we detail the performance of our model against its peers in Section 6.2.2. The goal here is to aptly group a collection of sentences or paragraphs. Within the MTEB benchmark suite, a mini-batch k-means model is employed, operating with a batch size of 32. Here, k represents the number of unique labels in the dataset. Model performance is evaluated using the V measure, a metric insensitive to cluster label permutations, guaranteeing that assessments are independent of label configurations. Clustering: We incorporate two new clustering tasks featuring extended documents within the MTEB clustering task subset. The inaugural task, named PatentClustering, draws from the BigPatent[6] dataset [Sharma et al., 2019], challenging the kmeans model to organize patents by their respective categories. Patent documents average 6, 376 tokens, spanning a range from a brief 569 tokens to an extensive 218, 434 tokens. Our second task, titled WikiCitiesClustering, sources from the English subset of the refined Wikipedia dump [Foundation, 2022], available as a dataset on Hugging Face[7] . For this task, we curate a roster of nations from Wikidata and extract Wikipedia articles of their cities from the refined dataset. The objective is to group cities by their parent country. On average, articles consist of 2, 031 tokens, with the length varying between a succinct 21 tokens to a comprehensive 20, 179 tokens. This task entails a dataset comprising a corpus, a set of queries, and associated mappings connecting each query to pertinent corpus documents. The mission is to discern relevant documents for a specific query. Both queries and corpus documents undergo encoding, after which their similarity scores are derived using cosine similarity. Subsequently, metrics like nDCG@10 (which serves as the primary metric), MRR@k, MAP@k, precision@k, and recall@k are computed for diverse k values. This task is inspired by datasets and evaluation methods presented by BEIR [Thakur et al., 2021]. Retrieval: To expand the scope of the MTEB, we introduce a new retrieval task named NarrativeQA, derived from the narrativeqa[8] dataset. This dataset boasts realistic QA instances, curated from literature (encompassing both fiction and non-fiction) and film scripts. The corpus averages 74, 843 tokens per document, with the lengthiest document tallying up to 454, 746 tokens, and the most concise one comprising 4, 550 tokens. 6.2.1 Results on MTEB The evaluation of embedding models within the MTEB benchmark, as illustrated in Table 3, reveals significant contrasts between Jina’s text embedding models, namely jina-small-v2 and jina-base-v2, and other contemporary models. These differences are especially pronounced in tasks showing marked performance disparities, such as Classification (CF) and Retrieval (RT). In Classification (CF), the jina-base-v2 model, equipped with 137 million parameters, emerges as a leading performer. It records superior scores, outpacing most competing models, underscoring its efficacy in text classification. Conversely, the jina-small-v2 model, equipped with a modest 33 million parameters, trails behind some other models in this task. This underscores the pivotal role model size plays in certain downstream tasks, with more extensive architectures yielding potential benefits. For the Retrieval (RT) task, jina-small-v2 showcases formidable performance, signaling its adeptness for information retrieval. It ranks amidst top-tier models, indicating its prowess in retrieval-centric tasks. Similarly, jina-base-v2 excels, registering a slightly superior score, reaffirming its formidable retrieval aptitude. Both models underscore their credibility in tasks necessitating adept information retrieval. Given that models all-MiniLM-L6-v2 and all-mpnet-base-v2 omit the second-stage finetuning which jina-small-v2 and jina-base-v2 undergo, it’s foreseeable that our models would excel in these tasks. In conclusion, both the base and small text embedding models display commendable performance within the MTEB benchmark. Their standout performance, relative to other models in tasks like Classification and Retrieval, suggests model size’s influential role in specific text processing endeavors. Both models reaffirm their potency in retrieval, marking them as pivotal tools for a plethora of natural language processing tasks. 6.2.2 Impact of Maximum Sequence Length As delineated in Section 6.1, the pre-training generalizes across extended sequence lengths. Consequently, the MLM accuracy for long sequences, spanning up to 8192 tokens, mirrors that of shorter sequences, despite the exclusive training on abbreviated text sequences. During finetuning, our models train solely on texts not exceeding 512 tokens, yet they cater to texts reaching 8192 tokens for the MTEB evaluation detailed in Section 6.2. To discern how sequence length impacts the accuracy of downstream tasks, we executed long document clustering and retrieval tasks, modulating the tokenizer’s maximum sequence length. This allows us to gauge the models’ performance on variable sequence lengths through truncation. Since a majority of the extant tasks in the MTEB feature documents under 512 tokens, we resort to our three novel datasets elucidated in Section 6.2, accessible on Hugging Face[10]. Furthermore, we employ the SciFact dataset [Wadden et al., 2020], given its substantial count of texts exceeding 512 tokens. Figure 3 depicts the nDCG@10 retrieval and the V measure scores for the jina-base-v2 alongside four other renowned embedding models. Given that only jina-base-v2 and OpenAI’s text-embedding-ada-002 support an 8k sequence length, results reported for an 8191 sequence length for other models are truncated to their intrinsic maximum, typically 512. Generally, Figure 3 suggests that elongated sequence lengths contribute to enhanced outcomes. This assertion is particularly true for the NarrativeQA task, where extending the sequence length substantially bolsters performance. Due to the inherent nature of the dataset, models limited to the text’s commencement frequently underperform. On the BigPatent clustering task, larger sequence lengths also result in better performance. However, on the WikiCities clustering task, longer sequence lengths seem to slightly diminish the models’ performance in most instances. This suggests that an increase in sequence length doesn’t always yield better outcomes. One explanation for this observation is that the initial paragraph of a Wikipedia article about a city typically mentions the country the city is in. Information towards the middle and end of the articles is often less pertinent for identifying the country and might alter the attributes that influence the clustering of the city embeddings. [5] https://gluebenchmark.com [6] https://huggingface.co/datasets/big_patent [7] https://huggingface.co/datasets/wikipedia [8] https://huggingface.co/datasets/narrativeqa [9] For e5-base-v2, we abstained from employing specific prefixes like “query: ”, which might result in varied evaluation outcomes. text-embedding-ada-002 caps its encoding at 8191 tokens, not 8192. [10] Our datasets are available at https://huggingface.co/ jinaai

JINA EMBEDDINGS 2: 8192-Token General-Purpose Text Embeddings for Long Documents: Evaluation

This paper is available on arxiv under CC 4.0 license. Authors: (1) Michael Günther, michael.guenther; (2) Jackmin Ong, jackmin.ong; (3) Isabelle Mohr, isabelle.mohr; (4) Alaeddine Abdessalem, alaeddine.abdessalem; (5) Tanguy Abel, tanguy.abel; (6) Mohammad Kalim Akram, kalim.akram; (7) Susana Guzman, susana.guzman; (8) Georgios Mastrapas, georgios.mastrapas; (9) Saba Sturua, saba.sturua; (10) Bo Wang, bo.wang; (11) Maximilian Werk, maximilian.werk; (12) Nan Wang, nan.wang; (13) Han Xiao, han.xiao}@jina.ai. Table of Links Abstract & Introduction Related Work Training Process Overview Backbone Pre-training Fine-Tuning for Embeddings Evaluation Conclusion & References Appendix 3 Training Process Overview The training process for Jina Embeddings v2 is divided into three stages: For the backbone pre-training, we design a modified BERT model capable of encoding documents with up to 8192 tokens. This model is trained on a full-text corpus using a masked language modeling objective. I Pre-training the Backbone: To encode a text passage into a single vector representation, the model is fine-tuned in an unsupervised manner on text pairs. II First Fine-tuning with Text Pairs: The model is further fine-tuned using text pairs complemented with hard negatives. This III Second Fine-tuning with Hard Negatives: stage is crucial for enabling the model to better distinguish between relevant passages and related, but irrelevant text passages. While both stages II and III are geared towards training the models for embedding tasks, the latter is especially critical for improving the model’s performance in retrieval and classification tasks (refer to Section 6.2).

Stories, News & Mentions about DOCUTEN

Docuten’s Stories on HackerNoon