Datuum is an AI-powered no-code tool that accelerates data onboarding from various data sources into one place. Our unique AI engine simplifies data integration and management for data professionals of all levels. The platform builds resilient connectors to various data sources, providing instant access to your data, automating mapping, accelerating data operations, and doing it all without coding. We are building Datuum to enable data engineers and analysts to reduce manual work and burnout while helping organizations save up to 80% of the time and effort typically required for data projects. Using Datuum is far more efficient than manually figuring out all the data complexities, making Datuum a game-changer for optimization and cost reduction of the day-to-day data operations in organizations. Datuum AI engine simplifies data operations: ◾ 3rd party data ingestion ◾ Data extraction from a complex data source ◾ Data migration from system A to system B ◾ Accelerating new client onboarding ◾ Mater Record creation ◾ Data deduplication and consolidation

Datuum.ai

We all know how important it is to have data interoperability and smooth data exchange, right? Well, FHIR (Fast Healthcare Interoperability Resources) is like the superhero of standardization and unification. It's the answer to reducing the crazy costs of data integration. When healthcare professionals adopt FHIR, they get the awesome power of having a complete view of patients’ medical histories across different clinics and hospitals, allowing them to make informed decisions. Plus, FHIR makes it super simple to integrate medical devices from different manufacturers, making data exchange a breeze. All of this saves a ton of time and money while boosting the overall efficiency of healthcare. However, let's not overlook the challenges involved in adopting FHIR. While many companies aspire to embrace FHIR, mapping existing data exchange interfaces to FHIR can be a significant obstacle, akin to fitting a square peg into a round hole. Bridging the Gap: Converting EHR Extracts to FHIR In many cases, healthcare providers still rely on old-school flat files for batch data exchange. Sure, some interfaces support real-time integrations with FHIR messages, but most of the time, it's up to the IT/Data Integration teams to build those data pipelines and convert EHR data extracts into FHIR resources. Sounds like a real drag, right? Well, it is! It's manual, time-consuming, hard to maintain, and it'll cost you a pretty penny. Allow me to share an inspiring story about one of our clients—a HealthTech provider. They embraced FHIR as a Canonical data model internally but encountered a roadblock when they needed to onboard new clients (hospitals) who were still sending good ol' EHR flat files. They soon realized they were spending way too much time and money on implementing and supporting these integrations. Determined to find a better solution, they embarked on a quest. Picking the Right Solution Their quest involved seeking a solution that could save them time and money while remaining non-disruptive to their existing infrastructure. It also had to be flexible enough to support multiple data ontologies. That's when they discovered Datuum.ai, the ultimate sidekick for this job. Datuum’s AI engine analyzes all inbound data, automatically creates a mapping between the source data and FHIR ontology, and generates FHIR resources. Moreover, the entire process is conducted through a no-code interface, eliminating the need for developers to spend time on data mapping and integration tasks. Positioned at the beginning of the data journey, Datuum seamlessly integrates into existing data infrastructure. Notably, one of the key benefits of Datuum is its transparency, enabling data analysts to understand the transformation process and troubleshoot any issues that may arise. Unlocking the Potential of Interoperability By harnessing the power of Datuum's no-code tool, the company successfully overcame the challenges associated with adopting FHIR. This not only enabled them to establish smooth data exchange capabilities but also expedited the onboarding process for their customers. As a result, the company experienced significant time and cost savings while demonstrating its commitment to ensuring customer satisfaction. Also published here

Mapping Your Data to FHIR at a Fraction of the Cost: What You Need to Know

Last time I spoke about the need for the context to actually make a proper judgment about the semantics of the simple list of names. This time I want to slightly expose how messy the situation is with date and time values. There is so much “wrong” about them that I don’t even know where to begin. This is going to be done in 3 separate parts just to keep every piece relatively short. Part I. Date Literals Let me start with the notion that the very concept of “date” as an absolute thing simply does not exist. What we learned from our childhood living in western (Christian) culture is just one of many alternatives co-existing in the modern world. Well, probably any culture on this planet has a concept of a year (Earth moves around the Sun for all cultures), a month (Moon moves around the Earth for everyone), and a day (you see the pattern here, right?). Well, this is the point where the similarities basically end. Anything outside of the astronomical objective events is subject to cultural interpretation. Let’s look at a few examples: Western Date: April 27, 2021 Chinese: Mar 16, Xin Chou Year, Year of Ox Umm-Al-Qura: Ramadan 15, 1442 AH Hebrew: 15 Iyar 5781 Please, also note that all of these are written using Latin characters, while in their corresponding cultural contexts they will be represented by different scripts. It is completely confusing for western readers, so I will just give one example of the current date in Hebrew here: ט״ו בְּאִיָיר תשפ״א I grabbed this , and I won’t lie, I myself have no clue how this works and how I could possibly recognize it if I happened to see it somewhere in the real world. here Ok, anyways, computers are a product of western culture, so for simplicity, let’s limit the scope of this discussion to western dates, shall we? When we say “western date,” we actually mean Gregorian Calendar, which is currently universally accepted by all, let’s say, traditionally Christian countries in the world. But, if we happen to look at some database containing historical events, we must ask ourselves: is this particular date really Gregorian? In fact, there was quite a long historical transition period between Julian and Gregorian calendars when both were in use concurrently and the transition was happening at different times with different speeds in different places. The Gregorian calendar was first adopted in 1582 in some European countries, and the final one was Russia and its satellites in 1918. It took 336 years to more or less standardize calendars across Christian countries. Let’s limit our scope even further and assume we only deal with Gregorian Calendar. Are we done with our problems? Well, no. Let’s look at this: 3/2/21 What does it actually mean? US citizens will tell you that it is the 2nd of March 2021. UK citizens will argue that it is the 3rd of February 2021. I might argue that no matter where you live, this date refers to 1921. Alternatively, you can see other styles of numerical date representation: 03.02.21 03-II-2021 3.ii.1921 English is not the only language used in countries that adopted the Gregorian calendar. The same date can look very different depending on the country: November 13, 1976 13 ноября 1976 13 листопада 1976 13 listopada 1976r Then there is, of course, ISO 8601, which standardizes the literal representation of dates, making it fairly universal and pretty convenient to use: 1976-11-13 Obviously, in many cases, it is just impossible to universally recognize and normalize date literals without introducing a wider context. Often, necessary context can be derived from other pieces of data, in some cases, human intervention must happen, to inject the missing piece of information. As soon as context is sufficient to resolve ambiguity, datuum.ai will correctly detect date literals and convert them into some standard representation (like ISO 8601). In Part II, we will look at time literal and discuss time zones and DST for a bit and in Part III we will dip our toes into machine representations of date/time values. Thanks to Dmytro Zhuk, Founder & CTO of datuum.ai for this text. - Art Credit, Salvador Dali "Melting Clocks" - Also Published Here

Date and Time Values are a Mess - Here's Why

. Let’s take, for instance, the following list of strings: Context matters If we assume all items in the list above have the same semantic value, what is it exactly? The obvious answer to this question is “geographical place”, right? Let’s look at it in a wider context. Now we know that we were looking at the list of surnames, not cities. We had no way of telling the difference without extending the context. In this particular case, larger sample size would barely help, as nearly any city name can be a surname for someone and a lot of cities are named after some people. In normal situations, our brain doesn’t even notice how tricky this kind of distinction is, because, as humans, we rarely operate without a rich context. NLP techniques have some of the required context for token classification from the text which surrounds any particular token and use word embeddings to make a “best guess” of what this word might mean. When we analyze columns of data (let’s say, from a CSV file), we don’t have any sentence to derive a context from. Instead, we have other columns and, more often than not, other files. datuum.ai technology actually utilizes all the context which is available in a data source and takes a full set of columns, their order, full set of files/tables, etc. into account in order to determine a semantic type of the data. It took a lot of time and countless iterations to come up with neural network architecture and feature set which would allow us to achieve what we did achieve. There are still a lot of improvements ahead. And this is just the tip of the iceberg, one rather simple problem of many we are solving to make our platform work. Thanks to Dmytro Zhuk, Founder & CTO at @datuum for the story! Previously published here.

Context Matters in Semantic Ambiguity

We all are shocked by the terrible war Russia started, and we care for Ukraine and its citizens. While working as a developer, I used to work with many talented Ukrainians (the world knows how great Ukrainian programmers are!) and have made a lot of friends among them. I deeply empathize with them now, especially when I hear stories of my friends. One of them has to sleep in the subway with his family because it’s the closest shelter to his home. The other lost the house he built in Irpin, a small cozy town near Kyiv. He and his family put so much effort and spent so many years creating the home of their dreams, and now it’s gone, ruined by Russian troops. His kids are devastated, but he says: “at least we’re alive.” When I hear it, my heart breaks. Since the beginning of the war, I’ve been helping Ukraine, for example, by donating money. But it still feels like it’s not enough. I think the world doesn’t pay enough attention to Ukraine’s and Ukrainians’ problems. Probably because we can’t imagine what they are going through. I believe we aren’t helping them enough, and we could do more. So, I decided to collect different ways of supporting this great people struggling for freedom and share them with you. If you’ve already donated money or offered your help to refugees from Ukraine but still feel you could do more (like I do), this list is for you. As for me, I decided to teach and tutor Ukrainians for free. Giving them a fresh start in a new field (because many of them have lost their previous positions, maybe forever) is my way of helping Ukraine. 1. Hire a Ukrainian Unfortunately, many people have already lost their jobs after the beginning of the Russian invasion. It’s hard to estimate the number of Ukrainian developers who lost their jobs precisely, but in general, 53% of Ukrainians became unemployed after the war started (according to the survey conducted by Rating Group in March). And many more to go. Ukrainian developers, who have been highly appreciated on the global market, face a huge problem now. Due to the martial law declared by President Zelenskyy, men from 18 to 60 years old can’t leave the country, which makes it impossible for developers to relocate. But they can and are willing to work. Also, Ukrainian women who fled the country can and want to work. And working for these people nowadays means supporting not only themselves but the Ukrainian economy as well. Many of them are top-notch specialists. Many companies have considered Ukraine a high-risk territory and started to shut down some outsource projects involving Ukrainians. Also, Ukrainians on a startup’s team mean potential problems with investment attraction. If the company can’t guarantee the safety of its employees, it’s an additional risk for investors. I think it’s unfair, and I urge you to help if you can. For example, consider hiring a Ukrainian developer or other IT specialist (in marketing, design, management, etc.) if you have a job opening. There are several websites you can use to find the person you need: — there are over 2,000 active profiles of Ukrainian professionals on this website Hire for Ukraine — you can send the information about a job opening to them or become a volunteer if you have HR-related experience Hire The Ukrainian — you can post a job description and it won’t cost you a penny. Currently, there are 7,000+ candidates and 2,500+ jobs here. Employ Ukraine 2. Learn with Ukrainian apps Remember I told you that Ukrainian developers are outstanding? So, you can trust me when I say they can develop a quality app for studying. For example: — if you’ve been reading me for some time, you’ve probably heard about this service. I think it offers an extremely practical way to learn Java, and I’m happy to mention that it’s a Ukrainian project. It helps you become a Java developer even if you’ve never tried coding before. Practical tasks, a minimal amount of theory, plus an option to study with a mentor. Also, this company recently started a project aiming to provide Ukrainians who lost their jobs with a new career in IT. When you buy a subscription, two subscriptions go to the Ukrainians who need them to start a new profession (details — ). CodeGym.cc here — this service can be useful for those developers who work in multinational teams and want to learn a new language (or several languages). It offers help in finding a language tutor, a convenient platform for studying, and a virtual classroom. Also, you can track your progress and see how your speaking and vocabulary improve. Preply — helps to learn more efficiently by training your brain. With this app, you can start reading much faster, improve your memory, and become more focused and concentrated on what you’re doing at the moment. ReadLax Sure, this isn’t a complete list of Ukrainian companies worth your support. If you know others, feel free to write about them in the comments. And here is one more way to help Ukrainians to which I’m currently devoting almost all my spare time: 3. Use Ukrainian software and services As the world gets more and more interested in the war updates, the awareness of Ukrainian businesses and products grows. Here are some of them you probably have never heard about (although you should, in my humble opinion): — a DevOps platform that helps companies maximize the overall return on software development. It enables a team to perform all the project tasks: from planning to deployment. GitLab — this company’s productivity applications work on iPhone, iPad, and Mac. Some of them were named Editors’ Choice and Apps of the Year by Apple. Readdle — Ukrainian service for finding developers for startups. This company suggests a match for the opening in 24 hours and calls itself “the exclusive community of startup sidekicks.” Lemon.io — this service makes meetings work by recording, transcribing, and creating summaries by AI. It is compatible with Zoom, Google Meet, Teams, and Webex. Sembly.ai — this data management platform helps scale your company’s data operations. As the company says, using its product may lead to an 80% reduction in time and money. Datuum.ai 4. Use your professional skills to help Ukrainians For instance, if you’re an IT consultant, offer your services to people in need. If you’re a high-skilled professional, become a mentor for an aspiring Ukrainian building a career in your field. If you’re an educator, launch a free online course. If you have a site — you could add on it. this widget We have so much to give, and we can’t be stingy now. Ukraine didn’t start this war, but it must win it for all of us. The world needs Ukraine’s victory to continue believing that freedom still means something, and people have the right to be themselves without the fear of getting killed for it. The war will end eventually, and Ukraine will face the next challenge: recovering. Ukrainians will have to rebuild the country, so their economy needs a lot of support. If we can do it, we should. And the final note. Also Published Here

Supporting Ukraine as a Programmer

Speed. Performance. Responsiveness. These ideals reign supreme in the world of web development, especially JavaScript and . A slow or janky website is the hallmark of an amateur, while a slick, optimized experience delights users and sets professionals apart. Nodejs But creating truly high-performance web apps is littered with pitfalls. Mistakes abound that can drag down the pace of your without you even realizing it. Tiny oversights that bloat your code and surreptitiously sap speed away bit by bit. JavaScript You vigilantly minify your code and leverage caching...yet your site still feels oddly sluggish at times. The UI stutters during scrolling or when buttons are clicked. Pages take eons to load. What's going on? Turns out, there are many common ways we inadvertently slow down our JavaScript. Anti-patterns that, over time, can hobble site performance. The good news? These mistakes can be avoided. Today we're spotlighting the top 19 performance pitfalls that can secretly slow down JavaScript and Node.js apps. We'll explore what causes them through illustrative examples and actionable solutions to optimize your code. Identifying and eliminating these hazards is key to crafting buttery smooth web experiences that delight users. So, let's dive in! 1. Improper Variable Declarations and Scope When first learning JavaScript, it's tempting to declare all variables globally. However, this causes problems down the road. Let's look at an example: // globals.js

var color = 'blue';

function printColor() {
  console.log(color); 
}

printColor(); // Prints 'blue' This works fine, but imagine if we loaded another script: // script2.js

var color = 'red';

printColor(); // Prints 'red'! Because is global, script2.js overwritten it! To fix this, declare variables locally inside functions whenever possible: color function printColor() {
  var color = 'blue'; // local variable
  
  console.log(color);
}

printColor(); // Prints 'blue' Now, changes in other scripts won't affect . printColor Declaring variables in the global scope when unnecessary is an anti-pattern. Try to limit globals to configuration constants. For other variables, declare them locally in the smallest possible scope. 2. Inefficient DOM Manipulation When updating DOM elements, batch changes instead of manipulating one node at a time. Consider this example: const ul = document.getElementById('list');

for (let i = 0; i < 10; i++) {
  const li = document.createElement('li');
  li.textContent = i;
  
  ul.appendChild(li);
} This appends list items one by one. It is better to build a string first then set : .innerHTML const ul = document.getElementById('list');
let html = '';

for (let i = 0; i < 10; i++) {
  html += `<li>${i}</li>`; 
}

ul.innerHTML = html; Building a string minimizes reflows. We update the DOM once instead of 10 times. For multiple updates, build up changes, then apply at the end. Or better yet, use DocumentFragment to batch appends. 3. Excessive DOM Manipulation Frequent updates can crush performance. Consider a chat app that inserts messages into the page. DOM Bad: // New message received
const msg = `<div>${messageText}</div>`;
chatLog.insertAdjacentHTML('beforeend', msg); This naively inserts on each message. Better to throttle updates: Good: let chatLogHTML = '';
const throttleTime = 100; // ms

// New message received  
chatLogHTML += `<div>${messageText}</div>`;

// Throttle DOM updates
setTimeout(() => {
  chatLog.innerHTML = chatLogHTML;
  chatLogHTML = ''; 
}, throttleTime); Now, we update at most every 100ms, keeping DOM operations low. For highly dynamic UIs, consider virtual DOM libraries like React. These minimize DOM manipulation using a virtual representation. 4. Lack of Event Delegation Attaching event listeners to many elements creates needless overhead. Consider a table with delete buttons on each row: Bad: const rows = document.querySelectorAll('table tr');

rows.forEach(row => {
  const deleteBtn = row.querySelector('.delete');  

  deleteBtn.addEventListener('click', handleDelete);
}); This adds a listener to each delete button. Better to use event delegation: Good: const table = document.querySelector('table');

table.addEventListener('click', e => {
  if (e.target.classList.contains('delete')) {
    handleDelete(e);
  }
}); Now, there's a single listener on the . Less memory overhead. <table> Event delegation utilizes event bubbling. One listener can handle events from multiple descendants. Use delegation whenever applicable. 5. Inefficient String Concatenation When concatenating strings in a loop, performance suffers. Consider this code: let html = '';

for (let i = 0; i < 10; i++) {
  html += '<div>' + i + '</div>';
} Creating new strings requires memory allocation. Better to use an array: const parts = [];

for (let i = 0; i < 10; i++) {
  parts.push('<div>', i, '</div>');
}

const html = parts.join(''); Building an array minimizes intermediate strings. concatenates once at the end. .join() For multiple string additions, use array joining. Also, consider template literals for embedded values. 6. Unoptimized Loops Loops often cause performance problems in JavaScript. A common mistake is repeatedly accessing array length: Bad: const items = [/*...*/];

for (let i = 0; i < items.length; i++) {
  // ...
} Redundantly checking inhibits optimizations. Better: .length Good: const items = [/*...*/];  
const len = items.length;

for (let i = 0; i < len; i++) {
  // ...
} Caching length improves speed. Other optimizations include hoisting invariants out of loops, simplifying termination conditions, and avoiding expensive operations inside iterations. 7. Unnecessary Synchronous Operations JavaScript's async capabilities are a key advantage. But beware of blocking I/O! For example: Bad: const data = fs.readFileSync('file.json'); // blocks! This stalls execution while reading from the disk. Instead, use callbacks or promises: Good: fs.readFile('file.json', (err, data) => {
  // ...
}); Now, the event loop continues while the file is read. For complex flows, simplifies asynchronous logic. Avoid synchronous operations to prevent blocking. async/await 8. Blocking the Event Loop JavaScript uses a single-threaded event loop. Blocking it stalls execution. Some common blockers: Heavy computational tasks Synchronous I/O Unoptimized algorithms For example: function countPrimes(max) {
  // Unoptimized loop
  for (let i = 0; i <= max; i++) {
    // ...check if prime...
  }
}

countPrimes(1000000); // Long running! This executes synchronously, blocking other events. To avoid: Defer unnecessary work Batch data processing Use Worker threads Look for optimization opportunities Keep the event loop running smoothly. Profile periodically to catch blocking code. 9. Inefficient Error Handling It's vital to handle errors properly in JavaScript. But beware of performance pitfalls! Bad: try {
  // ...
} catch (err) {
  console.error(err); // just logging
} This captures errors but takes no corrective action. Unhandled errors often lead to memory leaks or data corruption. Better: try {
  // ...
} catch (err) {
  console.error(err);
  
  // Emit error event 
  emitError(err); 
  
  // Nullify variables
  obj = null;
  
  // Inform user
  showErrorNotice();
} Logging isn't enough! Clean up artifacts, notify users, and consider recovery options. Use tools like Sentry to monitor errors in production. Handle all errors explicitly. 11. Memory Leaks Memory leaks happen when memory is allocated but never released. Over time, leaks accumulate and degrade performance. Common sources in JavaScript include: Uncleaned-up event listeners Outdated references to deleted DOM nodes Cached data that's no longer needed Accumulating state in closures For example: function processData() {
  const data = [];

  // Use closure to accumulate data
  return function() {
    data.push(getData()); 
  }
}

const processor = processData();

// Long running...keeps holding reference to growing data array! The array keeps getting larger but is never cleared. To fix: Use weak references Clean up event listeners Delete no-longer-needed references Limit closured state size Monitor memory usage and watch for growing trends. Proactively eliminate leaks before they pile up. 11. Overuse of Dependencies While npm offers endless options, resist the urge to over-import! Each dependency increases bundle size and attack surface. Bad: import _ from 'lodash';
import moment from 'moment'; 
import validator from 'validator';
// etc... Importing entire libraries for minor utilities. Better to cherries pick helpers as needed: Good: import cloneDeep from 'lodash/cloneDeep';
import { format } from 'date-fns';
import { isEmail } from 'validator'; Only import what you need. Review dependencies regularly to prune unused ones. Keep bundles lean and minimize dependencies. 12. Inadequate Caching Caching allows skipping expensive computations by reusing prior results. But it's often overlooked. Bad: function generateReport() {
  // Perform expensive processing
  // to generate report data... 
}

generateReport(); // Computes
generateReport(); // Computes again! Since inputs haven't changed, the report could be cached: Good: let cachedReport;

function generateReport() {
  if (cachedReport) {
    return cachedReport;
  }

  cachedReport = // expensive processing...
  return cachedReport; 
} Now, repeated calls are fast. Other caching strategies: Memory caches like Redis HTTP caching headers LocalStorage for client caching CDNs for asset caching Identify cache opportunities - they often provide big speedups! 13. Unoptimized Database Queries When interfacing with databases, inefficient queries can bog down performance. Some issues to avoid: Bad: // No indexing
db.find({name: 'John', age: 35}); 

// Unecessary fields
db.find({first: 'John', last:'Doe', email:'john@doe.com'}, {first: 1, last: 1});

// Too many separate queries
for (let id of ids) {
  const user = db.find({id});
} This fails to utilize indexes, retrieves unused fields, and executes excessive queries. Good: // Use index on 'name' 
db.find({name: 'John'}).hint({name: 1});

// Only get 'email' field
db.find({first: 'John'}, {email: 1}); 

// Get users in one query
const users = db.find({
  id: {$in: ids} 
}); Analyze and explain plans. Create indexes strategically. Avoid multiple piecemeal queries. Optimize datastore interactions. 14. Improper Error Handling in Promises Promises simplify asynchronous code. But unhandled rejections are silent failures! Bad: function getUser() {
  return fetch('/user')
    .then(r => r.json()); 
}

getUser(); If rejects, exception goes unnoticed. Instead: fetch Good: function getUser() {
  return fetch('/user')
    .then(r => r.json())
    .catch(err => console.error(err));
} 

getUser(); Chaining handles errors properly. Other tips: .catch() Avoid promise nesting hell Handle rejections at the top level Configure unhandled rejection tracking Don't ignore promise errors! 15. Synchronous Network Operations Network requests should be asynchronous. But sometimes sync variants get used: Bad: const data = http.getSync('http://example.com/data'); // blocks! This stalls the event loop during the request. Instead, use callbacks: Good: http.get('http://example.com/data', res => {
  // ...
}); Or promises: fetch('http://example.com/data')
  .then(res => res.json())
  .then(data => {
    // ...
  }); Async network requests allow other processing while waiting for responses. Avoid synchronous network calls. 16. Inefficient File I/O Operations Reading/writing files synchronously blocks. For example: Bad: const contents = fs.readFileSync('file.txt'); // blocks! This stalls execution during disk I/O. Instead: Good: fs.readFile('file.txt', (err, contents) => {
  // ...
});

// or promises

fs.promises.readFile('file.txt')
   .then(contents => {
     // ...  
   }); This allows the event loop to continue during the file read. For multiple files, use streams: function processFiles(files) {
  for (let file of files) {
    fs.createReadStream(file)
      .pipe(/*...*/);
  }
} Avoid synchronous file operations. Use callbacks, promises, and streams. 17. Ignoring Performance Profiling and Optimization It's easy to overlook performance until there are obvious issues. But optimization should be ongoing! Measure first with profiling tools: Browser devtools timeline Node.js profiler third-party profilers This reveals optimization opportunities even if performance seems fine: // profile.js
function processOrders(orders) {
  orders.forEach(o => {
    // ...
  });
}

processOrders(allOrders); The profiler shows taking 200ms. We investigate and find: processOrders Unoptimized loop Expensive inner operation Unnecessary work We optimize iteratively. The final version takes 5ms! Profiling guides optimization. Establish performance budgets and fail if exceeded. Measure often and optimize judiciously. 18. Not Utilizing Caching Mechanisms Caching improves speed by avoiding duplicate work. But it's often forgotten. Bad: // Compute expensive report
function generateReport() {
  // ...heavy processing...
}

generateReport(); // Computes
generateReport(); // Computes again! The same inputs always produce the same output. We should cache: Good: // Cache report contents
const cache = {};

function generateReport() {
  if (cache.report) {
    return cache.report;
  }

  const report = // ...compute...

  cache.report = report;
  return report;
} Now, repeated calls are fast. Other caching strategies: Memory caches like Redis HTTP caching headers LocalStorage for client caching CDNs for asset caching Identify cache opportunities - they often give big wins! 19. Unnecessary Code Duplication Duplicated code harms maintainability and optimizability. Consider: function userStats(user) {
  const name = user.name;
  const email = user.email;
  
  // ...logic...
}

function orderStats(order) {
  const name = order.customerName;
  const email = order.customerEmail;

  // ...logic... 
} The extraction is duplicated. We refactor: function getCustomerInfo(data) {
  return {
    name: data.name, 
    email: data.email
  };
}

function userStats(user) {
  const { name, email } = getCustomerInfo(user);
  
  // ...logic...
}

function orderStats(order) {
  const { name, email } = getCustomerInfo(order);

  // ...logic...
} Now, it's just defined once. Other fixes: Extract utility functions Build helper classes Leverage modules for reusability Deduplicate whenever you can. It improves code health and optimization. Conclusion Optimizing JavaScript application performance is an iterative process. By learning efficient practices and being diligent about profiling, dramatic speed improvements can be achieved. Key areas to focus on include minimizing DOM changes, leveraging asynchronous techniques, eliminating blocking operations, reducing dependencies, utilizing caching, and removing unneeded duplication. With attention and experience, you can discover bottlenecks and hone in on optimizations for your specific workload. The result is faster, leaner, and more responsive web apps your users will love. So be relentless about optimizations - employ these tips and see your JavaScript fly! If you like my article, feel free to follow me on . HackerNoon Also published . here

Is Your Code Slow?: Avoid These 19 Common JavaScript and Node.js Mistakes

Extracting data from an existing database is one of the most challenging and tedious jobs that typically fall on the plate of the Data Engineering team. And when it comes to dealing with true legacy systems, well, let’s just say it adds an extra layer of intrigue to the mix. Unraveling the secrets left behind by generations of engineers is like solving a complex puzzle. Inconsistent data modeling and potential challenges like insufficient documentation, limited knowledge of the existing team, and unclear semantics are just a glimpse of the challenges ahead. Yet, amidst these daunting obstacles, the team is expected to deliver results with lightning speed. In this article, I aim to impart insights gained through hard-earned experience, offering a lifeline that may save you precious time. So, let us explore these pivotal non-technical starting points: Even if a system/database is very old, there is usually at least one person who has some understanding of how it works and where data is coming from. This individual can be an invaluable resource to you, but keep in mind that they may be busy, have knowledge gaps, or may not like your project at all. Nevertheless, building a positive relationship with this person can be critical to your success. They can help you save significant time and effort. Do not criticize the current state of things, even if there are obvious flaws and improvement opportunities. Focus on extracting the data as efficiently as possible with minimum dependencies on the source system. Identify a knowledge holder: It’s important to educate stakeholders about the nature of data extraction projects. Many people mistakenly assume it’s a one-time task, often due to a lack of experience in this area. However, unless you’re working with a very small database, it’s likely to require multiple iterations to ensure that the data is complete and accurate. Therefore, it’s essential to communicate realistic timeframes and the need for iterative processes to stakeholders. It’s a very challenging task to estimate that effort. So try to find the most critical piece of data you can focus on initially. Manage project stakeholders’ expectations: In any domain, people tend to use different terms for the same things. Therefore, involving those who use the source system can help you explain the data behavior and identify any data anomalies that may arise. For instance, when we had to extract data from a widely-used EHR system, it was a surprise that patient information was stored in a table named ‘users.’ By collaborating with the end-users and the data source users, we were able to address such semantic inconsistencies and prevented any confusion down the line. Establish a shared understanding of semantics with both the users of the source system and the end users of the data: There are always some standardized reports generated from the legacy system. Typically you can use those reports as a set of reconciliation metrics. Don’t be surprised that you discover some inconsistencies and/or errors during your reconciliation process. But it’s always helpful to establish some baseline to evaluate the fullness and quality of the data. Establish early success criteria: There are many approaches and tools you can use to do the job. From plain all the way to tools. It’s up to you and your arsenal of skills, but here are some high-level suggestions that typically help: Ok, and what about the technical side of the project? SQL no-code It’s essential to understand how the data changes over time, including retrospective changes, in order to establish reliable Change Data Capture for the tables you’re interested in. Many legacy databases don’t have an OOTB CDC feature, so you may have to deal with a full data refresh. An alternative would be to use a data replication tool with CDC capabilities integrated. Tools like Airbyte and Fivetran may help to save time there. Change Data Capture (CDC): Table names alone can be misleading, so analyzing the database workload can help you better understand which tables are critical for your extraction process. It can help you to identify critical objects in the data model. Analyze the workload to identify the most and least used objects/tables: It’s always a good practice to extract the raw data and store it as is. Don’t perform any transformations in that step. Use your Data Lake or Staging area to store it until the iteration/project is complete. This minimizes the performance impact on the source database. This also enables you to compare the extracted data from the previous run to the current version, helping to identify changes you may have missed during implementation. Another advantage is the ability to rerun transformations without impacting the source database. Extract and store data in its raw form: When it comes to transforming the extracted data, it’s best to break it down into multiple steps rather than cramming everything into a single, complex set of instructions. This helps with the maintainability of the data transformation code and makes it easier for you and your team to understand what’s going on later on. Overcomplicated code can be a hindrance, especially when revisiting it after several iterations. Good documentation and structured code can save you and your team time. Structure transformations in multiple stages: Maintaining both the code and the documentation can be a lot of work. Having a no-code tool that can handle data lineage, transformations, and documentation can be helpful. When selecting a tool, always consider who will maintain the pipeline and how to make that process as efficient as possible. How technical that person will be. Some tools require more technical skills than others. Consider using a no-code tool: A semantic model can be extremely useful when integrating data from multiple sources into a single data model. Normalizing data and using transformations that can be demonstrated are highly beneficial features. This approach ensures that your data is well-structured and easy to understand, making it more accessible and simpler for data users to access and analyze. Create a Semantic Model: Data extraction can be challenging, but following these key takeaways can help you avoid issues, especially if you’re doing it for the first time. We intentionally skipped all parts related to security and compliance and assumed you have access to the database. It’s essential to build strong relationships with knowledge holders, manage project stakeholders’ expectations, establish a common definition of semantics with data users, set clear success criteria, and structure transformations in multiple stages. No-code automation tools like Datuum and data replication tools like Airbyte or Fivetran can simplify the process. Documenting transformations, maintaining data lineage, and creating a semantic model is crucial to establishing a sustainable and maintainable data pipeline. These steps will benefit the entire organization in the long run. Also published here.

Effective Strategies for Efficient Data Extraction

Five Weeks in a Balloon by Jules Verne, is part of the HackerNoon Books Series. You can jump to any chapter in this book . Chapter XII here CHAPTER TWELFTH. Crossing the Strait.—The Mrima.—Dick’s Remark and Joe’s Proposition.—A Recipe for Coffee-making.—The Uzaramo.—The Unfortunate Maizan.—Mount Dathumi.—The Doctor’s Cards.—Night under a Nopal. The air was pure, the wind moderate, and the balloon ascended almost perpendicularly to a height of fifteen hundred feet, as indicated by a depression of two inches in the barometric column. At this height a more decided current carried the balloon toward the southwest. What a magnificent spectacle was then outspread beneath the gaze of the travellers! The island of Zanzibar could be seen in its entire extent, marked out by its deeper color upon a vast planisphere; the fields had the appearance of patterns of different colors, and thick clumps of green indicated the groves and thickets. The inhabitants of the island looked no larger than insects. The huzzaing and shouting were little by little lost in the distance, and only the discharge of the ship’s guns could be heard in the concavity beneath the balloon, as the latter sped on its flight. “How fine that is!” said Joe, breaking silence for the first time. He got no reply. The doctor was busy observing the variations of the barometer and noting down the details of his ascent. Kennedy looked on, and had not eyes enough to take in all that he saw. The rays of the sun coming to the aid of the heating cylinder, the tension of the gas increased, and the attained the height of twenty-five hundred feet. Victoria The looked like a mere cockle-shell, and the African coast could be distinctly seen in the west marked out by a fringe of foam. Resolute “You don’t talk?” said Joe, again. “We are looking!” said the doctor, directing his spy-glass toward the mainland. “For my part, I must talk!” “As much as you please, Joe; talk as much as you like!” And Joe went on alone with a tremendous volley of exclamations. The “ohs!” and the “ahs!” exploded one after the other, incessantly, from his lips. During his passage over the sea the doctor deemed it best to keep at his present elevation. He could thus reconnoitre a greater stretch of the coast. The thermometer and the barometer, hanging up inside of the half-opened awning, were always within sight, and a second barometer suspended outside was to serve during the night watches. At the end of about two hours the , driven along at a speed of a little more than eight miles, very visibly neared the coast of the mainland. The doctor, thereupon, determined to descend a little nearer to the ground. So he moderated the flame of his cylinder, and the balloon, in a few moments, had descended to an altitude only three hundred feet above the soil. Victoria It was then found to be passing just over the Mrima country, the name of this part of the eastern coast of Africa. Dense borders of mango-trees protected its margin, and the ebb-tide disclosed to view their thick roots, chafed and gnawed by the teeth of the Indian Ocean. The sands which, at an earlier period, formed the coast-line, rounded away along the distant horizon, and Mount Nguru reared aloft its sharp summit in the northwest. The passed near to a village which the doctor found marked upon his chart as Kaole. Its entire population had assembled in crowds, and were yelling with anger and fear, at the same time vainly directing their arrows against this monster of the air that swept along so majestically away above all their powerless fury. Victoria The wind was setting to the southward, but the doctor felt no concern on that score, since it enabled him the better to follow the route traced by Captains Burton and Speke. Kennedy had, at length, become as talkative as Joe, and the two kept up a continual interchange of admiring interjections and exclamations. “Out upon stage-coaches!” said one. “Steamers indeed!” said the other. “Railroads! eh? rubbish!” put in Kennedy, “that you travel on, without seeing the country!” “Balloons! they’re the sort for me!” Joe would add. “Why, you don’t feel yourself going, and Nature takes the trouble to spread herself out before one’s eyes!” “What a splendid sight! What a spectacle! What a delight! a dream in a hammock!” “Suppose we take our breakfast?” was Joe’s unpoetical change of tune, at last, for the keen, open air had mightily sharpened his appetite. “Good idea, my boy!” “Oh! it won’t take us long to do the cooking—biscuit and potted meat?” “And as much coffee as you like,” said the doctor. “I give you leave to borrow a little heat from my cylinder. There’s enough and to spare, for that matter, and so we shall avoid the risk of a conflagration.” “That would be a dreadful misfortune!” ejaculated Kennedy. “It’s the same as a powder-magazine suspended over our heads.” “Not precisely,” said Ferguson, “but still if the gas were to take fire it would burn up gradually, and we should settle down on the ground, which would be disagreeable; but never fear—our balloon is hermetically sealed.” “Let us eat a bite, then,” replied Kennedy. “Now, gentlemen,” put in Joe, “while doing the same as you, I’m going to get you up a cup of coffee that I think you’ll have something to say about.” “The fact is,” added the doctor, “that Joe, along with a thousand other virtues, has a remarkable talent for the preparation of that delicious beverage: he compounds it of a mixture of various origin, but he never would reveal to me the ingredients.” “Well, master, since we are so far above-ground, I can tell you the secret. It is just to mix equal quantities of Mocha, of Bourbon coffee, and of Rio Nunez.” A few moments later, three steaming cups of coffee were served, and topped off a substantial breakfast, which was additionally seasoned by the jokes and repartees of the guests. Each one then resumed his post of observation. The country over which they were passing was remarkable for its fertility. Narrow, winding paths plunged in beneath the overarching verdure. They swept along above cultivated fields of tobacco, maize, and barley, at full maturity, and here and there immense rice-fields, full of straight stalks and purple blossoms. They could distinguish sheep and goats too, confined in large cages, set up on piles to keep them out of reach of the leopards’ fangs. Luxuriant vegetation spread in wild profuseness over this prodigal soil. Village after village rang with yells of terror and astonishment at the sight of the , and Dr. Ferguson prudently kept her above the reach of the barbarian arrows. The savages below, thus baffled, ran together from their huddle of huts and followed the travellers with their vain imprecations while they remained in sight. Victoria At noon, the doctor, upon consulting his map, calculated that they were passing over the Uzaramo country. The soil was thickly studded with cocoa-nut, papaw, and cotton-wood trees, above which the balloon seemed to disport itself like a bird. Joe found this splendid vegetation a matter of course, seeing that they were in Africa. Kennedy descried some hares and quails that asked nothing better than to get a good shot from his fowling-piece, but it would have been powder wasted, since there was no time to pick up the game. [2] U and Ou signify country in the language of that region. [2] The aëronauts swept on with the speed of twelve miles per hour, and soon were passing in thirty-eight degrees twenty minutes east longitude, over the village of Tounda. “It was there,” said the doctor, “that Burton and Speke were seized with violent fevers, and for a moment thought their expedition ruined. And yet they were only a short distance from the coast, but fatigue and privation were beginning to tell upon them severely.” In fact, there is a perpetual malaria reigning throughout the country in question. Even the doctor could hope to escape its effects only by rising above the range of the miasma that exhales from this damp region whence the blazing rays of the sun pump up its poisonous vapors. Once in a while they could descry a caravan resting in a “ ,” awaiting the freshness and cool of the evening to resume its route. These kraals are wide patches of cleared land, surrounded by hedges and jungles, where traders take shelter against not only the wild beasts, but also the robber tribes of the country. They could see the natives running and scattering in all directions at the sight of the . Kennedy was keen to get a closer look at them, but the doctor invariably held out against the idea. kraal Victoria “The chiefs are armed with muskets,” he said, “and our balloon would be too conspicuous a mark for their bullets.” “Would a bullet-hole bring us down?” asked Joe. “Not immediately; but such a hole would soon become a large torn orifice through which our gas would escape.” “Then, let us keep at a respectful distance from yon miscreants. What must they think as they see us sailing in the air? I’m sure they must feel like worshipping us!” “Let them worship away, then,” replied the doctor, “but at a distance. There is no harm done in getting as far away from them as possible. See! the country is already changing its aspect: the villages are fewer and farther between; the mango-trees have disappeared, for their growth ceases at this latitude. The soil is becoming hilly and portends mountains not far off.” “Yes,” said Kennedy, “it seems to me that I can see some high land on this side.” “In the west—those are the nearest ranges of the Ourizara—Mount Duthumi, no doubt, behind which I hope to find shelter for the night. I’ll stir up the heat in the cylinder a little, for we must keep at an elevation of five or six hundred feet.” “That was a grand idea of yours, sir,” said Joe. “It’s mighty easy to manage it; you turn a cock, and the thing’s done.” “Ah! here we are more at our ease,” said the sportsman, as the balloon ascended; “the reflection of the sun on those red sands was getting to be insupportable.” “What splendid trees!” cried Joe. “They’re quite natural, but they are very fine! Why a dozen of them would make a forest!” “Those are baobabs,” replied Dr. Ferguson. “See, there’s one with a trunk fully one hundred feet in circumference. It was, perhaps, at the foot of that very tree that Maizan, the French traveller, expired in 1845, for we are over the village of Deje-la-Mhora, to which he pushed on alone. He was seized by the chief of this region, fastened to the foot of a baobab, and the ferocious black then severed all his joints while the war-song of his tribe was chanted; he then made a gash in the prisoner’s neck, stopped to sharpen his knife, and fairly tore away the poor wretch’s head before it had been cut from the body. The unfortunate Frenchman was but twenty-six years of age.” “And France has never avenged so hideous a crime?” said Kennedy. “France did demand satisfaction, and the Said of Zanzibar did all in his power to capture the murderer, but in vain.” “I move that we don’t stop here!” urged Joe; “let us go up, master, let us go up higher by all means.” “All the more willingly, Joe, that there is Mount Duthumi right ahead of us. If my calculations be right we shall have passed it before seven o’clock in the evening.” “Shall we not travel at night?” asked the Scotchman. “No, as little as possible. With care and vigilance we might do so safely, but it is not enough to sweep across Africa. We want to see it.” “Up to this time we have nothing to complain of, master. The best cultivated and most fertile country in the world instead of a desert! Believe the geographers after that!” “Let us wait, Joe! we shall see by-and-by.” About half-past six in the evening the was directly opposite Mount Duthumi; in order to pass, it had to ascend to a height of more than three thousand feet, and to accomplish that the doctor had only to raise the temperature of his gas eighteen degrees. It might have been correctly said that he held his balloon in his hand. Kennedy had only to indicate to him the obstacles to be surmounted, and the sped through the air, skimming the summits of the range. Victoria Victoria At eight o’clock it descended the farther slope, the acclivity of which was much less abrupt. The anchors were thrown out from the car and one of them, coming in contact with the branches of an enormous nopal, caught on it firmly. Joe at once let himself slide down the rope and secured it. The silk ladder was then lowered to him and he remounted to the car with agility. The balloon now remained perfectly at rest sheltered from the eastern winds. The evening meal was got ready, and the aëronauts, excited by their day’s journey, made a heavy onslaught upon the provisions. “What distance have we traversed to-day?” asked Kennedy, disposing of some alarming mouthfuls. The doctor took his bearings, by means of lunar observations, and consulted the excellent map that he had with him for his guidance. It belonged to the Atlas of “Der Neuester Endeckungen in Afrika” (“The Latest Discoveries in Africa”), published at Gotha by his learned friend Dr. Petermann, and by that sent to him. This Atlas was to serve the doctor on his whole journey; for it contained the itinerary of Burton and Speke to the great lakes; the Soudan, according to Dr. Barth; the Lower Senegal, according to Guillaume Lejean; and the Delta of the Niger, by Dr. Blaikie. savant Ferguson had also provided himself with a work which combined in one compilation all the notions already acquired concerning the Nile. It was entitled “The Sources of the Nile; being a General Survey of the Basin of that River and of its Head-Stream, with the History of the Nilotic Discovery, by Charles Beke, D.D.” He also had the excellent charts published in the “Bulletins of the Geographical Society of London;” and not a single point of the countries already discovered could, therefore, escape his notice. Upon tracing on his maps, he found that his latitudinal route had been two degrees, or one hundred and twenty miles, to the westward. Kennedy remarked that the route tended toward the south; but this direction was satisfactory to the doctor, who desired to reconnoitre the tracks of his predecessors as much as possible. It was agreed that the night should be divided into three watches, so that each of the party should take his turn in watching over the safety of the rest. The doctor took the watch commencing at nine o’clock; Kennedy, the one commencing at midnight; and Joe, the three o’clock morning watch. So Kennedy and Joe, well wrapped in their blankets, stretched themselves at full length under the awning, and slept quietly; while Dr. Ferguson kept on the lookout. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Jules Verne (2002). Five Weeks in a Balloon. Urbana, Illinois: Project Gutenberg. Retrieved October 2022 https://www.gutenberg.org/cache/epub/3526/pg3526-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Crossing the Strait

The abundance of tabular data being accumulated by enterprises internally and online represents a big challenge for data integration and analysis for companies that strive to be data-driven. While the main focus of AI/ML research and community during the last several years has been on processing unstructured data, tabular data still is where the most time and money are spent in the Data Integration world. These days, every company, whether big or small, has dozens of different systems (data sources). Many of those have great integration capabilities, but still, every day people export data from various systems and ingest it into proprietary enterprise Data Lakes, Data Warehouses, Reporting databases, etc. That requires a lot of time and qualified Data Engineers. To keep up with quickly changing business requirements and growing data size, there have been many great technologies and tools developed by companies and communities: big data stack, data wrangling tools, quality and observability, visualization, and NLU interfaces to the data. All those technologies allowed the collection and processing of unprecedented amounts of data. Though it also makes the modern data engineering stack very complex, which, in turn, results in long and expensive development cycles that don’t meet business requirements Tools assume that the user understands the data, knows how it’s structured, and how to cross-reference data that comes from different sources. And that represents probably the biggest problem in today’s corporate world. In order to do something with the data you have to understand it, know where it is, and how it’s structured. There is a misconception that there are people who fully understand all the nuances of companies' data and can efficiently manage it. And it becomes exponentially more difficult when a company grows and people change. In its work, Datuum specifically targets this challenge by applying state-of-the-art AI models to help Data Workers handle the variety and complexity of tabular data. Datuum speeds up and optimizes the integration of multiple data sources by helping data workers find the data they need and connect the dots between source and destination. As a result of interacting with your data, our AI learns more with every iteration and that data so the institutional knowledge is not lost. Why Is It Not Easy? Imagine that you have two different data sources with seemingly similar data stored in them. For instance, data from medical or insurance domains, in which some tables could contain hundreds or even thousands of columns. Our major task is to unify this data into a single source of truth. The cornerstone of this unification is the process of column matching – discovering combinations of columns from different data sources which store the same information (Fig. 1). While it could be trivial to match columns with generalized data like name, email, or address, it's not the case for columns containing domain-specific information – to map this kind of data, we usually need domain expertise and manual intervention. The other important phase of the unification pipeline would be record matching – the process in which records from different data sources are merged into a single entity. For highly regulated domains like healthcare, this step could be extremely important. And it's even harder to get right when large datasets are being processed, even for human annotators. Transformers to the Rescue! Thankfully, the latest research in large language models gives us the key to the simplification of the matching process. It doesn't yet fully remove the need for human experts (especially in complex domains) but could give them a substantial boost compared to standard settings. By definition, language modeling is the usage of various statistical and probabilistic techniques to determine the probability of a given sequence of words occurring in a sentence. Language models train by analyzing text to correctly predict the next or masked word in a sequence (Fig. 2). The task of language modeling is not new, but in recent years, it has become one of the main areas for research and development in the machine learning field. There are 3 reasons why this sub-field experienced such enormous progress: The nature of language modeling tasks allowed us to use tremendous amounts of data for unsupervised learning. Transformer attention-based architecture allowed the massive parallelism of training on sequential data. Transformers pre-trained on a language modeling task turned out to be great at solving other, unrelated to language modeling, tasks. As a result, transformer models have found use in a wide range of tasks, ranging from question answering to video understanding. Naturally, people also looked at applying this architecture to tabular data as well. Tabular data is textual in its nature. One could argue that numeric columns do not constitute text, but in fact, numerical values are nothing other than sequences of specialized text characters. This means that tabular data, similar to textual one, could be effectively modeled as a language using Transformers. Modeling tabular data with Transformers facilitates the solution of other downstream tasks, such as column type prediction or column name recovery. Some of the proposed models could also learn efficient dense representations of the table parts: columns and rows. Deep Learning Journey Machine learning approaches using tabular data can hardly be called new. For many years, tree-based solutions, such as random forests or gradient boosting algorithms, were used for various classification and regression tasks, such as credit score prediction or fraud detection. However, the complexity of the aforementioned models lets us learn and understand patterns of data on the observation (row) level. For a much deeper understanding of data, as relations between columns in a table or between tables in a database, neural networks became very handy. In the pre-transformer era, deep learning models had a limited capability in processing sequential data like column values or names and, hence, learning contextual information from it. As a result, they often relied on manually engineered features for model input and processed each column separately. One prominent example of such a model is Sherlock [1] – the first successful application of deep learning to the task of semantic type classification. Sherlock is a regular feed-forward neural network that uses a set of statistical features calculated from column values to predict one of 78 semantic column types. It was the first deep-learning model used to improve column matching in Datuum. Despite being quite effective for common column types, like name, address, or zip code, Sherlock had a set of limitations, such as high reliance on annotated data, especially in previously unseen domains, or inability to take table context and column name into account. Obviously, that represented an obstacle to successfully applying Datuum in new verticals fast. As that architecture required additional training on the new data. Fortunately enough, the Transformer-based model proved to be a perfect alternative to Sherlock. The Case for Transformers Our history with the application of transformers to the task of column matching started with extensive research of related work. As a result, we chose TABBIE [2] and TaBERT [3] models for further experiments. It should be noted that at the beginning of 2022, almost all SOTA approaches to the task of column-type classification were based on Transformers (Fig. 3). By fine-tuning these models on multiple relevant datasets, we proved the superiority of transformer-based methods over Sherlock. For example, the TABBIE F1-macro score on the TURL dataset [4] was almost 20% higher than Sherlock’s. In fact, the performance of transformers on classification tasks wasn’t what we were looking for. The property that caught our attention most was the representational ability of these models. For instance, by training transformer-based models like TABBIE on domain-specific datasets for the task of semantic column type classification, we were able to learn dense representations of the elements of these tables: rows and columns. This, in turn, allowed us to extend the application of the model to previously unseen domains without the need to annotate new data. Our column-matching pipeline is specifically designed for a system that has a regular inflow of new data from previously unseen domains (Fig. 4). Generally, our approach works the following way: We fine-tuneTABBIE model with domain-specific data for a task that does not require additional annotation. For example, corrupted cell detection or cell in-filling. Such training is called semi-supervised. The model learns dense representations of columns that leverage both column information and table context. At runtime, we encode columns from tables that need to be matched and combine them into matching pairs based on distance measures between dense representations in vector space. Human experts review low-confidence matches for correctness. This pipeline has a few important benefits compared to the pipeline based on the Sherlock model, namely: Matching is not limited to a defined set of column types. The need for annotation is eliminated. Ability to include encoded column data, metadata, and context into a single representation. Combined, these advantages set a new standard in the accuracy and adaptivity of column matching employed in Datuum. Conclusion The representational ability of Transformer-based language models is bringing new capabilities to almost all tasks that rely on textual data, and table cross-reference is no exception. By employing a Transformer model in our column-matching pipeline, we were able to increase the matching accuracy and generalization ability of the system that doesn’t depend on annotated data, which is especially useful in domains where data access is restricted. References https://hci.stanford.edu/\~cagatay/projects/table-understanding/Sherlock-KDD19.pdf https://arxiv.org/pdf/2105.02584.pdf https://arxiv.org/pdf/2005.08314.pdf https://arxiv.org/pdf/2006.14806.pdf Also published here

Datuum.ai

datuum.ai

11-50 employees

Since 2021

COMPANY RANKING

DATUUM.AI

EVERGREEN INDEX #2775

Datuum.ai WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Datuum.ai

datuum.ai

11-50 employees

Since 2021

COMPANY RANKING

DATUUM.AI

EVERGREEN INDEX #2775

RELATED COMPANIESRANK

STORIES ABOUT DATUUM.AI

Datuum.ai WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!