When Jara Helmes was a toddler walking with stiff, heavy leg braces, her father made her a lighter, softer pair 3D-printed in her favorite color, pink. When

New mix-and-match computer accessories give people with … – Microsoft

Summary Informatica Inc. provides software and services for data integration and management in companies worldwide. The company has transformed into a cloud-based SaaS company, offering various data management solutions.

Informatica Ups Guidance But Valuation Looks Full

Vice President & Principal Analyst, Enterprise Data Technologies, Robert Kramer explains how Informatica is boosting its capabilities in enterprise data management.

Informatica Embraces AI To Help Companies Manage Their Data

It said that Edinburgh, the Scottish capital, had been nominated to host the computer, bringing high-skilled jobs to the area. According to TOP500, the current best supercomputer in the UK is the ARCHER2 which is also operated from Edinburgh. It has a ...

UK to build exascale computer 50 times more powerful than its best supercomputer0 0

The Visual Computing Group at Microsoft Research Asia consists of an elite team of researchers whose expertise spans the spectrum of research topics in computer vision, from mathematical theory to practical applications, from physical systems to software ...

Visual Computing

Become a MEED subscriber for unlimited access to: Exclusive news, comment and analysis on the MENA region An unrivalled 20-year old archive Exclusive announcements on project news, bids, awards and updates The latest Industry data and competitor insights

Bahrain: Computer hardware and software

In this article, I will delve into whether artificial intelligence (AI), particularly the emergence of tools like generative AI (GenAI), will replace classical software outsourcing as we know it.

AI vs software outsourcing: An opportunity or a threat?

MONTREAL — The Canadian National Railway computer outage that disrupted GO Transit and VIA Rail Canada trains on Tuesday was caused by a software upgrade, CN said on Wednesday. CN spokesman Jonathan Abecassis told CTV News that the railroad had ...

Software upgrade led to CN computer outage

During the ASAP’s fourth quarterly meeting of 2020, the panel highlighted the absence of the ability to test a complete set of the flight computer avionics and software that will fly on upcoming ...

ASAP urges NASA to build cross-program computer test...ASAP urges NASA to build cross-program computer test capability for Artemis

Australian software company Atlassian released emergency security updates to fix a maximum severity zero-day vulnerability in its Confluence Data Center and Server software, which has been ...

Atlassian patches critical Confluence zero-day exploited in attacks

Informatica

informatica

Comparison of Machine Learning Methods: Conclusions and Future Work, and References

Overview and Practical Examples of CDC Pattern With Kafka Connect

Authors:
(1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University;
(2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Related work Evaluation for Max-Cut Evaluation for SAT Summary and Outlook, References Supplementary Materials 5 SUMMARY and OUTLOOK Through our empirical evaluations, our goal is to promote an insightful comparison within the research focusing on the intersection of combinatorial optimization and machine learning. In order to provide the research community with valuable guidance, we believe it is imperative to communicate both the strengths and weaknesses of the proposed approaches. Poor instances and baseline selection may give the wrong impression about the performance of learned heuristics. Specifically, it is important to articulate the degree of improvement achieved through integrating classical heuristics with deep learning architectures and conduct a thorough comparison with classical heuristics. This will aid in elucidating the degree to which deep learning architectures enhance integrated heuristics. It assists in ascertaining whether the integration endeavor is justified and warrants the allocation of computational resources, time, and investment necessary for integrating deep learning with classical heuristics. References Albert, R. and Barab´asi, A.-L. (2002). Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47. Barrett, T., Clements, W., Foerster, J., and Lvovsky, A. (2020). Exploratory combinatorial optimization with reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3243–3250. Barrett, T. D., Parsonson, C. W., and Laterre, A. (2022). Learning to solve combinatorial graph partitioning problems via efficient exploration. arXiv preprint arXiv:2205.14105. Battiti, R. and Protasi, M. (1997). Reactive search, a history-sensitive heuristic for max-sat. Journal of Experimental Algorithmics (JEA), 2:2–es. Battiti, R. and Tecchiolli, G. (1994). The reactive tabu search. ORSA journal on computing, 6(2):126–140. Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. (2016). Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940. Bengio, Y., Lodi, A., and Prouvost, A. (2021). Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290(2):405–421. Benlic, U. and Hao, J.-K. (2013). Breakout local search for the max-cutproblem. Engineering Applications of Artificial Intelligence, 26(3):1162–1173. Boettcher, S. and Percus, A. G. (2001). Extremal optimization for graph partitioning. Physical Review E, 64(2):026114. CPLEX (2023). Cplex. https://www.ibm.com/products/ilog-cplexoptimization-studio/cplex-optimizer. Dong, Y., Goldberg, A. V., Noe, A., Parotsidis, N., Resende, M. G., and Spaen, Q. (2021). New instances for maximum weight independent set from a vehicle routing application. In Operations Research Forum, volume 2, pages 1–6. Springer. E´en, N. and S¨orensson, N. (2003). An extensible satsolver. In International conference on theory and applications of satisfiability testing, pages 502–518. Springer. Elsokkary, N., Khan, F. S., La Torre, D., Humble, T. S., and Gottlieb, J. (2017). Financial portfolio management using d-wave quantum optimizer: The case of abu dhabi securities exchange. Technical report, Oak Ridge National Lab.(ORNL), Oak Ridge, TN (United States). Erd˝os, P., R´enyi, A., et al. (1960). On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5(1):17–60. Festa, P. and Resende, M. G. (2009). Hybrid grasp heuristics. In Foundations of Computational Intelligence Volume 3: Global Optimization, pages 75–100. Springer. Flint, A. and Blaschko, M. (2012). Perceptron learning of sat. Advances in neural information processing systems, 25. Fujishige, S. (2005). Submodular functions and optimization. Elsevier. Ganesh, V., Singh, R., Near, J. P., and Rinard, M. (2009). Avatarsat: An auto-tuning boolean sat solver. Glover, F. (1990). Tabu search: A tutorial. Interfaces, 20(4):74–94. Goemans, M. X. and Williamson, D. P. (1995). Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6):1115–1145. Grozea, C. and Popescu, M. (2014). Can machine learning learn a decision oracle for np problems? a test on sat. Fundamenta Informaticae, 131(3- 4):441–450. Haim, S. and Walsh, T. (2009). Restart strategy selection using machine learning techniques. In International Conference on Theory and Applications of Satisfiability Testing, pages 312–325. Springer. Helmberg, C. and Rendl, F. (2000). A spectral bundle method for semidefinite programming. SIAM Journal on Optimization, 10(3):673–696. Hoos, H. H. et al. (2002). An adaptive noise mechanism for walksat. In AAAI/IAAI, pages 655–660. Jaszczur, S., Luszczyk, M., and Michalewski, H. (2020). Neural heuristics for sat solving. arXiv preprint arXiv:2005.13406. Karp, R. M. (1972). Reducibility among Combinatorial Problems, pages 85–103. Springer US, Boston, MA. Khalil, E., Dai, H., Zhang, Y., Dilkina, B., and Song, L. (2017). Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30. KhudaBukhsh, A. R., Xu, L., Hoos, H. H., and LeytonBrown, K. (2016). Satenstein: Automatically building local search sat solvers from components. Artificial Intelligence, 232:20–42. Kirkpatrick, S., Gelatt Jr, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. science, 220(4598):671–680. Kurin, V., Godil, S., Whiteson, S., and Catanzaro, B. (2020). Can q-learning with graph networks learn a generalizable branching heuristic for a sat solver? Advances in Neural Information Processing Systems, 33:9608–9621. Lauria, M., Elffers, J., Nordstr¨om, J., and Vinyals, M. (2017). Cnfgen: A generator of crafted benchmarks. In Theory and Applications of Satisfiability Testing–SAT 2017: 20th International Conference, Melbourne, VIC, Australia, August 28–September 1, 2017, Proceedings 20, pages 464–473. Springer. Lederman, G., Rabe, M., Seshia, S., and Lee, E. A. (2019). Learning heuristics for quantified boolean formulas through reinforcement learning. In International Conference on LearningRepresentations. Leleu, T., Yamamoto, Y., McMahon, P. L., and Aihara, K. (2019). Destabilization of local minima in analog spin systems by correction of amplitude heterogeneity. Physical review letters,122(4):040607. Li, C. M., Wei, W., and Zhang, H. (2007). Combining adaptive noise and look-ahead in local search for sat. In Theory and Applications of Satisfiability Testing– SAT 2007: 10th International Conference, Lisbon, Portugal, May 28-31, 2007. Proceedings 10, pages 121–133. Springer. Liang, J. H., Ganesh, V., Poupart, P., and Czarnecki, K. (2016). Learning rate based branching heuristic for sat solvers. In Theory and Applications of Satisfiability Testing–SAT 2016: 19th International Conference, Bordeaux, France, July 5-8, 2016, Proceedings 19, pages 123–140. Springer. Manchanda, S., Mittal, A., Dhawan, A., Medya, S., Ranu, S., and Singh, A. (2019). Learning heuristics over large graphs via deep reinforcement learning. arXiv preprint arXiv:1903.03332. Mazure, B., Sais, L., and Gr´egoire, E. (1997). Tabu ´ search for sat. In Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence, pages 281–285. McAllester, D., Selman, B., Kautz, H., et al. (1997). Evidence for invariants in local search. In AAAI/IAAI, pages 321–326. Rhode Island, USA. Mitchell, D., Selman, B., Levesque, H., et al. (1992). Hard and easy distributions of sat problems. In Aaai, volume 92, pages 459–465. Perdomo-Ortiz, A., Dickson, N., Drew-Brook, M., Rose, G., and Aspuru-Guzik, A. (2012). Finding low-energy conformations of lattice protein models by quantum annealing. Scientific reports, 2(1):1–7. Selman, B., Kautz, H. A., Cohen, B., et al. (1993). Local search strategies for satisfiability testing. Cliques, coloring, and satisfiability, 26:521–532. Selman, B., Mitchell, D. G., and Levesque, H. J. (1996). Generating hard satisfiability problems. Artificial intelligence, 81(1-2):17–29. Tiunov, E. S., Ulanov, A. E., and Lvovsky, A. (2019). Annealing by simulating the coherent ising machine. Optics express, 27(7):10288–10295. Toenshoff, J., Ritzert, M., Wolf, H., and Grohe, M. (2019). Graph neural networks for maximum constraint satisfaction. Frontiers in artificial intelligence, 3:580607. Venturelli, D. and Kondratyev, A. (2019). Reverse quantum annealing approach to portfolio optimization problems. Quantum Machine Intelligence, 1(1- 2):17–30. Wolpert, D. H. and Macready, W. G. (1997). No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82. Ye, Y. (2003). The gset dataset. https://web.stanford.edu/ yyye/yyye/Gset/. Yolcu, E. and P´oczos, B. (2019). Learning local search heuristics for boolean satisfiability. Advances in Neural Information Processing Systems, 32. This paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Authors: Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Abstract & Introduction Related work Related work Evaluation for Max-Cut Evaluation for Max-Cut Evaluation for SAT Evaluation for SAT Summary and Outlook, References Summary and Outlook, References Supplementary Materials Supplementary Materials 5 SUMMARY and OUTLOOK Through our empirical evaluations, our goal is to promote an insightful comparison within the research focusing on the intersection of combinatorial optimization and machine learning. In order to provide the research community with valuable guidance, we believe it is imperative to communicate both the strengths and weaknesses of the proposed approaches. Poor instances and baseline selection may give the wrong impression about the performance of learned heuristics. Specifically, it is important to articulate the degree of improvement achieved through integrating classical heuristics with deep learning architectures and conduct a thorough comparison with classical heuristics. This will aid in elucidating the degree to which deep learning architectures enhance integrated heuristics. It assists in ascertaining whether the integration endeavor is justified and warrants the allocation of computational resources, time, and investment necessary for integrating deep learning with classical heuristics. References Albert, R. and Barab´asi, A.-L. (2002). Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47. Barrett, T., Clements, W., Foerster, J., and Lvovsky, A. (2020). Exploratory combinatorial optimization with reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3243–3250. Barrett, T. D., Parsonson, C. W., and Laterre, A. (2022). Learning to solve combinatorial graph partitioning problems via efficient exploration. arXiv preprint arXiv:2205.14105. Battiti, R. and Protasi, M. (1997). Reactive search, a history-sensitive heuristic for max-sat. Journal of Experimental Algorithmics (JEA), 2:2–es. Battiti, R. and Tecchiolli, G. (1994). The reactive tabu search. ORSA journal on computing, 6(2):126–140. Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. (2016). Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940. Bengio, Y., Lodi, A., and Prouvost, A. (2021). Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290(2):405–421. Benlic, U. and Hao, J.-K. (2013). Breakout local search for the max-cutproblem. Engineering Applications of Artificial Intelligence, 26(3):1162–1173. Boettcher, S. and Percus, A. G. (2001). Extremal optimization for graph partitioning. Physical Review E, 64(2):026114. CPLEX (2023). Cplex. https://www.ibm.com/products/ilog-cplexoptimization-studio/cplex-optimizer. Dong, Y., Goldberg, A. V., Noe, A., Parotsidis, N., Resende, M. G., and Spaen, Q. (2021). New instances for maximum weight independent set from a vehicle routing application. In Operations Research Forum, volume 2, pages 1–6. Springer. E´en, N. and S¨orensson, N. (2003). An extensible satsolver. In International conference on theory and applications of satisfiability testing, pages 502–518. Springer. Elsokkary, N., Khan, F. S., La Torre, D., Humble, T. S., and Gottlieb, J. (2017). Financial portfolio management using d-wave quantum optimizer: The case of abu dhabi securities exchange. Technical report, Oak Ridge National Lab.(ORNL), Oak Ridge, TN (United States). Erd˝os, P., R´enyi, A., et al. (1960). On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5(1):17–60. Festa, P. and Resende, M. G. (2009). Hybrid grasp heuristics. In Foundations of Computational Intelligence Volume 3: Global Optimization, pages 75–100. Springer. Flint, A. and Blaschko, M. (2012). Perceptron learning of sat. Advances in neural information processing systems, 25. Fujishige, S. (2005). Submodular functions and optimization. Elsevier. Ganesh, V., Singh, R., Near, J. P., and Rinard, M. (2009). Avatarsat: An auto-tuning boolean sat solver. Glover, F. (1990). Tabu search: A tutorial. Interfaces, 20(4):74–94. Goemans, M. X. and Williamson, D. P. (1995). Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6):1115–1145. Grozea, C. and Popescu, M. (2014). Can machine learning learn a decision oracle for np problems? a test on sat. Fundamenta Informaticae, 131(3- 4):441–450. Haim, S. and Walsh, T. (2009). Restart strategy selection using machine learning techniques. In International Conference on Theory and Applications of Satisfiability Testing, pages 312–325. Springer. Helmberg, C. and Rendl, F. (2000). A spectral bundle method for semidefinite programming. SIAM Journal on Optimization, 10(3):673–696. Hoos, H. H. et al. (2002). An adaptive noise mechanism for walksat. In AAAI/IAAI, pages 655–660. Jaszczur, S., Luszczyk, M., and Michalewski, H. (2020). Neural heuristics for sat solving. arXiv preprint arXiv:2005.13406. Karp, R. M. (1972). Reducibility among Combinatorial Problems, pages 85–103. Springer US, Boston, MA. Khalil, E., Dai, H., Zhang, Y., Dilkina, B., and Song, L. (2017). Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30. KhudaBukhsh, A. R., Xu, L., Hoos, H. H., and LeytonBrown, K. (2016). Satenstein: Automatically building local search sat solvers from components. Artificial Intelligence, 232:20–42. Kirkpatrick, S., Gelatt Jr, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. science, 220(4598):671–680. Kurin, V., Godil, S., Whiteson, S., and Catanzaro, B. (2020). Can q-learning with graph networks learn a generalizable branching heuristic for a sat solver? Advances in Neural Information Processing Systems, 33:9608–9621. Lauria, M., Elffers, J., Nordstr¨om, J., and Vinyals, M. (2017). Cnfgen: A generator of crafted benchmarks. In Theory and Applications of Satisfiability Testing–SAT 2017: 20th International Conference, Melbourne, VIC, Australia, August 28–September 1, 2017, Proceedings 20, pages 464–473. Springer. Lederman, G., Rabe, M., Seshia, S., and Lee, E. A. (2019). Learning heuristics for quantified boolean formulas through reinforcement learning. In International Conference on LearningRepresentations. Leleu, T., Yamamoto, Y., McMahon, P. L., and Aihara, K. (2019). Destabilization of local minima in analog spin systems by correction of amplitude heterogeneity. Physical review letters,122(4):040607. Li, C. M., Wei, W., and Zhang, H. (2007). Combining adaptive noise and look-ahead in local search for sat. In Theory and Applications of Satisfiability Testing– SAT 2007: 10th International Conference, Lisbon, Portugal, May 28-31, 2007. Proceedings 10, pages 121–133. Springer. Liang, J. H., Ganesh, V., Poupart, P., and Czarnecki, K. (2016). Learning rate based branching heuristic for sat solvers. In Theory and Applications of Satisfiability Testing–SAT 2016: 19th International Conference, Bordeaux, France, July 5-8, 2016, Proceedings 19, pages 123–140. Springer. Manchanda, S., Mittal, A., Dhawan, A., Medya, S., Ranu, S., and Singh, A. (2019). Learning heuristics over large graphs via deep reinforcement learning. arXiv preprint arXiv:1903.03332. Mazure, B., Sais, L., and Gr´egoire, E. (1997). Tabu ´ search for sat. In Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence, pages 281–285. McAllester, D., Selman, B., Kautz, H., et al. (1997). Evidence for invariants in local search. In AAAI/IAAI, pages 321–326. Rhode Island, USA. Mitchell, D., Selman, B., Levesque, H., et al. (1992). Hard and easy distributions of sat problems. In Aaai, volume 92, pages 459–465. Perdomo-Ortiz, A., Dickson, N., Drew-Brook, M., Rose, G., and Aspuru-Guzik, A. (2012). Finding low-energy conformations of lattice protein models by quantum annealing. Scientific reports, 2(1):1–7. Selman, B., Kautz, H. A., Cohen, B., et al. (1993). Local search strategies for satisfiability testing. Cliques, coloring, and satisfiability, 26:521–532. Selman, B., Mitchell, D. G., and Levesque, H. J. (1996). Generating hard satisfiability problems. Artificial intelligence, 81(1-2):17–29. Tiunov, E. S., Ulanov, A. E., and Lvovsky, A. (2019). Annealing by simulating the coherent ising machine. Optics express, 27(7):10288–10295. Toenshoff, J., Ritzert, M., Wolf, H., and Grohe, M. (2019). Graph neural networks for maximum constraint satisfaction. Frontiers in artificial intelligence, 3:580607. Venturelli, D. and Kondratyev, A. (2019). Reverse quantum annealing approach to portfolio optimization problems. Quantum Machine Intelligence, 1(1- 2):17–30. Wolpert, D. H. and Macready, W. G. (1997). No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82. Ye, Y. (2003). The gset dataset. https://web.stanford.edu/ yyye/yyye/Gset/. Yolcu, E. and P´oczos, B. (2019). Learning local search heuristics for boolean satisfiability. Advances in Neural Information Processing Systems, 32. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv

Assessing the Justification for Integrating Deep Learning in Combinatorial Optimization

As we approach the second quarter of 2024, Artificial Intelligence (AI) is driving significant changes in the field of data engineering. Integrating data engineering with AI has led to the popularity of modern data integration and the expertise required. I want to highlight here how the rise of AI will change data engineering trends in 2024 compared to previous years. First thing first, a quick recap on traditional data engineering Let's quickly recap how the data engineering role is born and grows with the passage of time. Almost a decade ago, businesses realised the importance of data-driven decisions. That opened roles like with expertise in Microsoft SSIS/SSRS, Talend, Informatica, and other similar tools. However, with the emergence of social media applications, these traditional ETL development tools faced limitations in processing billions of records. The idea of distributed storage and processing came into the field of play, such as Spark and NoSQL databases. As a result, the demand for Hadoop/Spark has increased. Exciting Hadoop times ahead! BI/ETL Developers Big Data Developers In 2006, AWS launched its cloud services S3 and EC2, followed by Microsoft and Google launching Azure and Google Cloud in 2008. This led to an increase in the use of cloud technologies. The cloud technologies are based on a pay-as-you-use model that avoids buying expensive servers and operating system licenses. These cloud technologies have now been called "Modern Data Warehouses" since the union of Big Data and Cloud. In early 2010, r jobs rose in the market with the release of AWS EMR, etc.  Because now, each and every company wants to migrate from on-premise to the cloud. With the passage of time, modern cloud-based data warehousing solutions have become costly and inefficient in handling unstructured data, leading to the birth of file-based data lake solutions. Cloud Develope Different Variations of Data Engineers The Data Engineering Landscape is quite complex and vast. Engineers and businesses struggle to keep up with the latest tools and technologies to build and maintain a scalable data platform. Each business creates its definition of data engineers, and some require data engineers to focus primarily on creating pipelines. Some require software engineering and reverse engineering expertise or the ability to build KPI models as an analytics engineer along with different cloud-based specializations. This is the current state of Data Engineering variation and demand, but  this is not stopping here. I want to highlight some other trends that are rising this year. AI Meets Data Engineers Recently, I saw on LinkedIn that many positions are open for . They mostly wrote that the responsibilities are to build LLM Models or prepare the batch/streaming data for these AI-based models. This includes regular data engineering tasks to build pipelines for product feature enhancement and enable data-driven business with a centralized data warehouse or data lake solution. AI Data Engineers Quality Assurance Engineer Merge with Data Engineer As a Data Engineer by profession, I realise that a data team should have at least one Quality Assurance Engineer. The QA takes all the responsibility for testing data quality, as they used to do web or mobile application testing. That person is always up-to-date with business domain knowledge. While the engineers are mostly focusing on technical aspects. With the rising demand for Gen AI solutions like GitHub Codepilot and ChatGPT, there is no guarantee that their solutions are always reliable; hence, it increases data quality problems with their use in production. Quality is one of the biggest challenges for small to enterprise companies. You are definitely going to see a lot more job openings as in the coming days.  This role is going to be the next demanding job in the coming months. Data Quality Engineer Batch Data Processing Could Obsolete Soon Stream data integration and processing keep adding value daily for the business allowing them to make faster decisions and offer better product features. The CDC (Change Data Capture) is already taking place for batch pipeline integration, primarily for the main source database. However, Due to less demand for batch processing and the high output of stream processing architecture, it may slowly become obsolete or less required in the future. Lakehouse Architecture Will Remain The First Choice Hundreds and thousands of companies have already taken advantage of building Lakehouse Architecture. They moved from the costly “Modern cloud data warehouse“ to the file-based Lakehouse Architecture either using Delta Lake, Apache Hudi or Apache Iceberg open storage format, as its ability to manage large-scale data with ACID transactions and schema enforcement will remain the first choice for next some year(s) unless we see any other new tools or architecture in the market. The popularity of No Code Data Integration Tools Modern Data Integration Tools like Airbyte, Mage.ai, Stitch, and Fivetran simplify data integration and workflow and reduce the need for custom Python-based data pipeline development. These tools speed up the process of pipeline development, but I will come up with the cost of the complexity of debugging these tools if something breaks; it will not be very straightforward to fix it. On the other hand, expert engineers will be required to kick-start the pipeline integration in one click. I would like to see how long we can go to adapt these tools, no-code-based tools.

The Emerging Data Engineering Trends You Should Check Out In 2024

This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: (1) Tomáš Jakl, Czech Academy of Sciences and Czech Technical University; (2) Dan Marsden, School of Computer Science University of Nottingham; (3) Nihil Shah, Department of Computer Science University of Oxford. Table of Links Abstract & Introduction Prelimenaries FVM Theorems for Positive Existential Fragments FVM Theorems for Counting Logic FVM Theorems for The Full Logic Abstract FVM Theorems for Products Adding Equality and Other Enrichment Conclusions, Acknowledgments & References Appendix A FVM theorems for coproducts Appendix B Proofs Omitted from Section III Appendix C Proofs Omitted from Section IV Appendix D Proofs Omitted from Section V Appendix E Proofs Omitted from Section VI Appendix F Proofs Omitted from Section VII VIII. CONCLUSION We presented a categorical approach to the composition method, and specifically Feferman–Vaught–Mostowski theorems. We exploit game comonads to encapsulate the logics and their model comparison games. Surprising connections to classical constructions in category theory, and especially the monad theory of bilinear maps emerged from this approach, cf. Section V-A. For finite model theorists, our work provides a novel highlevel account of many FVM theorems, abstracting away from individual logics and constructions. Furthermore, concrete instances of these theorems are verified purely semantically, by finding a suitable collection homomorphisms forming a Kleisli law satisfying (S2’), instead of the usual delicate verification that strategies of model comparison games compose. For game comonads, we provide a much needed tool that enables us to handle logical relationships between structures as they are transformed or viewed in terms of different logics. The FVM results in this paper, combined with judicious use of the techniques described in Section VII can be pushed significantly further. FVM theorems and the compositional method are of particular significance in the setting of monadic second-order (MSO) logic [49]. Exploiting the results we have presented for the composition method, a follow up paper will give a comonadic semantics for MSO, and develop a semantic account of Courcelle’s algorithmic meta-theorems [5]. The original theorem of Feferman–Vaught [3] is very flexible, including incorporating structure on the indices of families of models to be combined by an operation, as well as the models themselves. This aspect is currently outside the scope of our comonadic methods. An ad-hoc adaptation of our approach also gives a proof of the FVM theorem for free amalgamations, but this does not follow directly from our theorems. Developing these extensions is left to future work. The present work bears some resemblance to Turi and Plotkin’s bialgebraic semantics [50], a categorical model of structural operational semantics (SOS) [51]. Turi and Plotkin noticed that the SOS rules assigning behaviour to syntax could be abstracted as a certain distributive law λ. An algebra encodes the composition operations of the syntax, and a coalgebra the behaviour. If this pair is suitably compatible with λ, forming a so-called λ-bialgebra, then crucially bisimulation is a congruence with respect to the composition operations. Both bialgebraic semantics and our work presented in this paper give a categorical account of well-behaved composition operations, with the interaction between composition and observable behaviour mediated by some form of distributive law. There are also essential differences: bialgebraic semantics typically focuses on assigning behaviour to syntax, whereas FVM theorems encompass operations on models. Our FVM theorems are parametric in a choice of logic or observable behaviour, while the notion of bisimulation in bialgebraic semantics is fixed by the coalgebra signature functor. The natural presence of positive existential and counting quantifier variants of FVM theorems, and the incorporation of resource parameters, do not seem to have a direct analogue in bialgebraic semantics. The similarities are intriguing, and exploring the relationship between bialgebraic semantics and our approach to FVM theorems is left to future work. We would like to thank Clemens Berger for suggesting the connection with parametric adjoints and Nathanael Arkor for his suggestions on terminology. We are also grateful to the members of the EPSRC project “Resources and co-resources” for their feedback. Acknowledgements: REFERENCES [1] L. Libkin, Elements of finite model theory. Springer, 2004, vol. 41. [2] A. Mostowski, “On direct products of theories,” The Journal of Symbolic Logic, vol. 17, no. 1, pp. 1–31, 1952. [3] S. Feferman and R. L. Vaught, “The first-order properties of algebraic systems,” Fundamenta Mathematicae, vol. 47, 1959. [4] J. A. Makowsky, “Algorithmic uses of the Feferman–Vaught theorem,” Annals of Pure and Applied Logic, vol. 126, no. 1-3, pp. 159–213, 2004. [5] B. Courcelle and J. Engelfriet, Graph structure and monadic secondorder logic: a language-theoretic approach. Cambridge University Press, 2012, vol. 138. [6] A. Ehrenfeucht, “An application of games to the completeness problem for formalized theories,” Fundamenta mathematicae, vol. 49, no. 129- 141, p. 13, 1960. [7] R. Fraïssé, “Sur quelques classifications des relations, basées sur des isomorphismes restreints,” Publications Scientifiques de l’Université d’Alger. Série A (mathématiques), vol. 2, pp. 273–295, 1955. [8] P. G. Kolaitis and M. Y. Vardi, “Infinitary logics and 0–1 laws,” Information and computation, vol. 98, no. 2, pp. 258–294, 1992. [9] M. Hennessy and R. Milner, “On observing nondeterminism and concurrency,” in International Colloquium on Automata, Languages, and Programming. Springer, 1980, pp. 299–309. [10] S. Abramsky, A. Dawar, and P. Wang, “The pebbling comonad in finite model theory,” in 32nd Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2017, Reykjavik, Iceland, June 20-23, 2017. IEEE Computer Society, 2017, pp. 1–12. [11] S. Abramsky and N. Shah, “Relating structure and power: Comonadic semantics for computational resources,” Journal of Logic and Computation, vol. 31, no. 6, pp. 1390–1428, 2021. [12] E. Moggi, “Computational lambda-calculus and monads,” in Proceedings of the Fourth Annual Symposium on Logic in Computer Science (LICS ’89), Pacific Grove, California, USA, June 5-8, 1989. IEEE Computer Society, 1989, pp. 14–23. [13] ——, “Notions of computation and monads,” Information and Computation, vol. 93, no. 1, pp. 55–92, 1991. [14] E. G. Manes, Algebraic theories. Springer Science, 1976, vol. 26. [15] S. Abramsky and D. Marsden, “Comonadic semantics for guarded fragments,” in 36th Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2021, Rome, Italy, June 29 - July 2, 2021. IEEE, 2021, pp. 1–13. [16] A. Ó. Conghaile and A. Dawar, “Game comonads & generalised quantifiers,” in 29th EACSL Annual Conference on Computer Science Logic, CSL 2021, January 25-28, 2021, Ljubljana, Slovenia (Virtual Conference), ser. LIPIcs, C. Baier and J. Goubault-Larrecq, Eds., vol. 183. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021, pp. 16:1–16:17. [17] Y. Montacute and N. Shah, “The pebble-relation comonad in finite model theory,” in LICS ’22: 37th Annual ACM/IEEE Symposium on Logic in Computer Science, Haifa, Israel, August 2 - 5, 2022, C. Baier and D. Fisman, Eds. ACM, 2022, pp. 13:1–13:11. [Online]. Available: https://doi.org/10.1145/3531130.3533335 [18] T. Paine, “A pebbling comonad for finite rank and variable logic, and an application to the equirank-variable homomorphism preservation theorem,” in Proceedings of the 36th Conference on the Mathematical Foundations of Programming Semantics, MFPS 2020, Online, October 1, 2020, ser. Electronic Notes in Theoretical Computer Science, P. Johann, Ed., vol. 352. Elsevier, 2020, pp. 191–209. [19] A. Dawar, T. Jakl, and L. Reggio, “Lovász-type theorems and game comonads,” in 36th Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2021, Rome, Italy, June 29 - July 2, 2021. IEEE, 2021, pp. 1–13. [20] S. Abramsky and D. Marsden, “Comonadic semantics for hybrid logic,” in 47th International Symposium on Mathematical Foundations of Computer Science, MFCS 2022, August 22-26, 2022, Vienna, Austria, ser. LIPIcs, S. Szeider, R. Ganian, and A. Silva, Eds., vol. 241. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022, pp. 7:1–7:14. [21] L. Reggio, “Polyadic sets and homomorphism counting,” Advances in Mathematics, vol. 410, p. 108712, 2022. [22] S. Abramsky, “Structure and power: an emerging landscape,” Fundam. Informaticae, vol. 186, no. 1-4, pp. 1–26, 2022. [23] S. Abramsky and L. Reggio, “Arboreal categories and resources,” in 48th International Colloquium on Automata, Languages, and Programming, ICALP 2021, July 12-16, 2021, Glasgow, Scotland (Virtual Conference), ser. LIPIcs, N. Bansal, E. Merelli, and J. Worrell, Eds., vol. 198. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021, pp. 115:1–115:20. [24] H. Ebbinghaus and J. Flum, Finite model theory, ser. Perspectives in Mathematical Logic. Springer, 1995. [25] E. G. Manes and P. Mulry, “Monad compositions I: general constructions and recursive distributive laws,” Theory and Applications of Categories, vol. 18, no. 7, pp. 172–208, 2007. [26] A. Kock, “Bilinearity and cartesian closed monads,” Mathematica Scandinavica, vol. 29, no. 2, pp. 161–174, 1971. [27] T. Jakl, D. Marsden, and N. Shah, “Generalizations of bilinear maps,” 2022, technical report, arXiv preprint. [Online]. Available: https://arxiv.org/abs/2205.05382 [28] A. Dawar, S. Severini, and O. Zapata, “Pebble games and cospectral graphs,” Electronic Notes in Discrete Mathematics, vol. 61, pp. 323– 329, 2017. [29] S. Abramsky and N. Tzevelekos, “Introduction to categories and categorical logic,” in New structures for physics. Springer, 2010, pp. 3–94. [30] S. Awodey, Category theory. Oxford University Press, 2010. [31] C. Stirling, Modal and temporal properties of processes. Springer Science & Business Media, 2001. [32] A. W. Roscoe, Understanding concurrent systems. Springer Science & Business Media, 2010. [33] H. Kleisli, “Every standard construction is induced by a pair of adjoint functors,” Proceedings of the American Mathematical Society, vol. 16, no. 3, pp. 544–546, 1965. [34] B. Jacobs, “Semantics of weakening and contraction,” Annals of Pure and Applied Logic, vol. 69, no. 1, pp. 73–106, 1994. [35] S. Abramsky, T. Jakl, and T. Paine, “Discrete density comonads and graph parameters,” in Coalgebraic Methods in Computer Science - 16th IFIP WG 1.3 International Workshop, CMCS 2022, Colocated with ETAPS 2022, Munich, Germany, April 2-3, 2022, Proceedings, ser. Lecture Notes in Computer Science, H. H. Hansen and F. Zanasi, Eds., vol. 13225. Springer, 2022, pp. 23–44. [Online]. Available: https://doi.org/10.1007/978-3-031-10736-8_2 [36] P. T. Johnstone, “Adjoint lifting theorems for categories of algebras,” Bulletin of the London Mathematical Society, vol. 7, no. 3, pp. 294– 297, 1975. [37] A. Kock, “Closed categories generated by commutative monads,” Journal of the Australian Mathematical Society, vol. 12, no. 4, pp. 405–424, 1971. [38] G. Seal, “Tensors, monads and actions,” Theory and Applications of Categories, vol. 28, no. 15, pp. 403–434, 2013. [39] F. E. Linton, “Coequalizers in categories of algebras,” in Seminar on triples and categorical homology theory. Springer, 1969, pp. 75–90. [40] R. Street, “The petit topos of globular sets,” Journal of Pure and Applied Algebra, vol. 154, no. 1-3, pp. 299–315, 2000. [41] M. Weber, “Generic morphisms, parametric representations and weakly cartesian monads,” Theory and Applications of Categories, vol. 13, no. 14, pp. 191–234, 2004. [42] ——, “Familial 2-functors and parametric right adjoints,” Theory and Applications of Categories, vol. 18, no. 22, pp. 665–732, 2007. [43] C. Berger, P.-A. Mellies, and M. Weber, “Monads with arities and their associated theories,” Journal of Pure and Applied Algebra, vol. 216, no. 8-9, pp. 2029–2048, 2012. [44] T. Altenkirch, J. Chapman, and T. Uustalu, “Monads need not be endofunctors,” in International Conference on Foundations of Software Science and Computational Structures. Springer, 2010, pp. 297–311. [45] P. J. Freyd and G. M. Kelly, “Categories of continuous functors, I,” Journal of Pure and Applied Algebra, vol. 2, no. 3, pp. 169–191, 1972. [46] B. Bednarczyk and M. Urbanczyk, “Comonadic semantics for description logics games,” in Proceedings of the 35th International Workshop on Description Logics (DL 2022) co-located with Federated Logic Conference (FLoC 2022), Haifa, Israel, August 7th to 10th, 2022, ser. CEUR Workshop Proceedings, O. Arieli, M. Homola, J. C. Jung, and M. Mugnier, Eds., vol. 3263. CEUR-WS.org, 2022. [Online]. Available: http://ceur-ws.org/Vol-3263/paper-5.pdf [47] M. Bojanczyk, “Separator logic and star-free expressions ´ for graphs,” 2021, arXiv preprint. [Online]. Available: https://arxiv.org/abs/2107.13953 [48] N. Schirrmacher, S. Siebertz, and A. Vigny, “First-Order Logic with Connectivity Operators,” in 30th EACSL Annual Conference on Computer Science Logic (CSL 2022), ser. Leibniz International Proceedings in Informatics (LIPIcs), F. Manea and A. Simpson, Eds., vol. 216. Dagstuhl, Germany: Schloss Dagstuhl – LeibnizZentrum für Informatik, 2022, pp. 34:1–34:17. [Online]. Available: https://drops.dagstuhl.de/opus/volltexte/2022/15754 [49] Y. Gurevich, “Monadic second-order theories,” in Model-theoretic logics, J. Barwise and S. Feferman, Eds. Cambridge University Press, 1985, pp. 479–506. [50] D. Turi and G. Plotkin, “Towards a mathematical operational semantics,” in Proceedings of Twelfth Annual IEEE Symposium on Logic in Computer Science. IEEE, 1997, pp. 280–291. [51] G.D.Plotkin, “A structural approach to operational semantics,” Aarhus University, Tech. Rep. DAIMI FN-19, 1981. [52] J. Adámek, J. Adamek, J. Rosicky et al., Locally presentable and accessible categories. Cambridge University Press, 1994, vol. 189.

A Categorical Account of
Composition Methods in Logic: Conclusions, Acknowledgments & References

Ever wondered how a data warehouse transforms from a concept to a cornerstone of business intelligence? 🌟 Imagine a data warehouse as a vast library, each book brimming with stories (data) waiting to be told. But how does this library come to be? It's not just about stacking books on shelves; it's an art, a science, and a journey. And who's our librarian in this tale? The ! They're the masterminds weaving together technology, strategy, and data to create a repository that not only stores information but makes it meaningful. Data Architect So, grab your hard hats and blueprint rolls! We're about to embark on the fascinating journey of building a data warehouse, guided by our adept Data Architect. Understanding the Need for a Data Warehouse A data warehouse, optimized for , is a centralized repository for structured and processed data. OLAP (Online Analytical Processing) Unlike traditional systems, it's designed for efficient querying and reporting. The use of in data warehouses allows for quicker data retrieval, especially beneficial for analytical queries. OLTP (Online Transaction Processing) columnar storage This analytical powerhouse differs from , which store raw, unstructured data, and approaches that handle large data volumes but may not optimize specific analytical queries. data lakes big data In essence, a data warehouse is not just a storage unit but a sophisticated tool for advanced analytics and reporting aiding in generating business intelligence. Dreaming and Designing Picture an architect dreaming up a grand building. That's our Data Architect, envisioning the data warehouse. It's not just a wild dream; it's a meticulously planned one, understanding the business needs, data sources, and end goals. Here, the architect might consider using a , an approach to database design that allows for intricate and multifaceted data relationships. This schema, resembling a snowflake's structure, is known for its normalization, which reduces data redundancy and improves efficiency. snowflake schema Laying the Foundations Every great building needs a solid foundation, and so does a data warehouse. This phase involves choosing the right infrastructure, which can vary significantly depending on specific needs. There are various data warehouse products available in the market, each with its unique features. On-premise data warehouses have been the traditional choice, offering control and customization. However, the rise of cloud data warehousing has brought about a shift, favoring flexibility and scalability. Cloud data warehouses like , , and offer various degrees of these services, allowing businesses to select the one that aligns best with their objectives. Amazon Redshift Google BigQuery Snowflake Most cloud providers lay out the fundamental architecture, making it easier for businesses to implement and scale their data warehouses efficiently. Our Data Architect collaborates with IT teams and , selecting the right tools and technologies. Database Engineers They're like the builders and engineers, ensuring our library is robust and weatherproof. Structuring the Warehouse Structuring the warehouse's structure is a pivotal phase. Here, the and determine the most efficient and accessible way to organize data. Often, they employ the snowflake schema, an advanced data model known for its detailed normalization. Data Architect Data Modelers In the snowflake schema, data is segmented into fact tables and dimension tables. Let’s take an example of a retail giant called 'Y', the structure could include: These tables contain quantifiable data about business events. In a retail context, this might include a 'Sales' table, capturing every sales transaction. Key metrics such as sales amount, quantity sold, and transaction timestamps would be stored here. Fact Tables: These tables provide the context for facts. For our retail company 'Y', dimension tables could include: Dimension Tables: Customer: Stores customer data like name, address, and contact information. Product: Details about products, such as product ID, name, category, and price. Store: Information on store locations, including store ID, address, and region. In the snowflake schema, dimension tables are normalized, meaning related data is further broken down into additional tables. For instance, the 'Product' might link to a 'Product Category,' providing more detailed categorization. This design allows for efficient data processing and complex querying. The separation of facts and dimensions, coupled with the detailed breakdown in the snowflake schema, supports a wide range of analytical capabilities from basic sales tracking to in-depth customer behavior analysis. Populating With Data Populating a data warehouse involves several critical steps to ensure that the data is not only accurately captured but also effectively usable for analysis. Define the Data Sources The first step involves identifying and defining all the data sources that will feed into the warehouse. This includes a variety of sources, such as internal systems like CRM and ERP, external data from market research, social media, and more. Understanding the nature, format, and quality of data from these sources is crucial for effective integration. Design the Data Model Once the data sources are identified, the next step is designing the data model. This step serves as the blueprint for how data is stored, accessed, and related within the warehouse. Configure the ETL or ELT Jobs With the data sources defined and the data model designed, the focus shifts to configuring the (Extract, Transform, Load) or (Extract, Load, Transform) jobs. ETL ELT Monitor the Data Quality The final step is the continuous monitoring of data quality after it is populated into the warehouse. This step ensures data accuracy, consistency, and integrity. Regular audits, data validation processes, and error-checking mechanisms are integral to maintaining the high quality of data in the warehouse. Various tools can assist in these steps. For ETL and ELT, popular tools like and ) can be employed. Additionally, data quality monitoring tools and software can help in maintaining the integrity of the data warehouse. Informatica PowerCenter, Talend, Apache NiFi, Microsoft SQL Server Integration Services (SSIS Testing the Integrity Before opening the doors, we need to ensure everything is in order. This is where rigorous testing comes in. Data Quality Analysts and QA Engineers scrutinize every corner of the warehouse, checking for data accuracy, performance issues, and security vulnerabilities. It's like a pre-opening inspection of our library, making sure every book is in place and the cataloging system works flawlessly. They'll sift through each test case, marking them as a pass or a fail. Any hiccups? Back to the kitchen (the development team) for tweaks. Once the final nod of approval comes from the stakeholders, it's a go-go for the final development. This testing phase is critical to ensure that the data warehouse not only meets the functional requirements but also adheres to the highest standards of data quality and security. Launching the Warehouse With a green light from testing, it's time for the grand opening. The data warehouse goes live, ready to serve its purpose. Key to this phase is establishing secure and appropriate access for different users: for general users who need to view and analyze data. Read-Only Access: for specialized roles that require modification privileges. Read-Write Access: for Data Administrators managing the warehouse. Administrative Access: Access is provisioned through Role-Based Access Control (RBAC), ensuring users have permissions tailored to their role. Strong authentication methods and regular audits of access privileges are crucial for maintaining security and efficiency as the warehouse goes live. Continuous Improvement and Maintenance A great library evolves with time, and so does a data warehouse. The Data Architect, along with the support team, monitors the warehouse, ensuring it adapts to changing business needs, data volumes, and technological advancements. They're continually optimizing, updating, and maintaining the warehouse, much like a librarian acquiring new books and preserving old ones. Wrapping It Up And there you have it, the lifecycle of a data warehouse, from a mere concept to a fully functional and evolving repository of knowledge. It's a testament to the skill, collaboration, and foresight of the Data Architect and their team, ensuring that the data warehouse not only stores data but turns it into a treasure trove of insights. So, next time you query a data warehouse, remember, it's not just a technology; it's a carefully crafted masterpiece, a library of data, built to enlighten and empower. 🚀✨

The Lifecycle of a Data Warehouse

Introduction: Large enterprises, which have more than one business unit, usually have more than one data platform environment. In fact, companies trying to be agile to make quick decisions, all have business unit-specific data platforms. With the computational advancements and rapid growth in data analytics and science, data integration tasks for data platforms are increasing exponentially as they need to support transactional as well as batch processing jobs. Here, the challenge is choosing the orchestration tool, as data integrations have to work in conjunction with different platforms. Though there are several tools in the market, in the article, we will cover how Apache airflow became the most trusted tool for the orchestration industry and its advantages. ETL: Traditional ETL (Extract, Transform, and loading) tools like Informatica, Ab Initio, and Talend all come with their own orchestration process. But as they are proprietary tools, it's expensive as well. Cloud service providers Azure, AWS (Amazon Web Services), and Google Cloud are all trying cash in the orchestration space so Azure has Synapse and AWS has Glue, Step function, etc. Basics: Orchestration is a simple or complex series of workflows defining a process. Workflows depend on one another to perform a business function or a series of business func‐tions calling other services. Then what is the ? process A process is a business function or set of business functions achieved by executing one or more tasks. Then what is the ? task A task is a step in a process. A task may call a service to perform a particular data-related function. Then what is the ? Service A service is software like Azure Synapse Analytics, Python data frame, or AWS DynamoDB that performs tasks. Common Problems in the Legacy Tools: The most common problem in the orchestration tools is their ability to successfully implement and maintain processes across the organization. Engineers and managers want to choose a tool that addresses the below issues: Dependency management: Lack of dependency management results in maintaining process-to-process and task-to-tasks dependency checks within the codebase along with business logic. This increases development time based on the number of processes or tasks involved in a workflow. Task modification to existing processes: Workflow is defined as each step within the code. Embedding the workflow in the code requires manual effort to fully review the code to make changes to one or more tasks resulting in additional time involved in project enhancements and testing. Troubleshooting: Troubleshooting a particular process should not require the need to understand the dependencies, as it results in additional support time. Restart: It should be fairly easy to restart the failed task or trigger an ad-hoc task. Orchestration should not warrant the need to rerun the entire process if a particular task within the process failed. User-friendly interface to track the progress: Orchestration tools should provide best in class user interface with the details to visualize the tasks involved within the processes in one place. Engineers investigating the issue should have a UI to understand the flow of a particular process. UI should display the current process in execution, the time it's taking, etc. Apache Airflow Airflow is a platform to programmatically author, schedule, and monitor workflows geared at data pipelining. Airflow is used to author workflows as DAGs of tasks. The Airflow scheduler executes tasks on an array of workers while following the specified dependencies. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. Code Development: Authoring a data pipeline using existing tasks involves writing a Python module and deploying it to the Airflow installation. Adding a new kind of task to an airflow installation involves writing and deploying Python code in packages called plugins. This aligns with the primary coding skill set of Data Engineers and scientists, which is Python. Benefits At a high level, process dependency management is easy to develop and maintain via user interface (UI) and support in Airflow, which helps immensely to save time for developers and supporting resources. DAG creation is all in Python scripting language which is the primary coding skillset for Data Engineers. Changes within Airflow can be captured using the GIT repository enabling version controls and change management practices. Airflow has good documentation and an active internal community for answering questions. What are the skillsets required to develop and maintain Airflow? Python is a major skill required to use Airflow. Initial learnings are required to set up Airflow and understand the architecture so it's evident for future maintenance. Can Engineers visualize the Jobs? Yes, Airflow has a Web interface to start and stop processes and also to monitor the current status of the processes. Does Airflow support any integration with Data Quality checkers? DataDog can be integrated with Airflow. How do we make ensure Airflow’s availability? Airflow workers and Web servers can be load balanced. Airflow Scheduler is a single point of failure. How does Airflow scale? Airflow Web server can be scaled based on the number of requests. Airflow workers can be scaled up and down based on requirements.

Apache Airflow And Its Contribution to Enterprise Data Integration

When talking about data processing, people often abbreviate it as “ETL.” However, if we look closely, data processing has undergone several iterations from ETL, ELT, XX ETL (such as Reverse ETL, Zero-ETL) to the currently popular EtLT architecture. While the Hadoop era mainly relied on ELT (Extract, Load, Transform) methods, the rise of real-time data warehouses and data lakes has rendered ELT obsolete. EtLT has emerged as the standard architecture for real-time data loading into data lakes and real-time data warehouses. Let’s explore the reasons behind the emergence of these architectures, their strengths and weaknesses, and why EtLT is gradually replacing ETL and ELT as the global mainstream data processing architecture, along with practical open-source methods. ETL Era (1990–2015) In the early days of data warehousing, Bill Inmmon, the proponent of data warehousing, defined it as a data storage architecture for partitioned subjects, where data was categorized and cleaned during storage. During this period, most data sources were structured databases (e.g., MySQL, Oracle, SQLServer, ERP, CRM), and data warehouses predominantly relied on OLTP databases (e.g., DB2, Oracle) for querying and historical storage. Handling complex ETL processes with such databases proved to be challenging. To address this, a plethora of ETL software emerged, such as Informatica, Talend, and Kettle, which greatly facilitated integrating complex data sources and offloading data warehouse workloads. Advantages: Clear technical architecture, smooth integration of complex data sources, and about 50% of the data warehouse work handled by ETL software. Disadvantages: All processing is done by data engineers, leading to longer fulfillment of business requirements; high hardware costs with double investments, especially when dealing with large data volumes. During the early and mid stages of data warehousing, when data source complexity was significantly higher, the ETL architecture became the industry standard and remained popular for over two decades. ELT Era (2005–2020) As data volumes grew, both data warehouse and ETL hardware costs escalated. New MPP (Massively Parallel Processing) and distributed technologies emerged, leading to the gradual shift from ETL to ELT architecture in the later stages of data warehousing and the rise of big data. Teradata, one of the major data warehouse vendors, and Hadoop Hive, a popular Hadoop-based data warehousing solution, adopted ELT architecture. They focused on direct loading (without complex transformation like join and group) of data into the data warehouse’s data staging layer, followed by further processing using SQL or H-SQL from the staging layer to the data atomic layer and finally to the summary layer and indicator layer. While Teradata targeted structured data and Hadoop targeted unstructured data, they adopted similar 3–4 layer data storage architectures and methodologies for data warehousing globally. Advantages: Utilizing data warehouse high-performance computing for large data volume processing, resulting in higher hardware ROI. Complex business logic can be handled by SQL, reducing the overall cost of data processing personnel by employing data analysts and business-savvy technical personnel, without the need to understand ETL tools like Spark and MapReduce. Disadvantages: Suitable for simple and large data volume scenarios. Inadequate for complex data sources and lacking support for real-time data warehousing requirements. ODS (Operational Data Store) was introduced as a transitional solution to handle complex data sources that ELT-based data warehouses couldn’t load and to improve real-time capabilities. ODS involved processing complex data sources through real-time CDC (Change Data Capture), real-time APIs, or short-batch processing (Micro-Batch) into a separate storage layer before ELT-ing them into the enterprise data warehouse. Currently, many enterprises still adopt this approach. Some companies place the Operational Data Store (ODS) within the data warehouse and use Spark and MapReduce for initial ETL (Extract, Transform, Load) processes. Later, they perform business data processing within the data warehouse (using tools like Hive, Teradata, Oracle, and DB2). At this stage, the early EtLT (Extract, Transform, Load, and Transform) community has already formed. It is characterized by a division of roles, where the complex processes of data extraction, Change Data Capture (CDC), data structuring, and standardization are often handled by data engineers, referred to as “t.” Their objective is to move data from the source system to the underlying data preparation layer or data atomic layer within the data warehouse. On the other hand, the processing of complex data atomic layers with business attributes, data aggregation, and generating data metrics (involving operations such as Group by and Join) is typically performed by business data engineers or data analysts who are skilled in using SQL. As a result of the emergence of the EtLT architecture, standalone projects like ODS have gradually faded out of the limelight due to the increase in data volume and the adoption of EtLT principles. EtLT Era (2020-Present) The EtLT architecture, as summarized by James Densmore in the , is a modern and globally popular data processing framework. EtLT emerged in response to the transformations in the modern data infrastructure. Data Pipelines Pocket Reference 2021 Background of the EtLT Architecture The modern data infrastructure has the following characteristics, which led to the emergence of the EtLT architecture: Cloud, SaaS, and Hybrid Local Data Sources Data Lake and Real-time Data Warehouses New Generation Big Data Federation Proliferation of AI Applications Fragmentation of Enterprise Data Community The appearance of complex data sources In the current global enterprise landscape, the advent of cloud and SaaS has made an already complex data source environment even more intricate. Dealing with SaaS data has given rise to a new concept of data ingestion, exemplified by tools like Fivetran and Airbyte, aiming to address the ELT (Extract, Load, Transform) challenges of ingesting SaaS data into data warehouses like Snowflake. Additionally, the complexity of data sources has increased with the proliferation of cloud-based data storage services (e.g., AWS Aurora, AWS RDS, MongoDB Service) and the coexistence of traditional on-premises databases and software (SAP, Oracle, DB2) in hybrid cloud architectures. Traditional ETL and ELT architectures are unable to cope with the intricacies of processing data in such a complex environment. Data Lake and Real-time Data Warehouses In modern data architecture, the emergence of data lakes has combined the features of traditional ODS (Operational Data Store) and data warehouses. Data lakes enable real-time data processing and facilitate data changes at the source (e.g., Apache Hudi, Apache Iceberg, Databricks Delta Lake). Simultaneously, the concept of real-time data warehouses has surfaced, with various new computing engines (e.g., Apache Pinot, ClickHouse, Apache Doris) making real-time ETL a priority. However, traditional CDC ETL tools or real-time stream processing platforms face challenges in providing adequate support for data lakes and real-time data warehouses, either due to compatibility issues with new storage engines or limitations in connecting to modern data sources, lacking robust architecture and tool support. The Emergence of a New Generation of Big Data Federation In modern data architecture, a new breed of architectures has emerged, aiming to minimize data movement across different data stores and enabling complex queries directly through connectors or rapid data loading. Examples include Starburst’s TrinoDB (formerly PrestoDB) and OneHouse based on Apache Hudi. These tools excel at data caching and on-the-fly cross-data-source queries, making them unsuitable for support by traditional ETL/ELT tools in this new Big Data Federation paradigm. The Rise of Large-scale Models With the emergence of ChatGPT in 2022, AI models have become algorithmically feasible for widespread enterprise applications. The bottleneck for AI application deployment now lies in data supply, which has been addressed by data lakes and Big Data Federation for data storage and querying. However, traditional ETL, ELT, and stream processing have become bottlenecks for data supply, either unable to quickly integrate various complex traditional and emerging data sources or failing to support diverse data requirements for both AI training and online AI applications using a single codebase. Fragmentation of the Enterprise Data Community As data-driven approaches become more ingrained in enterprises, the number of data users within organizations has rapidly increased. These users range from traditional data engineers to data analysts, AI practitioners, sales analysts, and financial analysts, each with diverse data requirements. After experiencing various shifts like No-SQL and New-SQL, SQL has emerged as the sole standard for complex business analysis. A considerable number of analysts and business unit engineers now use SQL to address the “last mile” problem of data analysis within enterprises. Meanwhile, the handling of complex unstructured data is left to professional data engineers using technologies like Spark, MapReduce, and Flink. Consequently, the demands of these two groups diverge significantly, making traditional ETL and ELT architectures inadequate to meet the needs of modern enterprise users. EtLT Architecture Emerges! Against the backdrop mentioned above, data processing has gradually evolved into the EtLT architecture: EtLT Architecture Overview: EtLT splits the traditional ETL and ELT structures and combines real-time and batch processing to accommodate real-time data warehouses and AI application requirements. E(xtract) — Extraction: EtLT supports traditional on-premises databases, files, and software as well as emerging cloud databases, SaaS software APIs, and serverless data sources. It can perform real-time CDC on database binlog and real-time stream processing (e.g., Kafka Streaming) and also handle bulk data reading (multi-threaded partition reading, rate limiting, etc.). t(ransform) — Normalization: In addition to ETL and ELT, EtLT introduces a small “t,” focusing on data normalization. It rapidly converts complex and heterogeneous extracted data sources into structured data that can be readily loaded into the target data storage. It deals with real-time CDC by splitting, filtering, and changing field formats, supporting both batch and real-time distribution to the final Load stage. L(oad) — Loading: The loading stage is no longer just about data loading but also involves adapting data source structures and content to suit the target data destination (Data Target). It should handle data structure changes (Schema Evolution) in the source and support efficient loading methods such as Bulk Load, SaaS loading (Reverse ETL), and JDBC loading. EtLT ensures support for real-time data and data structure changes, along with fast batch data loading. (T)ransform — Conversion: In cloud data warehouses, on-premises data warehouses, or new data federations, business logic is processed. This is typically achieved using SQL, either in real-time or batch mode, to transform complex business rules accurately and quickly into data usable by business or AI applications. In the EtLT architecture, different user roles have distinct responsibilities: EtL Phase: Primarily handled by data engineers who convert complex and heterogeneous data sources into data that can be loaded into the data warehouse or data federation. They do not need to have an in-depth understanding of enterprise metric calculation rules but must be proficient in transforming various source and unstructured data into structured data. Their focus is on data timeliness and the accuracy of transforming source data into structured data. T Phase: Led by data analysts, business SQL developers, and AI engineers who possess a deep understanding of enterprise business rules. They convert business rules into SQL statements to perform analysis and statistics on the underlying structured data, ultimately achieving data analysis within the enterprise and enabling AI applications. Their focus is on data logic relationships, data quality, and meeting business requirements for final data results. Open-Source Implementation of EtLT There are several open-source implementations of EtLT in modern data architecture. Examples include DBT, which helps analysts and business developers quickly develop data applications based on Snowflake, and Apache DolphinScheduler, a visual workflow orchestration tool for big data tasks. DolphinScheduler plans to introduce a Task IDE, allowing data analysts to directly debug SQL tasks for Hudi, Hive, Presto, ClickHouse, and more, and create workflow tasks through drag-and-drop. As a representative of the EtLT architecture, Apache SeaTunnel started with support for various cloud and on-premises data sources, gradually expanding its capabilities to include SaaS and Reverse ETL, as well as accommodating the demands of large-scale model data supply. It has been continually refining the EtLT landscape. The latest SeaTunnel Zeta computing engine delegates complex operations such as Join and Groupby to the ultimate data warehouse endpoint, focusing on data normalization and standardization. This approach aims to achieve the goal of unified real-time and batch data processing with a single set of code and a high-performance engine. Additionally, SeaTunnel now includes support for large-scale models, making it possible for these models to directly interact with over 100 supported data sources (refer to ), ranging from traditional databases to cloud databases and ultimately SaaS. https://seatunnel.apache.org/docs/Connector-v2-release-state/ Since joining the Apache Incubator in late 2022, Apache SeaTunnel has witnessed a five-fold growth in one year, and currently, it supports more than 100 data sources (as seen at ). The connector support has been progressively improved, encompassing traditional databases, cloud databases, and SaaS offerings. https://seatunnel.apache.org/docs/2.3.2/category/source-v2 The release of SeaTunnel Zeta Engine in Apache SeaTunnel 2.3.0 brings features such as data distributed CDC, schema evolution for target source data tables, and the synchronization of entire databases and multiple tables. Its excellent performance has garnered attention from numerous global users, including Bharti Airtel, the second-largest telecommunications operator in India, Shopee.com, an e-commerce platform in Singapore, and Vip.com, a major online retailer. Large Scale Model Training support One noteworthy aspect is that SeaTunnel now offers support for large-scale model training and vector databases, enabling seamless interactions between large models and the 100+ supported data sources of SeaTunnel (see the article ). Furthermore, SeaTunnel can leverage ChatGPT to directly generate SaaS Connectors, facilitating rapid access to a wide range of internet information for your large-scale models and data warehouses. Breakthrough in the book search field! Use Apache SeaTunnel, Milvus, and OpenAI to improve the accuracy and efficiency of book title similarity search As the complexity of AI, cloud, and SaaS continues to increase, the demand for real-time CDC, SaaS, data lakes, and real-time data warehouse loading has made simple ETL architectures inadequate to meet the needs of modern enterprises. EtLT architecture, tailored for different stages of enterprise development, is destined to shine in the modern data infrastructure. With the mission of “Connecting the world’s data sources and synchronizing them as swiftly as flying”, Apache SeaTunnel warrants the attention of all data professionals. Also published . here

ELT is Dead, and EtLT Will End Modern Data Processing Architecture

Fines related to sensitive data exposure are growing. For instance, major GDPR violations can cost companies up to , while gross HIPAA violations can result in imprisonment. 4% of their annual global turnover Your production environment might be thoroughly protected. But what is about testing initiatives and sales demos? Are you confident in the third-party contractors that have access to your sensitive data? Will they do their best to protect it? To ensure compliance and data safety, companies are turning to . If you are also interested, check out this guide answering the three important questions: data management service providers What is data masking? Why and when do you need it, and How could your company implement it successfully? It also presents a detailed data masking example from our portfolio. After reading the article, you will have enough information to negotiate with data masking vendors. Understanding data masking So, what is data masking? Data masking is defined as building a realistic and structurally similar but nonetheless fake version of organizational data. It alters the original data values using manipulation techniques while maintaining the same format and delivers a new version that can’t be reverse-engineered or tracked back to the authentic values. Here is an example of masked data: Do you need to apply data masking algorithms to all the data stored within your company? Most likely not. Here are the data types that you definitely need to protect: includes medical records, lab tests, medical insurance information, and even demographics. Protected Health Information (PHI) is related to credit and debit card information and transaction data under the Payment Card Industry Data Security Standard (PCI DSS). Payment card information , such as passport and social security numbers. Basically, any piece of information that can be used to identify a person. Personally identifiable information (PII) includes inventions, such as designs, or anything that has value to the organization and can be stolen. Intellectual property (IP) Why do you need data masking? Data masking protects sensitive information utilized for non-productive purposes. So, as long as you use any of the sensitive data types presented in the previous section in training, testing, sales demos, or any other types of non-production activities, you need to apply data masking techniques. This makes sense as non-production environments are normally less protected and introduce . more security vulnerabilities Moreover, if there is a need to share your data with third-party vendors and partners, you can grant access to masked data instead of forcing the other party to comply with your extensive security measures to access the original database. Statistics show that take place due to compromises on the business partner’s side. 19% of data breaches Additionally, data masking can provide the following advantages: Renders organizational data useless to cybercriminals in case they are able to access it Reduces risks posed by sharing data with authorized users and outsourcing projects Helps comply with data privacy and security-related regulations, such as the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), and any other regulations applicable within your field Protects data in the case of deletion, as the conventional file deletion methods still leave a trace of the old data values Safeguards your data in the case of unauthorized data transfer Data masking types There are five main types of data masking that aim to cover different organizational needs. 1. Static data masking Implies creating a backup of the original data and keeping it safe in a separate environment for production use cases. Then it disguises the copy by including fake but realistic values, and makes it available for non-production purposes (e.g., testing, research), as well as sharing with contractors. 2. Dynamic data masking Aims to modify an excerpt of the original data at runtime when receiving a query to the database. So, a user who is not authorized to view sensitive information queries the production database, and the response is masked on the fly without changing the original values. You can implement it via database proxy, as presented below. This data masking type is normally used in read-only settings to prevent overriding production data. 3. On-the-fly data masking This data masking type disguises data when transferring it from one environment to another, such as from production to testing. It is popular with organizations that continuously deploy software and perform large data integrations. 4. Deterministic data masking Replaces column data with the same fixed value. For instance, if you want to replace “Olivia” with “Emma”, you have to do it in all the associated tables, not only in the table you are currently masking. 5. Statistical data obfuscation This is used to reveal information about patterns and trends in a dataset without sharing any details on actual people represented there. 7 main data masking techniques Below you can find seven of the most popular data masking techniques. You can combine them to cover the various needs of your business. . You can shuffle and reassign data values within the same table. For example, if you shuffle the employee name column, you will get the real personal details of one employee matched to another. Shuffling . Rearranges characters and integers of a data field in random order. If an employee’s original ID is 97489376, after applying shuffling, you will receive something like 37798649. This is restricted to specific data types. Scrambling . This is a simple masking strategy where a data field is assigned a null value. This method has limited usage as it tends to fail the application’s logic. Nulling out . Original data is substituted by fake but realistic values. Meaning that the new value still needs to satisfy all domain constraints. For instance, you substitute someone’s credit card number with another number that conforms to the rules enforced by the issuing bank. Substitution . This is mostly applicable to financial information. One example is masking original salaries by applying +/-20% variance. Number variance . This method increases or decreases a date by a specific range, maintaining that the resulting date satisfies the application’s constraints. For instance, you can age all contracts by 50 days. Date aging . Involves replacing all the original data values by an average. For instance, you can replace every individual salary field by an average of salary values in this table. Averaging How to implement data masking the right way? Here is your 5-step data masking implementation plan. Step 1: Determine the scope of your project Before you start, you will need to identify which aspects you will cover. Here is a list of typical questions that your data team can study before proceeding with the masking initiatives: Which data are we looking to mask? Where does it reside? Who is authorized to access it? What is the access level of each user from the above? Who can only view and who can alter and delete values? Which applications are utilizing this sensitive data? What impact will data masking have on different users? What level of masking is required, and how often will we need to repeat the process? Are we looking to apply data masking across the whole organization or limit it to a specific product? Step 2: Define the stack of data masking techniques During this step, you need to identify which technique or a combination of data masking tools are the best fit for the task at hand. First of all, you need to identify which data types you need to mask, for instance, names, dates, financial data, etc., as different types require dedicated data masking algorithms. Based on that, you and your vendor can choose which open-source library(s) can be reused to produce the best-suited data masking solution. We advise turning to a , as they will help you customize the solution and integrate it painlessly into your workflows across the whole company without interrupting any business processes. Also, it’s possible to build something from zero to cover the company’s unique needs. software vendor There are ready-made data masking tools that you can purchase and deploy yourself, such as , , , and many more. You can opt for this strategy if you manage all your data by yourself, you understand how different data flows work, and you have an IT department that can help integrate this new data masking solution into the existing processes without hindering productivity. Oracle Data Masking IRI FieldShield DATPROF Step 3: Secure your selected data masking algorithms The security of your sensitive data largely depends on the security of the selected fake data-generating algorithms. Therefore, only authorized personnel can know which data masking algorithms are deployed, as these people can reverse engineer the masked data to the original dataset with this knowledge. It’s a good practice to apply separation of duties. For instance, the security department selects the best-suited algorithms and tools, while data owners maintain the settings applied in masking their data. Step 4: Preserve referential integrity Referential integrity means that each data type within your organization is masked in the same way. This can be a challenge if your organization is rather large and has several business functions and product lines. In this case, your company is likely to use different data masking algorithms for various tasks. To overcome this issue, identify all the tables that contain referential constraints and determine in which order you will mask the data as parent tables should be masked before the corresponding child tables. After completing the masking process, do not forget to check whether referential integrity was maintained. Step 5: Make the masking process repeatable Any adjustment to a particular project, or just general changes within your organization, can result in modifying the sensitive data and creating new data sources, posing the need to repeat the masking process. There are instances where data masking can be a one-time effort, such as in the case of preparing a specialized training dataset that will be used for a few months for a small project. But if you want a solution that will serve you for a prolonged time, your data can become obsolete at one point. So, invest time and effort in formalizing the masking process to make it fast, repeatable, and as automated as possible. Develop a set of masking rules, such as which data has to be masked. Identify any exceptions or special cases that you can foresee at this point. Acquire/build scripts and to apply these masking rules in a consistent manner. automated tools Your checklist for selecting a data masking solution Whether you work with a software vendor of your choice or opt for a ready-made solution, the final product needs to follow these data-masking best practices: Be non-reversible, making it impossible to reverse engineer the fake data to its authentic values Protect the integrity of the original database and doesn’t render it useless by making permanent changes by mistake Mask non-sensitive data if this is necessary to protect sensitive information Provide an opportunity for automation, as the data will change at some point and you don’t want to start from zero every time Generate realistic data that maintains the structure and the distribution of the original data, and satisfies business constraints Be scalable to accommodate any additional data sources that you want to incorporate into your business Compliant with all the applicable regulations, such as HIPAA and GDPR, and your internal policies Integrate well into the existing systems and workflows Data masking challenges Here is a list of challenges that you might face during implementation. Format preservation. The masking solution has to understand the data and be able to preserve its original format. Gender preservation. The selected data masking methodology needs to be aware of gender when masking people’s names. Otherwise, the gender distribution within the dataset will be altered. Semantic integrity. The generated fake values need to follow the business rules restricting different data types. For instance, salaries have to fall within a specific range, and social security numbers have to follow a predetermined format. This is also true for maintaining the geographical distribution of the data. Data uniqueness. If the original data has to be unique, like an employee ID number, the data masking technique needs to supply a unique value. Balancing security and usability. If the data is too heavily masked, it can become useless. On the other hand, if it’s not protected enough, users can gain unauthorized access. Integrating the data into the existing workflows might be highly inconvenient to employees in the very beginning, as people are used to working in a certain way, which is currently being disrupted. A data masking example from the ITRex portfolio An international healthcare organization was looking to obscure sensitive personally identifiable information (PII) presented in multiple formats and residing in both production and non-production environments. They wanted to build an that can discover and obfuscate PII while complying with the company’s internal policies, GDPR, and other data privacy regulations. ML-powered data masking software Our team immediately noticed the following challenges: The client had enormous volumes of data, over 10,000 data sources, and many corresponding data flows There was no clear data masking strategy that would cover all the different departments Due to this large variety, our team wanted to come up with a set of policies and processes that would guide different dataset owners on how to mask their data and would serve as the basis for our solution. For instance, someone could come up with a list of data points that they want to obfuscate, whether once or continuously, and the solution, guided by these principles, would study the data and select appropriate obfuscation techniques and apply them. We approached this project by surveying the landscape through the following questions: Which data management solutions are you using? The client was already using Informatica, so we went with that. Informatica’s data masking solution offers out-of-the-box features, which satisfied some of the client’s needs, but this wasn’t enough to cover all the requirements. Which data types are you willing to mask? Due to a large number of data sources, it was impossible to address everything at once. So, we asked the client to prioritize and identify what was mission-critical. Do you want to do it one time, or make it a repeatable process? After answering these questions, we suggested providing data masking as a service mainly because the client has too many data sources, to begin with, and it might have taken years to cover them all. In the end, we delivered data masking services with the help of a custom ML-driven tool that can semi-automatically perform data masking in four steps: Identify data types. Data owners put their data sources into the analysis tool that studies the columns’ data and reveals the data types it could identify in these columns, such as addresses, phone numbers, etc. A human expert verifies its output, allowing it to learn from mistakes. Suggest masking approaches for each column and apply them after human approval Deploy the results. After the masked data is generated, it needs to be deployed. We provided multiple options for data storage. This includes, but is not limited to, using a temporary database that remains live for several days, assigning a permanent location for masked environments, generating comma-separated values (CSVs) files, and more. Examine and give a badge of approval to a set of data or a set of environments as proof that they are properly masked and compliant This data masking solution helped the client comply with GDPR, dramatically reduced the time needed to form non-production environments, and lowered the costs of transferring data from production to sandbox. How to maintain masked data after implementation? Your efforts do not stop when confidential data is masked. You still need to maintain it over time. Here are the steps that will help you in this initiative: Establish policies and procedures that govern the masked data. This includes determining who is authorized to access this data and under which circumstances and which purposes this data serves (e.g., testing, reporting, research, etc.) Train employees on how to use and protect this data Regularly audit and update the masking process to ensure it remains relevant Monitor masked data for any suspicious activities, such as unauthorized access attempts and breaches Perform masked data backups to make sure it’s recoverable Concluding thoughts Data masking will protect your data in non-production environments, enable you to share information with third-party contractors, and help you with compliance. You can purchase and deploy a data obfuscation solution yourself if you have an IT department and control your data flows. However, keep in mind that improper data masking implementation can lead to rather unpleasant consequences. Here are some of the most prominent ones: Hindering productivity. The selected data masking techniques can cause large unnecessary delays in data processing, thereby slowing employees. Becoming vulnerable to data breaches. If your data masking methods, or the lack thereof, fail to protect sensitive data, there will be financial and legal consequences up to serving time in prison. Deriving inaccurate results from data analysis. This can happen if the data is masked incorrectly or too heavily. Researchers will misinterpret the experimental dataset and reach faulty conclusions that will lead to unfortunate business decisions. Hence, if a company isn’t confident in its abilities to execute data obfuscation initiatives, it’s best to contact an external vendor who will help select the right data masking techniques and integrate the final product into your workflows with minimal interruptions. Stay protected! Considering to implement a data masking solution? ! We will help you prioritize your data, build a compliant obfuscation tool, and deploy it without interrupting your business processes. Get in touch

Data Masking: How it Can be Implemented Correctly

Enmascaramiento de datos: cómo se puede implementar correctamente

Masquage des données : comment l'implémenter correctement

डेटा मास्किंग: इसे सही तरीके से कैसे लागू किया जा सकता है

データ マスキング: 正しく実装する方法

Mascaramento de dados: como pode ser implementado corretamente

Mặt nạ dữ liệu: Làm thế nào nó có thể được thực hiện một cách chính xác

数据屏蔽：如何正确实施

Whether it is answering a question or Elon preaching self-driving cars, business leaders see exciting machine learning use cases everywhere they look. The hype around artificial intelligence (AI) and machine learning (ML) is staggering. So much of the mainstream discussion centers around these exciting applications, but they often leave out the inputs and processes required to build these systems. Alexa Quick digression (Optional). Most media will incorrectly lump all AI applications as the same type. However, there are two types of artificial intelligence: narrow and general. Narrow AI optimizes a specific task or objective, which is 99.99% of use cases out there. Outside of this narrow objective, a human is smarter in every other situation. General AI is developing intelligence that is broader and more human-like (e.g. the Terminator). There is a HUGE difference between the AI used to forecast the weather and the AI to build the Terminator. What is Required to Build a Machine Learning System? The machine learning development lifecycle requires 4 key components: Business Problem Data Machine Learning Algorithm IT Infrastructure Stakeholders requesting a machine learning system define the business problem that is in need of a solution (e.g. forecasting future sales). A tech professional will create a model by training a machine learning algorithm based on a relevant dataset. The trained model will output a prediction based on the patterns found in the dataset. Underpinning this model is the IT infrastructure (e.g. AWS). The infrastructure is required for the storage, compute, and deployment involved in the development lifecycle. Data requires storage. Data prep and model training requires compute (i.e. CPU/GPU). Model deployment and serving requires a host (e.g. API endpoint in the cloud). What is Machine Learning? A simple example of machine learning would be predicting what I wear based on the weather. If you knew the temperature outside, the wind chill, humidity, and what I wore over the past 2 years, then you could likely create a model. The weather indicators (e.g. temp, humidity, wind) would be used to predict what I wear each day. There are likely patterns that map between the weather outside and what I chose to wear that day. Train the model on this data and voila — you have your predictive model. How could this possibly go wrong? Note: The weather indicators are referred to as features, predictors, or input. “What I wear each day” is referred to as the target variable or output. Why do ML Systems Fail? Many stakeholders have the misconception that machine learning is magic. All you need is some data fed into an algorithm and the result is a beautiful model. This couldn’t be further from the truth. Machine learning is a process that finds patterns in the data between the input (features/predictors) and output (target variable). Mathematical functions are used to map each type of input to the most common output in that scenario. The “magic” requires stable, unique patterns between various combinations of the input and output. If the patterns are unstable, or fail to provide enough signal, then the model will likely be useless. This branch of data science is called, data quality. Data quality refers to the set of activities applying quality management techniques to data in order to ensure it is fit to serve the needs of an organization in a specific context or problem. Data quality is the root cause for why many machine learning systems fail. The primary for measuring data quality are the following: dimensions : The data reflects the real-world objects and/or events it is intended to model Accuracy : The data makes all required records and values available Completeness : Data values drawn from multiple locations do not conflict Consistency : The data conforms to defined business rules and falls within allowable parameters when those rules are applied Validity : No record exists more than once in the dataset, even if it exists in multiple locations. Every record can be uniquely identified and accessed within the dataset and across applications. Uniqueness : Data is updated as frequently as necessary to ensure it meets user requirements for accuracy, accessibility, and availability Timeliness While specifies these 6 dimensions and their respective definitions, I would add the following: Informatica of information to encourage uniqueness of records Diversity sample of data from the population to reduce biases in how data was captured and what data was captured Representative data ensures that each value that is incorrect, corrupted, incorrectly formatted, duplicate, or incomplete is fixed or removed Clean Note: If you want to read about other dimensions, I’d check out this from Aug, 1991. MIT paper How Does Poor Data Quality Cause Failure? The weather model would be useless, or worse*, if the data doesn’t meet the standards of high quality per each of the dimensions listed above. Using the table below, we can zoom into the data to see how poor data quality could present itself. Assume that this information represents one adult, the weather where they live and work, and the clothes they wore that day. Data Quality Issue #1: Unclean Data Data scientists and data engineers spend the majority of their time cleaning data. The lack of standardized formatting in the “Humidity” column is problematic. Without standardized formatting, data is unclear. For example, is the “.05” humidity from January 3rd the actual humidity, a typo, or did the user intend to convert .05 to 5 percent? Best practice would suggest only storing numeric data in this column and specifying the “percentage” label in the column header or another document. If I received this dataset, I would remove all “%” signs, any “perc” or “percent” text, remove any whitespace, and either convert the Jan 3 humidity value from .05 to 5 or remove it completely. Data Quality Issue #2: Lack of Diverse data Ignoring the formatting issues, do you notice anything strange about this dataset? Why would someone wear a short sleeve shirt and shorts in January when it is freezing outside? Take a moment and try to brainstorm possible explanations before you continue reading. If you are struggling with ideas, I would suggest questioning each assumption you made about the dataset. . . . . . Here are 5 possible explanations I could come with for why someone would wear shorts and a t-shirt during a cold day in January: Temperature was recorded incorrectly by the user. The clothing worn was recorded incorrectly by the user. The user recorded the temperature in Celsius when it should be Fahrenheit. 22 degrees Celsius converts to 71.6 degrees Fahrenheit. The person never went outside and had to endure the cold temperatures. The person only went outside to exercise. While you are running, your body temperature increases. After digging into this problem further, imagine that we learned this person works from home some days. The dataset above is excluding this important piece of information. Without knowing whether the person is staying home or going to work, the dataset is likely not informative enough. Take a look at the temperature and clothes worn each day. The person seems to wear a short sleeve shirt and shorts on most days, regardless of the temperature. If you feed a model this data, it will likely predict a short sleeve shirt and shorts in the Winter and Summer because of the lack of differentiation across records. Data Quality Issue #3: Non-representative Data Set A dataset must take a sufficient sample of the entire population of data that aims to reduce bias. In the table above, there is no information captured in February, July, August, September, November, and December. There could be important information about how this person dresses during these months that is different than the other months. If this person is a teacher, they might have July and August off and dress in a swim suit because they go to the beach those months. The omission of these months adds bias to the model and limits its accuracy. Other Data Quality Issues While I chose to focus on these 3 data quality issues, the dataset had many more. Below is a list of other data quality problems, or problems, present within the dataset. potential Each column should only have one piece of information (e.g. month rather than full date) The year that the date represents should be captured The dataset is not a sufficient size to create a decent model It is unclear what time of day any weather columns represent It is unclear what time of day the clothing worn represents Column labels should not be included in the values, but rather in column headers or in another document (e.g. data definition document) Categorical data needs to be converted to numeric data before model training How Often Do These Situations Occur? Data quality issues are something you need to consider when analyzing data or making data-driven decisions. In a recent project at work, I had to explain to stakeholders that the machine learning model I developed cannot improve without changes to the data. There was not enough diversity of information for the patterns to yield a higher level of accuracy. always Because our society is becoming increasingly data-driven, data quality will continue to be a crucial branch of data science. Before you kick off development for the next data-intensive application, I suggest checking the data’s quality or seeking the opinion of data professionals who work with it every day. Also published by The Data Generalist here Data Science Career Advisor A model that draws the wrong conclusions is worse than a clearly useless model that gets ignored. Sometimes developing a proof of concept is necessary to convince stakeholders that a larger effort must be undertaken to improve data quality. This situation is one reason why iterative improvement (e.g. agile) is often touted as the best practice in software development. Image source: Stable Diffusion AI Demo

Stories, News & Mentions about INFORMATICA

Informatica’s Stories on HackerNoon

Informatica's News & Mentions