Data Incorporated ApS.

Fescar: A Distributed Transaction Solution Open Sourced by Alibaba

Alibaba is realizing a closed-loop self-healing hardware strategy to automatically resolve failures. In big data systems, operation and maintenance work plays a crucial role in ensuring that service interruptions from hardware and software failures do not threaten the overall stability of platforms. Given the challenges of doing so in massive data environments, groups such as Alibaba are increasingly seeking automated solutions that simplify response efforts from their responsible personnel, with one recent effort being self-healing hardware systems. For Alibaba, offline computing platform MaxCompute handles a massive 95 percent of all data storage and computing needs, which at the growing scale Alibaba’s business already encompasses hundreds of thousands of server units. Among these units, hardware failures are difficult to detect at the software level due to their offline operation. Meanwhile, Alibaba’s unified threshold for hardware malfunction alarming often misses hardware failures that impact applications, with each such failure posing a significant threat to the stability of the cluster. Dealing with these issues first of all involves the timely discovery of hardware failures, followed by effective service migration of failed machines. This article looks at how Alibaba has enabled machines to perform these tasks independently, first exploring the Tianji application management system in Alibaba’s Aspara operating system before introducing Alibaba’s automated self-healing hardware platform DAM. Application Management in Tianji The MaxCompute platform is built on the Alibaba Data Center’s Apsara operating system, within which all applications are managed by Tianji; as an automated data center management system, Tianji manages the hardware lifecycle of various types of static resources in the data center including programs, configurations, operating system images, and data, among others. Alibaba’s self-healing hardware system is closely integrated with Tianji, using Tianji’s healing mechanism to build a closed-loop system oriented to detecting hardware failures and performing automatic repair for complex services. Tianji makes it possible to issue restart, reload, and repair instructions from physical machines in the system. Tianji translates these instructions for each application on the physical machine, allowing the application to decide how to respond to them based on its own business characteristics and self-healing scenarios. Discovering Hardware Failures The most important hardware failures to detect are those which concern hard disk, memory, CPU, and network card power supply issues. The following list details some of the tools and approaches available for dealing with these common problems. Hard disk failures account for more than half of all hardware failures. Of these, the most common type is hard disk media failure, which is usually due to file read/write functions failing, becoming stuck, or running slowly. However, this does not mean that the read/write problem is the cause of the media failure, and the appearance of the media failure first needs to be explained at each layer, as discussed in the following sections. Kernel log Kernel log error refers to the ability to detect an error such as the following in /var/log/messages: Tsar io A tsar io indicator change refers to any change or mutation of indicators such as rs/ws/await/svctm/util. Since read/write operations will pause during the error-alarming period, the pause will usually be reflected in iostat and then collected into tsar. In the tsar io indicator, there is a rule that allows for distinguishing whether the hard disk is working properly, expressed qps=ws+rs<100 & util>90. If there is no large-scale kernel problem present, this generally indicates a hard disk failure. System indicator System indicator changes are usually caused by io changes. For instance, the uninterruptible sleep of processes leads to an increase of load. Smart value hopping Smart value hopping specifically refers to the hop of 197 (Current_Pending_Sector) /5 (Reallocated_Sector_Ct). The relationship between these two values and the read/write exception is twofold. First, after the media read/write exception occurs, 197 (pending) +1 can be observed on the smart, indicating there is a sector to be confirmed. Second, when the hard disk is idle, it will confirm the various pending sectors in the 197 (pending). If the read/write is passed, then 197 (pending) -1 will be confirmed; if the read/write fails, then 197 (pending) -1 and 5 (reallocate) +1 will be confirmed. General discovery conclusions Throughout the entire error-alarming link, it is inadequate to observe just one stage in isolation, as comprehensive analysis of multiple stages is needed to conclusively prove a hardware problem. Since media failures can be proven rigorously, the derivation can be reversed so as to quickly distinguish between software and hardware problems when an unknown problem occurs. The tools discussed above are based on operation and maintenance experience and lessons learned from failure scenarios. Meanwhile, a single source of discovery is far from enough, for which reasons Alibaba has introduced other sources of hardware failure detection and combined various inspection methods. Converging on Hardware Issues As the tools and pathways discussed in the previous section are applied to discover hardware failures, not every discovery should necessarily be reported. Alibaba Group observes two key principles for converging on hardware issues. First, indicators are regarded as much as possible as unrelated to applications and services. Some application indicators which are highly relevant in hardware failures are monitored without being treated as an actual source of discovery. For instance, when the io util rises above 90 percent, this indicates that the hard disk is especially busy but does not necessarily mean it is having problems, such as when this is caused by reading and writing hotspots. Actual problems are weighed when the hard disk is in a state of “io util>90 and iops< 30” for more than 10 minutes. Second, Alibaba observes a practice of being sensitive in collection and cautious in convergence. While all possible attributes of a hardware fault should be collected, most should only serve for reference and not as a basis for repair in the final automatic convergence analysis. To continue with the previous example, if io util > 90 and iops < 30 are the lone conditions present, the hard disk will not be automatically repaired due to the fact that a kernel problem may also lead to such a scenario. Only when explicit faults appear in relation to the indicators, such as the smartctl timeout/fault sector, can the hard disk be diagnosed as faulty. Short of these conditions, the hard disk will be isolated but not repaired. Rate of coverage Taking the IDC work order of a production cluster in a given month x of a given year 20xx as an example, the following statistics can be seen for hardware failures and work orders: Excluding out-of-band (oob) faults, the discovery rate for hardware failures in this month is 97.6 percent. Self-healing for Hardware Failures For every machine that experiences a hardware problem, Alibaba will open an automatic rotation work order to follow up. Currently, Alibaba uses two sets of self-healing processes: the with-application repair process and the no-application repair process; the former is used for hot-plugging hard drive failures, while the latter is used for all hardware failures throughout the rest of the machine. Alibaba’s automation process features several subtly effective designs. First, the process is designed for diskless diagnosis. For downed machines, it is not possible to enter ramos to open the “downtime without a reason” maintenance work order. This design can greatly reduce false positives and in turn the workload of service desk workers. Further, diskless stress tests can completely eliminate the impact of the current version of the kernel or software, and thus truly determine whether or not hardware is experiencing performance issues. Second, the process is designed for determining the impact range of problems and for impact upgrades. For with-application repairs, the method determines whether a process is or is not down. If it is down for more the 10 minutes, it can be inferred that the impact of the hard disk failure has extended to the whole machine, which requires rebooting the machine. If the machine cannot restart when this is happens, there is still no need for manual intervention as an impact upgrade will be performed directly. That is to say, the with-application repair process will be directly upgraded to a no-application repair process. Third, the process features automatic backup for unknown problems. During operation of some machines it is possible for diskless diagnosis to be accessed after the machine is already down while the stress test will not be able to detect any hardware problems. In such cases, the only solution is to reinstall the machine. In some cases, the hardware problem will be detected and fixed during reinstallation of the machine. Finally, the process is designed to enable hardware downtime analysis during the handling of hardware failures. It should be noted that the overall process is mainly oriented to solving problems and that the downtime analysis is only a by-product of the system. Meanwhile, Alibaba has also automatically introduced downtime diagnosis results for analysis, achieving the “1+1>2” effect. Statistical analysis of processes When a single hardware problem repeatedly triggers the self-healing process, then the statistics of process work orders become a means of discovering the problem. For example, in the case of a virtual serial port problem with a Lenovo RD640 machine, it was discovered through statistics that self-healing occurs repeatedly on machines of the same type, and that even after a machine is reinstalled the same problem will reappear. Alibaba thus isolated those machines to ensure the stability of the cluster while working to accommodate an investigation into the problem. Misconceptions about service association With Alibaba’s complete self-healing system, the category of “unknown problems” can be used to deal with problems affecting some services, kernels, and software. However, solving service problems with hardware self-healing methods tends to perpetuate a dependence on this as a backup solution for problems that continue to remain unclear. Presently, Alibaba is gradually removing processing of non-hardware problems and returning its orientation to self-healing for hardware. While general self-healing software can also be run on some systems, the strong coupling of such scenarios and services makes them unsuitable for generalization across the entire Alibaba Group; this approach is more conducive to the classification of hardware and software problems and the discovery of unknown problems. Evolving Architectures Alibaba Group’s self-healing architecture has gone through extensive changes over the course of its development. Its original version was implemented on the control machine of each cluster, as operation and maintenance personnel at that time dealt with various problems occurring on the control machine. With continuous deepening in its automation, the architecture proved a serious hindrance to the opening of data, leading Alibaba to rebuild it on a more centralized basis; this in turn led to the challenge of processing massive data, where a small number of servers proved inadequate. Cloudization To address the above challenges, Alibaba further reconstructed its system in a distributed, service-oriented way that supports large-scale business scenarios, disassembles the various modules comprising the architecture, and introduces three key artifacts: Alibaba Cloud Log Service (sls), Alibaba Cloud StreamCompute (blink), and Alibaba Cloud Analysis Database (ads). In this arrangement, the various collection and analysis tasks are shared by cloud products, and the server only retains the most central functions of hardware analysis and decision-making. The following diagram compares the architectures of self-healing platform versions DAM1 and DAM3: Data-oriented architecture With sustained exploration of its self-healing system, Alibaba has found that the data of each stage yields a stable output, and higher-dimension analysis of this data reveals clear, valuable information. Meanwhile, the group has also reduced the dimensionality of these higher-dimensional analysis results and scored the health of each machine. With this health score, operation and maintenance personnel can quickly determine the hardware situation on a single machine or an entire cabinet or cluster. Service-oriented architecture Based on control of full-link data, Alibaba is able to provide the entire failure self-healing system as a hardware lifecycle standardization service for different product lines. Based on full abstraction of decision-making, the system provides specific thresholds for perception, supports customization of different product lines, and forms a full lifecycle service suited to individualization. A Closed-loop Self-healing System for Failures The most common application scenario in the AIOps closed-loop perception, decision-making, and execution system is self-healing hardware and software. Industry-wide, enabling automatic solutions to failures is the first implementation point of AIOps. For Alibaba, providing a common closed-loop system for self-healing is the cornerstone of its work with both AIOps and NoOps (unmanned operation and maintenance). Such a system is especially important to coping with operation and maintenance needs in a massive system. In a complex distributed system, operational conflicts between different architectures are inevitable, due to information asymmetry between them. The cause of information asymmetry is that each type of distributed software architecture is designed to run as a closed loop. Such issues can be resolved through a strategic use of various mechanisms and operation and maintenance tools. However, these methods are akin to patching, and amid continuous architecture upgrades the need for such patches continues to grow relentlessly. Thus, the process must be abstracted to work on a self-healing basis and must be explicitly declared at the architectural level, so that software can take part in the entire process of self-healing and transform what are originally conflicts into a form of synergy. Presently, Alibaba is dedicating the focus of its efforts to the largest issues in operation and maintenance scenarios, as well as hardware and software conflicts, architecture construction and product design, and improvement of the overall robustness of complex distributed systems through self-healing. Key findings from Alibaba’s work with self-healing hardware across a large number of machines include that the side effects of operation and maintenance tools incorporated into the self-healing system gradually decrease due to their steady refinement. Further, the manual operation and maintenance activities incorporated in the system have gradually become automated. Each O&M operation has a stable SLA commitment time, and is no longer an operation and maintenance script that can run an error at any time. In all, the DAM self-healing system’s function is to achieve universality by fully abstracting operation and maintenance automation on the basis of complex distributed systems. By then constructing a closed-loop architecture for this purpose, its architecture ecology becomes more fully coordinated as a whole. (Original article by Zhong Jiong’en钟炯恩) Alibaba Tech First hand and in-depth information about Alibaba’s latest technology → Facebook: . Twitter: . “Alibaba Tech” “AlibabaTech”

When Machines Fix Themselves: Alibaba’s Self-healing Hardware Solution

How the Alibaba Tech Team is using DevOps to improve the agility and adaptability of their search enabling platform This article is part of the Search AIOps mini series. At the end of 2015, the Alibaba tech team released its enabling platform strategy for constructing organization and business mechanisms based on the creative and flexible concept of “Large Enabling platform, Small Frontend.” The basis of this concept is making the frontend agiler and giving it the ability to quickly adapt to the market, while the enabling platform is used to incorporate digital operation and products and technologies capacities to provide strong support for various frontline services. A crucial link in Alibaba’s enabling platform layout chain is allowing the enabling platform to handle search. However, this exposes the search enabling platform to huge challenges regarding techniques and products as a result of the intrinsic complexity and large business scale of search technologies. The Alibaba search enabling platform was established to support frontend services by increasing their agility, helping them adapt more quickly to changes in the market, and ultimately to eliminate inconvenient search functions. To accomplish this goal, the Alibaba tech team constructed their search enabling platform from scratch over a three-year period, accumulating cutting-edge knowledge and experience in the running of DevOps, AIOps, and offline services on the platform. How Alibaba Developed the Enabling Platform for Search The following figure shows the predicted development trends for the search enabling platform over a three-year period, including a summary of its implementation. The three key developmental stages represented in this figure are manual control, automated script operations, and integrated development and operations. Manual control During this stage, the operation of both the search service department and open source search technologies relied entirely on human efforts. Massive amounts of manpower were wasted on redundant, inefficient jobs. However, PE gradually accumulated experience over time and determined that automated scripts could be used to implement common, repeated operations and save manpower and improve operation efficiency at a time when expertise and domain know-how were only beginning to accumulate. Automated script operations This stage was characterized by the use of open source technology systems. However, working in this fashion naturally separated developers and operators and placed the two roles in opposition to each other. Developers required fast iterations, while operators focused on keeping fewer iterations to maintain online stability. This separation led to a mutual distrust due to online failures resulting from configuration changes and software updates. Ultimately, the two sides reached a compromise that updates would be released every Tuesday and Thursday during the release window. However, this compromise resulted in lower business operation efficiency, which in turn created a large gap between system capacity and the demand on iterations from the business side. Integrated development and operations This is the current stage of development for the Alibaba search enabling platform. To solve the issues with automated script operations, the team developed a new control system based on DevOps for integrating development and operations that provides better solutions for iteration releases. As the business scenarios are essentially a technical systems management process, the team believes DevOps should be used as something beyond just a simple methodology for the integration of development and operations for individual systems. Alibaba hope to establish DevOps as the “ops” above individual system ops. This intrinsically separates the work from other Alibaba DevOps platforms and is represented by the Apsara Base that manages end-to-end processes from deployment to service source codes updates. In essence, Apsara Base users are still considered operators. Therefore, Apsara Base uses IAC (Infrastructure As Code)’s dimension +Git management and deployment configurations to create products. This is a typical approach for designing a DevOps platform and meets the main job requirements. However, Alibaba is often faced with end users who lack expertise in online system operations and get lost when all they can see are configurations or codes. Fundamentally, the team must advance their understanding of DevOps and move towards viewing platforms as products. For this purpose, the team must avoid exposing configurations, codes, and the complexities of industry expertise to users, and transform system collaboration to control the end-to-end experience. Radical improvements in the iteration efficiency of complex search can only be achieved by simplifying processes and controlling the end-to-end full link experience. Years of effort were put into trying these two approaches, which resulted in the implementation of a series of DevOps systems, including Sophon-Bitmain, Bahamut, and Maat. Specific details for these undertakings are described in detail in subsequent sections. Integrating Development and Operations — DevOps Before examining the Sophon-Bitmain system for integrating development and operations, it is important to understand which systems are involved and how they work together when switching to a service with complex search scenarios. The system outlined in the previous figure is divided into three modules: OPS, Online, and Offline. As shown in the figure, the Ops layer is divided into online stateful service ops, online stateless service ops, and offline ops. In other words, each service separately relies on OPS for control. However, the previous figure also shows the complicated transaction results from the collaboration of multiple service systems. Before launching tisplus and before switching in a complex project, the online stateful service team, online stateless service team, offline DUMP team, business side, and PE were brought together to exchange opinions on how to release the project cooperatively. After release, online changes and troubleshooting were performed amid frequent tense exchanges in the support group, which hurt efficiency and only supported ten individual business units. With these pain points in mind, the team can now go back examine must-have features for the construction of DevOps as a complicated search system: 1. Full link OPS that offers an end-to-end experience and accurately matches the definition of DevOps for complex scenarios. 2. Common-sense, process-oriented operations updates in a complex operation control link to target-driven operations control. 3. Decent operations and products abstraction and better enabling of users. 4. Improved business iteration efficiency as the foundation for ensuring business reliability. Alibaba created the Sophon-Bitmain platform to address these pain points, which is elaborated on in subsequent sections. Target-driven Operations Many users are unfamiliar with target-driven operations and initially find the concept too abstract. However, examining operation scenarios from real searches can help achieve a better understanding of the need for target-driven operations control. Consider the following example: The index system is using index Version A and is requested to switch to Version B. When the system is rolling out Version B, a new instruction is received to switch to Version C. For previous methodologies, switching in this manner resulted in crashes. PEs could only perform the process successfully by killing the current switch process, checking which step each node advanced, clearing intermediate processes, and releasing the operations. This demonstrates how unproductive process-oriented operations control can be under complicated operation systems. Conversely, if the same scenario is encountered under a target-driven process, the switch is performed by setting up a new Version C for rolling by the system. This updates the system of the latest target, which is compared with the currently executed progressive target. Once detection changes to the target, the system immediately terminates the current execution route, automatically clears inconsistent states, and begins distributing the notice to execute critical routes of the latest target state. Upon receiving the latest orders, each node starts progressing toward the new target. This progressive and consistent operation method naturally shields the complexity of intermediate operation states, making complex operations control more simple and flexible. Alibaba’s operation platforms have all been changed to target-driven operation from the top down due to these benefits. Simplified Operation Concepts Another commonly-used method for simplified operations is replacing hosting with enabling, meaning that users must shoulder more responsibilities before enjoying more powerful search abilities. However, enabling end users does not mean they should be exposed to the abstruse, complex concepts of search systems and their operations. Simplifying the system’s operation concepts and sealing complex information and industry know-how inside the system is one of the core tasks of Alibaba’s Sophon-Bitmain. The lower portion of the previous figure displays the infrastructure and online services for each data center from the viewpoint of PE. Without a control abstract layer, users are exposed to the same level of complexity as PE, which can be extremely confusing. Sophon-Bitmain limits complexity in several ways. First, it abstracts objects into a group of data relation models called operation control models (as shown on the right side of the previous figure). However, the resulting operation abstract layer is still too complex for users, who should only be shown specific business abstracts containing business scenarios. As a solution, Sophon-Bitmain added a business abstract layer to the first layer of the operation abstract. An example is the three concept layers in the top left corner of the previous figure: business logic (plugin, configuration), service (deployment relations), and data (data source & offline data processing). Users can accept the definition of this layer at almost no cost. Therefore, Sophon-Bitmain’s ability in abstracting operations and simplifying business concepts makes it possible to switch from hosting to enabling users. Guaranteed service reliability Sophon-Bitmain guarantees service reliability in several aspects. As Sophon-Bitmain supports an increasing number of leading core businesses, Alibaba must provide SLA guarantees to search services, and respond to each business’ reliability requirements by deploying online and offline services flexibly. Automatic disaster tolerance switchover is also a required feature. At its current service reliability level, Sophon-Bitmain supports the unitization of online and offline search service, cold standby deployment of offline data, and automatic disaster tolerance switchover of the query and data backflow links, as shown in the following figure: One indicator of improved iteration efficiency is that iterations can now be released anytime, anywhere, instead of using the previous time window-based online release method. However, this does not mean release can be performed at will without considering potential risks for quick iteration releases. To achieve the target of safe and efficient iteration releases, Alibaba designed and standardized a set of iteration release codes. For example, a normal business iteration must be verified in both daily and pre-release environments. The team also added a multi-layer authentication mechanism to ensure the reliability of releases. For example, when upgrading plugins and algorithm strategies, the team requests that pressure tests be performed on a clone. If the performance degrades too much, the release process is rejected. In addition, single-data center tangential flow phased release, smoke authentication, and similar features can be defined in launch procedures. These changes have given Sophon-Bitmain powerful multi-layer authentication and quick disaster tolerance switch over abilities, while the risks of quickly-released business iterations and improving iteration efficiency. Accumulating expertise While the functions of search technology systems are strong, they operate under the control of numerous house rules (e.g., expertise requirements for complex operations control and business iterations) that overwhelm everyday users. For example, in search scenarios, changing a single field on the business side can necessitate multiple online and offline changes to associated configurations. Involving users in this process can require them to make complex judgments, such as: · Will online services, offline services, or both be affected? · Will pushed configurations take effect online or offline first? · Will all configurations take place simultaneously following a full backup? Requiring users to make this sort of judgment can result in them feeling irritated and confused. Therefore, it is important for Alibaba to utilize the engine service expertise from the Sophon-Bitmain DevOps layer to contain these complexities within the platform, and manage all associated complex knowledge and decision-making behind the scenes. The operations platform can break down and then execute complicated operations internally according to the pre-defined expert operations DAG drawing in organized stages, as shown in the following figure: Continuously inputting operational expertise into the system (execution flow of the operation DAG), decreases the platform’s usage costs while increasing the efficiency of iterations. In situations where operations are increasingly complex (for example, when the operations exposed to users cover more and more services), the operations DAG execution link can evolve from a simple stage to a complicated stage where multiple execution branches exist. Determining the optimal execution link also becomes a detailed undertaking (as shown on the right side of the previous figure). Alibaba refers to this operation as “shortest path routing.” This represents an attempt at achieving smart operations, as well as indicating the direction Alibaba is taking for future efforts in this discipline. From system to full link Before discussing offline platform technologies, it is important to cover some background on the universal demand placed on offline processing by search. This has been a common theme for offline cross-team discussions since before offline platforms for complex business were established. For engines, business data does not just consist of a simple database table, but rather it can come from a number of homogeneous data sources or heterogeneous data sources. Every search business demands full backup and incremental backup. Therefore, it is important to determine how the team turns the different data source relations corresponding to businesses into the upper-layer abstract, block internal processes, and consolidate incremental backup and full backup. Implementing full backup codes and incremental backup codes for every new business is not ideal. Looking back at the previous work, Alibaba’s insufficient offline support boiled down to engine schema-defined data sources being weakened to resource pairs for abstraction and management, resulting in a failure to extract fundamental abstracts. Currently, all switched-in data resources are Dynamic Tables. When defining these resources with a table abstract, certain universal APIs can converge and remove the need for repeated development, including: · Creating tables · Deleting tables · Modifying tables · Adding, deleting, modifying, and querying table data · Defining table relations, This inspired Alibaba’s approach and overall mentality when it established the offline components platform, bahamut. The platformization of online components only platformizes offline data processing abilities. A search scenario is always a dynamic process featuring online and offline collaboration and combination. All the business scenarios are, as mentioned previously, the result of coordinating technical systems. The most important and challenging part of this process is ensuring efficient online and offline collaboration and offering users an end-to-end experience. The following figure shows how when using offline data, end users always see visual data relation definitions and the simple execution list dump->Build->switchindex. Behind the system is a sophisticated state machine that manages online and offline collaboration, as the team have blocked all complexities. Next, the team will share how they transformed the personalized demand of every online search business on offline data processing into an abstract, and finally, satisfied the demand via a platform. The bahamut platform supports users to define relations between their respective data source information and tables (the team supports joint operations between heterogeneous tables, such as odps and mysql). The team then submits this frontend graph to Bahamut for translation. Bahamut then parses, optimizes, splits, and translates this graph into several graphs that can be executed by blink, including: · Incremental backup syncBlink tasks · Full backup BulkLoad MR tasks · Blink Join tasks. The two most critical graph nodes in this instance are merge and left join. Merge transforms the processing of all 1:1 and 1:N relational tables into an HBASE staging table. Regarding the processing of N:1 relational tables, driving is currently only supported on the side of the master table N (merchandise table). In other words, N side tables are updated via blink sync and then merged with tables on the 1 side (user tables) via blink Join1 to form complete row records. These row records are then sent to SwiftSink (incremental) and HDFSSink (full). Finally, they flow back to BuildService (an index construction service) to construct indexes, as shown in the following figure: Online and offline control and collaboration, together with the construction of the Bahamut components platform, gives users powerful ability via a visual means to process and compute complicated offline data relations. This ensures significantly more efficient business support and highlights the platform as a milestone in terms of its capabilities for offline consolidation and providing an end-to-end offline experience. Alibaba is also working on converting offline capacity to general online service capacity. Alibaba believe that in the near future, the offline component platform will not remain focused on HA3 search scenarios only, but it will extend to online search services in their entirety. (The original article is written by Liu Ming柳明) Alibaba Tech First hand and in-depth information about Alibaba’s latest technology → Facebook: . Twitter: . “Alibaba Tech” “AlibabaTech”

DATA INCORPORATED APS.

datainc.dk

-- employees

Since n.d.

COMPANY RANKING

DATA INCORPORATED APS.

EVERGREEN INDEX #3307

Representing this company? Click here to claim and customize this page!

Data Incorporated ApS. WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DATA INCORPORATED APS.

datainc.dk

-- employees

Since n.d.

COMPANY RANKING

DATA INCORPORATED APS.

EVERGREEN INDEX #3307

RELATED COMPANIESRANK

STORIES ABOUT DATA INCORPORATED APS.

Representing this company? Click here to claim and customize this page!

Data Incorporated ApS. WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!