You can stop ransomware attempts in their tracks if you have the right monitors in place, and if someone is watching them. Your systems should be set up to send automatic alerts when they detect anything unusual, and you should have protocols in place to  follow up on these alerts.

Know When and How to Stop Ransomware Attacks

Malware can compromise the integrity of your website, leading to potential data breaches, the loss of sensitive information, and damage to your online reputation ... check for any signs of ...

Malware Removal WordPress Website Safeguard Website: Protecting Your Online Space

The Straits Times spoke to cyber-security experts to understand how one can protect one’s phone from malware, and whether it is still possible to shop online ... and run an anti-virus software ...

askST: How can I protect my phone from malware? Can I still shop online safely?

𝙐𝙎𝘼, 𝙉𝙚𝙬 𝙅𝙚𝙧𝙨𝙚𝙮- Our report on the Global Malware Sandbox Solutions Market ... It also provides an in-depth analysis of the competitive landscape, the key ...

Malware Sandbox Solutions Market Size Assessment and Insights into 2023-2030

Detecting zero-day malware using dynamic analysis techniques has proven to be far more effective than traditional signature-based methods. One specific approach that has emerged in recent years is the use of graphs to represent executable behavior ...

Using Deep Graph Learning to Improve Dynamic Analysis-based Malware Detection in PE Files

The country ranks fourth globally in online banking malware detection, with 8.2% of the overall global threats. In H1 2023, 5,609 online malware threats were identified. India also ranks among the top five nations globally in malware detections ...

India's security landscape under attack as ransomware, malware threats spike: report

DUBAI, DUBAI, UNITED ARAB EMIRATES, October 10, 2023 /EINPresswire / -- ANY , a cybersecurity company developing an interactive sandbox analytical platform for malware researchers. ANY processes ...

Malware Trends Report From ANY.RUN: Q3 2023

The country ranks fourth globally in online banking malware detection, with 8.2 per cent of global ... Disclaimer: This post has been auto-published from an agency feed without any modifications to the text and has not been reviewed by an editor

India's security landscape under siege as ransomware, malware threats spike: Report

ANY.RUN is an interactive online malware analysis service created for dynamic as well as static research of multiple types of cyber threats. 

ANY.RUN

How to Use ChatGPT for Malware Analysis 

Malware Sandboxes: 5 Reasons You Need One

Analyzing GuLoader: How to Approach Deobfuscation of Complex Samples 

Since the dawn of the digital age, malware has been a constant concern to computer systems. In fact, every technological advancement provided threat actors with additional tools for making their creations more sophisticated and destructive. However, a year into the new era marked by the rise of , it seems this trend is reversing, as cybersecurity professionals are now becoming the main beneficiaries of solutions like ChatGPT. generative AI How exactly is AI useful in addressing malware attacks? The seemingly limitless capabilities of ChatGPT make it a highly versatile instrument, suitable for numerous scenarios in cybersecurity. To demonstrate, let’s focus on three tasks that are performed practically by every malware analyst, and that can be greatly facilitated with the help of an AI assistant. 1. YARA rules creation YARA rules are an essential mechanism for detecting malware based on certain patterns. To ensure proper threat coverage, analysts need to write many of these, and doing it is hardly a walk in the park, especially in terms of the time needed. Thankfully, and largely automate the entire process by printing such rules on the spot. All one needs is to provide the chatbot with proper instructions. Of course, in most cases, a little touch-up will be required. ChatGPT can significantly accelerate Here, ChatGPT failed to specify that strings can be in 2 encodings, ASCII and wide and missed an extra question in the $str4 string. Yet, for a rule produced basically in seconds, it is impressive and extremely useful for speeding up the workflow. Use the prompt to create your rules with ChatGPT: GPT, could you help me write a YARA rule? I am trying to detect a specific malware sample which has the following characteristics: [REPLACE WITH CHARACTERISTICS]. How can I write a YARA rule that accurately identifies this malware? Do not explain about YARA, provide a rule, following with an overview of the logic. 2. Suricata rules writing Suricata rules are another part and parcel of effective malware detection and analysis. In this regard, ChatGPT also proves to be a nifty tool, offering variants that are almost as good as those written by a junior analyst. As it becomes clear from the example, the chatbot still has room for improvement, but by treating its results as a rough draft that can give you a foundation to build on, you can once again save a great deal of time. Use this prompt to generate your rules: ChatGPT, please generate a Suricata rule that detects [YOUR CONDITION]. Use the following information if provided: Options: [options] Actions: [actions] Headers: [headers] Please note that these elements may not always be provided. If none of these details are given, please create a rule that just detects [YOUR CONDITION]. 3. Understanding malicious activities Still, the key use case for ChatGPT when it comes to malware analysis is the ability to learn more about specific actions undertaken by different threats. For instance, in the example below, we asked the chatbot about the ways malware can exploit the legitimate utility w32tm.exe, and it gave a solid response. In fact, you can access such information in a more convenient way by using the free sandbox. The service is intended for analyzing files and links through direct interaction in a safe cloud Windows VM. ANY.RUN It not only detects malicious network traffic, processes, and registry changes but also lets you gain additional insights into the objects of your interest, including triggered Suricata rules, using its built-in ChatGPT feature. Thanks to this, you can get a comprehensive understanding of how and why malware performs certain activities and what that means for your infrastructure’s security. Check out the video below to see how the chatbot breaks down the commands entered by the malware in the command line and highlights their purpose. https://www.youtube.com/watch?v=uJyAN5_c9ng&embedable=true Conclusion For now, generative AI is by no means spelling doom for the entire malware industry. However, the likes of ChatGPT are clearly making it easier than ever for professionals to do their work, enabling them to respond to attacks faster and improve organizations’ security posture. Integrating chatbots into routine workflows, particularly through platforms like ANY.RUN can be remarkably advantageous and serve to considerably boost your proficiency and efficiency as an analyst. Explore ANY.RUN’s entire range of features, including your private team space, Windows 10/11 VMs, and API integration, . using a 14-day free trial

ম্যালওয়্যার বিশ্লেষণের জন্য কীভাবে ChatGPT ব্যবহার করবেন

So verwenden Sie ChatGPT für die Malware-Analyse

Cómo utilizar ChatGPT para análisis de malware

Comment utiliser ChatGPT pour l'analyse des logiciels malveillants

मैलवेयर विश्लेषण के लिए चैटजीपीटी का उपयोग कैसे करें

ChatGPT をマルウェア分析に使用する方法

악성 코드 분석에 ChatGPT를 사용하는 방법

Como usar ChatGPT para análise de malware

Как использовать ChatGPT для анализа вредоносного ПО

Kötü Amaçlı Yazılım Analizi için ChatGPT Nasıl Kullanılır?

Cách sử dụng ChatGPT để phân tích phần mềm độc hại

如何使用 ChatGPT 进行恶意软件分析

Cyberattacks have long been one of the most devastating dangers to businesses of any size, and the problem is only getting worse. As , just over the past three years, the average cost of a data breach increased by 15% and climbed to $4.5 million. To withstand attacks, companies need to invest in an expansive security infrastructure. A is a core component of such an infrastructure, as it helps cyber professionals safely analyze malicious files to extract crucial information fast. reported by IBM malware sandbox What is a malware sandbox exactly? Most of us are well-familiar with antivirus software and, especially, the number of alerts it often generates. While many of these turn out to be false positives, some of them may point to actual threats that can deal a fatal blow to your system. To get a better understanding of these alerts, malware sandboxes are employed. Basically, they help analysts determine whether a certain file or link contains any malicious payload capable of compromising data. By uncovering the file or link's activities in a sandbox, analysts can make an informed decision about whether it is safe to open or click on it. Sandboxes work by that lets researchers run the file they want to analyze in a safe environment. Since the VM is isolated from the user’s computer, professionals can interact with the file and see how it behaves on a granular level. Once the analysis is over, the sandbox generates a detailed report on the activities of the sample, such as the network and registry ones, and returns a result on whether it is malicious. creating a virtual machine How can a malware sandbox help my organization? Having a malware sandbox as part of your security stack can greatly accelerate the work of your SOC and DFIR departments, helping them gain an in-depth insight into malware much faster. The intelligence collected as part of such analysis can be used to enrich your detection capabilities and strengthen the organization’s defense. Let's take a closer look at the key benefits of a malware sandbox. It cuts malware analysis time to seconds Speed is the most crucial quality of any sandbox. Analysts may spend hours trying to understand if a file is harmful when examining it by hand. largely automate this process and generate a comprehensive overview of the malware in seconds. As a result, a researcher who would otherwise process no more than a dozen potential threat alerts a day can easily increase this amount by five times with the help of a sandbox. Sandboxing solutions It examines malicious behavior with microscopic precision Fast does not mean superficial. Sandboxes provide a wealth of information and are considered among the most powerful tools available to security researchers. For instance, they automatically track the network stream related to the analyzed file or link and document its registry activity, flagging all the suspicious and malicious events. On top of that, criminals often resort to the use of code obfuscation and segmentation of execution into multiple stages as a means of deceiving researchers. Sandboxing platforms make untangling these clever (and not-so-clever) techniques a much simpler task. It offers real-time interaction and flexibility Although they involve a good deal of automation, sandboxes such as still provides users with full control over the process. ANY.RUN supports live interaction, which allows analysts to engage with the virtual machine environment and execute different operations in it. This can be useful for running programs and browsers to trigger malware. ANY.RUN Sandboxes are also configurable, with a wide variety of settings that can be tuned to better detect malicious activities. For example, by changing the VM's locale, it is possible to detonate malware that is intended to target specific geographic regions. It slashes overheads by a tangible amount Using a sandbox can save you resources across several areas. It can reduce the amount of time spent investigating potential malware, freeing up staff to focus on other tasks, such as incident response and security audits. It can also lower the competence barrier, making even junior specialists capable of malware research. Sandboxes can also help to get rid of the need to run a , as they provide a turn-key solution with an already built-in analysis toolset. In addition to reducing operational expenses, sandboxes, in combination with other security solutions, can save organizations from the cost of lost data, downtime, and remediation, which every successful attack entails. custom VM It improves compliance Many regulations, such as HIPAA, require organizations to take steps to protect their data from cyber threats. Using a malware sandbox can help you meet these requirements. Of course, it should not be viewed as a replacement for other essential technologies such as firewalls, access control systems, as well as intrusion detection and prevention systems. Nonetheless, it can provide an added layer of protection to your organization's existing security infrastructure. Conclusion Wrapping it up, a sandbox solution such as can strengthen security teams’ capabilities, helping them collect vital intelligence on cyberattacks. It also significantly speeds up and simplifies the threat analysis process, freeing up the hands of the staff for other tasks at hand. Overall, incorporating a sandbox can help your organization mitigate the risks of a successful cyberattack and the resulting lost productivity and reputation damage. ANY.RUN Use ANY.RUN’s to see how a malware sandbox can improve your security posture. 14-day free trial

Sandboxes de malware: 5 razones por las que necesita uno

Malware Sandbox : 5 raisons pour lesquelles vous en avez besoin

मैलवेयर सैंडबॉक्स: 5 कारण जिनकी आपको आवश्यकता है

マルウェア サンドボックス: サンドボックスが必要な 5 つの理由

Sandboxes de malware: 5 razões pelas quais você precisa de uma

Hộp cát phần mềm độc hại: 5 lý do bạn cần một

恶意软件沙箱：您需要沙箱的 5 个理由

As a researcher, you're often up against complex, heavily obfuscated samples. And GuLoader, the malware we're dealing with today, is a classic case in point. malware Take a look at this pseudo-code, produced by decompiling its assembly code — it is ugly and unreadable. Faced with a puzzle like this, you might find yourself at a loss. Where should you even begin? How do you tackle the analysis of this sample? Let's break it down. In this article, then, we’ll explore strategies to deobfuscate such code, using GuLoader as a reference. You’ll learn about: Common obfuscation tactics used by adversaries How to defeat them How to deobfuscate a GuLoader sample The article is based on the previously published by ANY.RUN. Visit our blog to find the sample we'll analyze, as well as unpacking instructions and a Ghidra script that partially automates much of what we’re going to cover from this point on. GuLoader malware analysis This article focuses on static analysis. But if you want to analyze a GuLoader sample dynamically, you can use cloud malware sandbox. ANY.RUN of the Enterprise plan using your business email. Take advantage of the extended lifetime of VM instances, unlimited tasks, and Windows versions 7 through 11. Apply for a free 14-day trial Dynamic analysis lets you see how malware works in the real world by linking its behavior to its technical setup. It's a way to check its actions across different system setups and collect IOCs. Without delaying further, let's dive in. Identifying Obfuscation Methods After unpacking the sample, we begin with a manual analysis of the shellcode and quickly realize that it’s obfuscated. Studying the code enables us to group the obfuscation techniques GuLoader employs into categories: XMM instructions Unconditional JMP instructions Junk instructions Fake comparison instructions Fake PUSHAD instructions Fake PUSH instructions Opaque predicates And arithmetic expressions This is a good time to stop, analyze the obfuscation methods being used, and develop a debofuscation strategy. For example, in our case, most of these techniques introduce code that doesn't alter the final execution outcome. Therefore, we can often safely “NOP” them to improve readability. But proceed with caution — not all obfuscated code is irrelevant to the program's operation, as we’ll soon discover. Now, let's examine these obfuscation techniques individually, and see how to defeat them. 1. XMM instructions The code is littered with many XMM instructions. These appear disordered and add complexity to the analysis process. We encounter them right from the first byte of the unpacked shellcode. Note that many emulation engines stumble over them due to the lack of default support. We put Angr, Triton, and Ghidra's embedded engines to the test — all fell short. How we dealt with it In GuLoader's case, XMM instructions don't actually impact the intended behavior of the code. You’ll encounter similar obfuscation methods in a lot of malware. Consequently, we can safely replace all XMM instructions with "NOP," as shown in the following table: Here's how the result looks in the Ghidra disassembler: 2. Unconditional JMP instructions Unconditional JMP instructions segment the code into smaller chunks. This method is frequently used to avoid detection by antivirus software and other security tools. Additionally, it can make the analysts' job more time-consuming and frustrating, as they must jump between these blocks, especially when dealing with a large amount of code. GuLoader and other malware commonly employ this technique. How we dealt with it This obfuscation method is quite easy to defeat. The disassembler in the decompiled code often successfully concatenates these blocks, improving code readability, even with these unconditional jumps present. As such, we can leave the small blocks as they are without needing to merge them. 3. Junk instructions Junk assembly instructions often come into play as an extra layer of obfuscation. These instructions do not perform any tangible function, typically leaving the register values, execution flow, or memory unaltered. You will encounter these within GuLoader as well. Look out for instructions that perform no action ("NOP", "FNOP") and those that shift or rotate by zero bits ("SHL reg, 0"; "ROL reg, 0"). Other non-impactful instructions like "OR reg, 0", "XOR reg, 0", "CLD", "WAIT" are also present. How we dealt with it Addressing fake comparison instructions is more challenging than just replacing junk ones with "NOP". We can't remove all comparison instructions as some are necessary for the correct code function. One way to tackle this is to “mark” all comparison instructions we encounter. If no instructions are found that use the comparison's result, it's safe to NOP it. If we find a conditional jump or similar, we unmark the comparison to avoid removal. The following table shows an example where all comparison instructions except "CMP EDX,0x0" were selectively replaced with NOP: 5. Fake PUSHAD instructions GuLoader also employs the obfuscation tactic of using fake "PUSHAD" instructions, coupled with a matching "POPAD" instruction. They can temporarily modify register values but are nullified by the "POPAD" restoring the original register values. How we dealt with it Our research showed that all "PUSHAD" instructions in GuLoader are extraneous. So, we address this by replacing "PUSHAD", "POPAD", and the intermediate instructions with NOP: However, not all "POPAD" instructions in GuLoader are junk. We leave those without a corresponding "PUSHAD" untouched. 6. Fake PUSH instructions Another obfuscation technique similar to the previous one is the use of fake "PUSH" instructions. These instructions push a value onto the stack only to pop it off immediately. An example is the inclusion of a "PUSH SS" instruction, possibly followed by instructions modifying a register or memory location. However, the subsequent "POP SS" restores the stack pointer to its initial value. How we dealt with it Defeating fake PUSH instructions resembles the process for fake PUSHAD, but it's crucial to leave the non-pushed registers unaltered. 7. Opaque predicates Opaque predicates are conditional statements that always return true or false, yet they're tough to analyze or predict. These are found in GuLoader's code and complicate the logic understanding. For instance, a pair of instructions like “MOV BL, 0xB6” and “CMP BL, 0xB6” could be followed by a conditional jump like “JNZ ADDR”. The comparison always returns false since the compared value equals the moved value, making the jump unnecessary and baffling. How we dealt with it Overcoming opaque predicates can seem challenging due to the requirement of "predicting" the jump condition. However, our situation is more straightforward as all opaque predicates fall within the "PUSHAD" and "POPAD" blocks. Therefore, we simply replace all predicates between these instructions with NOP. 8. Arithmetic Expressions Obfuscated arithmetic expressions are among the more interesting techniques used by GuLoader. They make it tougher to understand the actual operations performed. These expressions incorporate arithmetic actions like addition or subtraction. Sometimes, they’re mixed with other obfuscations like fake comparisons, opaque predicates, and junk instructions. One example is moving a constant value into a register and executing arithmetic operations: Another instance is pushing a constant value onto the stack and performing calculations on the memory: How we dealt with it To deobfuscate GuLoader's arithmetic expressions, we adopt an approach similar to handling fake comparisons. We mark all "MOV" instructions where the second argument is a scalar value, and all "PUSH" instructions where the argument is scalar. On encountering an arithmetic operation, we update the constant value in the first instruction and replace the current one with NOP. This way, the first instruction has the result, and subsequent arithmetic instructions are replaced by NOP. Below are examples with optimized “MOV” and “PUSH” operations: Be cautious with operand sizes. Preserving the correct size during operations is crucial. Wrapping up our analysis In this article from experts, we used GuLoader as a real-world example to highlight typical deobfuscation strategies. We started with analyzing the code, then grouped similar obfuscation tactics, and finally, thought out unique ways to tackle each. ANY.RUN But it's crucial to understand that these techniques aren't a one-size-fits-all solution. Each malware sample may present its unique obfuscation strategies that need unique countermeasures. Our discussion underlines the significance of breaking down a complex task like deobfuscation into clear, manageable steps. Remember, that successful deobfuscating of any malware involves detailed analysis, spotting obfuscation patterns, and creating optimized strategies.

Análisis de GuLoader: cómo abordar la desofuscación de muestras complejas

Analyse de GuLoader : comment aborder la désobfuscation d'échantillons complexes

GuLoader का विश्लेषण: जटिल नमूनों की अस्पष्टता को कैसे दूर किया जाए

GuLoader の分析: 複雑なサンプルの難読化を解除する方法

Analisando o GuLoader: como abordar a desofuscação de amostras complexas

Phân tích GuLoader: Cách tiếp cận gỡ mã nguồn mẫu phức tạp

分析 GuLoader：如何实现复杂样本的反混淆

Gamified Learning With An AI Board Game Tournament: Course Design

Looks like it’s part 2 of my series on Data Quality! In a , I explored Airbnb’s strategy for enhancing data quality through incentives. They implemented a single score and clear scoring criteria to establish a common understanding between data producers and consumers, fostering a genuine sense of ownership. previous post https://hackernoon.com/data-quality-score-one-score-to-rule-them-all?embedable=true Now, Lyft is taking a distinct approach, not attempting the same thing differently, but rather focusing on different aspects of data quality. And, Lyft’s strategy complements Airbnb’s efforts. While I regard Airbnb’s DQ score (or any similar score) as an effective means to consolidate various attempts at elevating data quality, Lyft is tackling this challenge from a different angle. The Airbnb’s DQ score serves as a valuable tool for providing a concrete visualization of data quality. In essence, any initiative to enhance data quality should manifest a discernible impact on this score. On the other hand, Lyft presents one possible initiative to proactively enhance quality by testing and validating data against specific quality criteria. https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6?source=post_page-----415baad6c95b--------------------------------&embedable=true Fundamentally, it’s a different point in the data quality lifecycle. Introducing a mechanism to improve quality necessitates the ability to measure it initially. So, while ’s focus lies on and data quality, relying on the producer’s interest to enhance this quality and “look good,” Lyft takes a different approach. Lyft places emphasis on actively and data quality, providing both producers and consumers with the means to effectively improve and control the quality. Airbnb measuring observing testing validating Collectively, these approaches provide a comprehensive strategy to address and enhance throughout its . data quality lifecycle For this reason, I was particularly interested in taking a closer look at Lyft’s approach. Another factor that intrigued me is testing, more specifically, contract testing, which has been used for many years now in basic software engineering with the emergence of microservice architecture. Data contracts, however, are something more recent in the domain of data engineering and are seen as the pinnacle or one of the ultimate steps to put in place on the path to building high-quality data pipelines. This is why I wanted to examine Lyft’s approach in more detail and explore some potential parallels. As mentioned, the approaches taken by Airbnb and Lyft are complementary and aim to achieve the same goal: improving data quality. has developed the DQ score, which focuses on measuring and enhancing 4 distinct : Airbnb aspects of data quality DQ Score has guiding principles, including full coverage, automation, actionability, multi-dimensionality, and evolvability. It has dimensions like Accuracy, Reliability, Stewardship, and Usability. Lyft’s Verity is a platform designed to enhance data quality across 5 dimensions Defines data quality as the measure of how well data can be used as intended, covering aspects like . semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness It’s easy to draw parallels between the 5 data quality aspects improved by Lyft’s Verity and the 4 data quality dimensions measured by the Airbnb’s DQ score. For example, aspects like Timeliness would certainly contribute to the DQ score’s Reliability, while Accuracy would be dependent on semantic correctness, completeness, and uniqueness of the data. The Usability score, on the other hand, is influenced by the consistency of the data among other factors. Lyft’s Verity focuses on defining related to . It follows a test-first and validation approach, whereas Airbnb’s unified DQ score emphasizes evaluating data quality through various dimensions. checks semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness If we wanted to incorporate the DQ score into this last schema, it would be on the side of the Alert/Debug steps. Airbnb’s DQ score uses different signals to assess data quality across the . Accuracy, Reliability, Stewardship, and Usability aspects We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). As discussed earlier, there are likely overlaps between Airbnb’s DQ score and Verity. While Airbnb focuses on pushing data quality to the right, emphasizing measurement and scoring, Lyft’s Verity takes a proactive approach by shifting the check definition configurations, testing, and validation processes to the left, emphasizing proactive improvement of data quality. Now, my primary interest lies to the left, in the check definition configurations, testing, and validation aspects. How Lyft integrates data quality testing into its processes? Let’s first examine the execution paths. Currently, Lyft’s Verity is primarily focused on ensuring the quality of data stored in its Hive data warehouse. However, there are plans to expand its capabilities to support other data sources in the future. Note that while they refer to Hive as a data workhouse, they actually utilize it as a hybrid data storage solution, operating as a data warehouse for structured, processed, and cleansed data (silver layer) while also serving as a data lake for raw event data (bronze layer). Poor data quality in Hive caused tainted experimentation metrics, inaccurate machine learning features, and flawed executive dashboards. Verity’s checks can be integrated into an Airflow DAG to ensure that only high-quality raw data is processed and stored in Hive as derived data. Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed inside or . Airflow Flyte … The VerityAirflowOperator can be used in a blocking fashion to halt a DAG upon a check failure, preventing bad data from ever reaching production. This utilizes the “ ” pattern: we create data in a staged schema, verify the data with a blocking operator, then promote it to production if it passes quality checks. Stage-Check-Exchange Checks can also be performed manually or scheduled automatically to verify both raw and derived data. Verity Scheduled Checks are isolated from any data orchestration engine, so they still run even if Airflow or Flyte are completely down. This remedies a common problem of checks not alerting because the Airflow Task never ran. So, there are essentially 3 primary ways to trigger a check: as part of an Airflow DAG, manually, or scheduled through the Verity platform/UI. I don’t believe that current checks can be integrated into real-time streaming pipelines (such as Flink + Kafka) to validate for example data as it enters Hive (or even earlier). Implementing this type of real-time check would enable prompt detection of discrepancies, leading to reduced storage and processing costs and overall data quality enhancement. Well, to be thorough, Verity checks are managed through an API server, which could be used to trigger checks programmatically through some APIs. Verity API Server — This service handles all the external APIs regarding running checks as well as persisting and retrieving their results. The API Server does not execute any checks, but rather writes a message to our Check Queue, which utilizes (SQS). SimpleQueueService So, maybe, you could potentially trigger these jobs in a more real-time fashion, such as from a streaming job or even, as a long stretch, by integrating with Hive change data capture (CDC). However, when executed outside of Airflow, these jobs would not be able to block the data processing job; instead, they would generate asynchronous alerts pushed to the Check Queue. Some consumers would prefer to have data processing delayed when a check fails, while others would rather proceed and receive an alert. Now, let’s look at these data quality tests. Here’s an example that checks whether is never null. This is accomplished through a combination of query and condition components. rider_events.session_id core rider events session_id is not null: # check name
  metadata:
    id: 90bde4fa-148b-4f06-bd5f-f15b3d2ad759
    ownership_slack: #dispatch-service-dev
    tags: [rides, core-data, high-priority]
  query:
    type: dsl
    data_source_id: hive.core.rider_events
    filters:
      - session_id = null
  condition:
    type: fixed_threshold
    max: 0
  notifier_group:
    pagerduty_policy: dispatch-service
    email: dispatch-service@lyft.pagerduty.com Verity is primarily focused on defining and enforcing data quality checks rather than defining complete data schemas. Schema validation is not a novel concept. Several methods exist for defining event data schemas in event-based systems, such as JSON Schema, Protocol Buffers, Avro, or storage formats like Parquet. The optimal choice depends on your tech stack, usage, and specific requirements. While data schemas are valuable for defining the overall structure of data objects or table rows, they fall short in capturing more sophisticated validation checks specific to consumers, such as data distribution, business rules, SLAs, and thresholds. Data contracts go beyond schema validation, which focuses on identifying . I personally find that JSON Schema offers here a more suitable and readable option, effectively separating these structural and syntactical validation capabilities from serialization or storage concerns. syntactical errors However, to address and enhance data accuracy, having an effective means of defining data checks allows for defining the other facet of data contracts. semantic errors This is where Verity DSL comes into play. Beyond semantical validation, data contracts offer another crucial aspect that deserves attention. From a syntactical standpoint, validation checks remain consistent regardless of the consumer or producer involved. The set of validation rules is not tied to any specific consumer or producer and can be defined once and for all as a single schema. However, the Verity data contract DSL offers a finer granularity defining small independent rules, which is particularly well-suited for this context: the semantic meaning and usage of the data vary depending on the specific consumer. Additionally, not all consumers need to utilize all of the properties of an object. Their expectations differ. This doesn’t imply they are contradictory (which would certainly be an issue), but rather complementary and distinct ones. Therefore, allowing all consumers to establish unique rules that, when combined collaboratively, can provide a comprehensive understanding of the semantic significance of data quality is quite crucial. And it’s this aspect that particularly resonates with me. Bear with me, this may seem like a stretch, but from my perspective, it’s worth mentioning. :) collaborative Data exchange enables different teams (producers and consumers) to collaborate effectively. Establishing a shared understanding of these data exchanges is paramount, just like APIs in traditional software development. In microservice architectures, a known as consumer-driven contracts (CDC) has emerged, where consumers define the expected behavior of APIs provided by producers. Producers are responsible for verifying these contracts before releasing new versions. collaborative testing approach I think data contracts share a similar collaborative spirit. Even though data validation is performed on real data, rather than at release time, and does not block releases, it’s based on cooperation and encourages teamwork between data producers and consumers. I strongly believe that this collaborative approach is key to improving data quality and should be further integrated into the process. Well, I’m a big fan of drawing parallels… Notice actually that this collaborative aspect is something mentioned also as part of Lyft’s verity charter. The VerityUI provides a streamlined data discovery experience via the Verity Homepage. Our full-text search on the Check Definition Metadata lets users see all the checks currently being enforced and their Check Results. This has useful aggregations like owning team, table name, and tags. I’m not entirely clear on how data contract issues are shared between consumers and producers through the Verity platform UI, but I definitely recognize the significance of collaboration through the dashboards: The producer of a data product interface can confidently ensure that they’re not inadvertently causing downstream breakages that they hadn’t anticipated. The consumer of the interface can rest assured that their reliance on the interface is not and will not be compromised. While Verity is a remarkable tool for defining data quality checks, it’s unfortunately not open-source. Fortunately, there’s another open-source data quality framework called Soda Core that provides similar functionality. Soda Core is a free and open-source command-line tool and Python library that enables data engineers to test data quality. It utilizes user-defined input to generate SQL queries that run checks on datasets in a data source to find invalid, missing or unexpected data. When checks fail, they surface the data that you defined as “bad” in the check. https://medium.com/@seckindinc/soda-core-the-simplest-open-source-data-reliability-tool-f35ae5b576d9?source=post_page-----415baad6c95b--------------------------------&embedable=true During a scan of a dataset, Soda Core evaluates the predefined checks to identify invalid, missing, or unexpected data. Here’s the equivalent check using Soda.core syntax for the Verity DSL test that was previously defined. name: rider_events_session_id_check
source: hive
query: SELECT * FROM rider_events WHERE session_id IS NULL;
raise_alert: true
threshold: 10
action: slack
message: "There are more than 10 rows that are null for the 'session_id' property in the 'rider_events' table. Please investigate this issue." soda run --check checks/rider_events_session_id_check.yaml Soda Core is a powerful tool for ensuring the quality of your data. It can help you to identify and fix data problems early, before they can cause issues for your business. It’s worth noting that Soda Core can also facilitate data quality checks for streaming data by seamlessly integrating with Spark DataFrames. While Verity’s data quality checks for Hive are applied to static datasets, the checks for streaming data need to be more lightweight and efficient. Data would typically be processed in small batches of events, with very low latency, making them suitable for real-time checks and specific use cases like anomaly detection. Finally, let’s mention that there are other data validation tools available, such as DeeQu, Great Expectations, … As we wrap up, I hope you have a clearer understanding of the steps you can take to enhance your data quality journey. We’ve seen two distinct approaches to data quality improvement, each with its own strengths and methodologies. One focuses on increasing visibility and observability, motivating data producers to raise the quality bar. The other prioritizes elevating the quality standard through a testing and validation-first approach. Both are complementary. Verity is not merely a domain-specific language (DSL) for defining data checks; it’s a centralized platform that empowers data practitioners to collaborate effectively. This platform helps producers and consumers align on data quality expectations, including format, structure, and accuracy. Verity’s data contract management capabilities could (is?) further enhanced by integrating with a broader set of features, such as metadata management and discovery, to address more sophisticated data quality needs. Similar to Airbnb’s DQ score, Lyft’s Verity fosters a collaborative feedback loop between data producers and consumers. By incentivizing and empowering each team to take ownership of data quality, Verity cultivates a supportive environment where data quality continuously improves. , Found this article useful? Follow me on Linkedin , and ! Please 👏 this article to share it! Hackernoon Medium Also published . here

Advancing Data Quality: Exploring Data Contracts with Lyft

ডেটা গুণমান উন্নত করা: Lyft-এর সাথে ডেটা চুক্তিগুলি অন্বেষণ করা

Verbesserung der Datenqualität: Erkundung von Datenverträgen mit Lyft

Mejora de la calidad de los datos: exploración de los contratos de datos con Lyft

Améliorer la qualité des données : explorer les contrats de données avec Lyft

डेटा गुणवत्ता को आगे बढ़ाना: Lyft के साथ डेटा अनुबंध की खोज करना

データ品質の向上: Lyft とのデータ契約の検討

데이터 품질 향상: Lyft를 통한 데이터 계약 탐색

Avançando na qualidade dos dados: explorando contratos de dados com Lyft

Повышение качества данных: изучение контрактов на передачу данных с Lyft

Veri Kalitesini Geliştirme: Lyft ile Veri Sözleşmelerini Keşfetmek

Nâng cao chất lượng dữ liệu: Khám phá hợp đồng dữ liệu với Lyft

提高数据质量：与 Lyft 探索数据合同

Pencroft, Herbert, and Gideon Spilett remained silent in the midst of the darkness

When you’re getting started with Docker, it is important to understand what is a Dockerfile. You will work with Dockerfiles in every project you work on regardless of the operating system you’re using in your image. If you want to follow along with the demos in this blog post, you will need the food trucks’ repo from the Docker community repos. You can find it on their . GitHub https://youtu.be/DqyNssbqEaE?embedable=true A Dockerfile is similar to a batch script that contains all the commands to build a . The first line states that the base image is to begin with, and then follow the instructions to install required modules, copy files, and deploy the code which results in a working environment. Docker image Every Dockerfile must start with the FROM instruction. This tells Docker what base image to use as you build your final image. In this case, we are using the ubuntu standard OS image. You can find images for many different operating systems and development environments for any technology you’re using. The LABEL instruction adds metadata to an image. A LABEL is a key-value pair. You can add multiple labels to a Dockerfile to include any custom information you want. In this example, we are creating a key named “maintainer” and setting it equal to the text on the right. Labels are visible when you use a docker inspect command to view low-level information on Docker objects which is helpful to keep your projects organized. LABEL maintainer="Prakhar Srivastav <[email protected]>" The RUN instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile. All instructions are executed in order so you have to be careful to add your run instructions in the exact order they should be run. In this example, we are installing the required system-wide dependencies for Python and node as these are needed to run the food trucks website. # install system-wide deps for python and node
RUN apt-get -yqq update
RUN apt-get -yqq install python3-pip python3-dev curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash
RUN apt-get install -yq nodejs The ADD instruction copies new files, directories, or remote file URLs from the <src> listed on the left and adds them to the filesystem of the image at the <dest> path listed on the right. The destination path is located inside the image that is being built by the Dockerfile. # copy our application code
ADD flask-app /opt/flask-app Open a terminal, execute the command and then inspect the filesystem of your container to find the destination file path. You can even save this image as a reference to use for later if you want. docker-compose up, To save an image from a running container, get the container id by typing . Then, run a command using the container id, and then, a custom image name and version. docker ps docker commit That’s it! You can run the docker images command to confirm your new image has been created. docker commit [container id] [image name]:[image version] The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY, and ADD instructions that follow it in the Dockerfile. In this example, the node package manager commands are run in the working directory that has been defined. WORKDIR /opt/flask-app

# fetch app specific deps
RUN npm install
RUN npm run build
RUN pip3 install -r requirements.txt The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. EXPOSE does not make the ports of the container accessible to the host. # expose port
EXPOSE 5000 And finally, we have the CMD instruction. There can only be one CMD instruction in a Dockerfile. If you list more than one CMD, then only the last CMD will take effect. The main purpose of a CMD is to provide defaults for an executing container. In this example, we are running the app script for the website located in the root of the working directory and naming the app python3. # start app
CMD [ "python3", "./app.py" ] That’s it! Stop your running container by executing the command. Go back to the Dockerfile and uncomment everything using the shortcut . Save your work, and execute docker-compose down Ctrl+k+u docker-compose up. Open a browser, and navigate to localhost on port 5000. You should see the Food Trucks website! Previously published here

What Is a Dockerfile? A Quick Overview

The Brothers Karamazov by Fyodor Dostoyevsky is part of the HackerNoon Books Series. You can jump to any chapter in this book here . Kolya Krassotkin Chapter I. Kolya Krassotkin It was the beginning of November. There had been a hard frost, eleven degrees Réaumur, without snow, but a little dry snow had fallen on the frozen ground during the night, and a keen dry wind was lifting and blowing it along the dreary streets of our town, especially about the market‐place. It was a dull morning, but the snow had ceased. Not far from the market‐place, close to Plotnikov’s shop, there stood a small house, very clean both without and within. It belonged to Madame Krassotkin, the widow of a former provincial secretary, who had been dead for fourteen years. His widow, still a nice‐looking woman of thirty‐two, was living in her neat little house on her private means. She lived in respectable seclusion; she was of a soft but fairly cheerful disposition. She was about eighteen at the time of her husband’s death; she had been married only a year and had just borne him a son. From the day of his death she had devoted herself heart and soul to the bringing up of her precious treasure, her boy Kolya. Though she had loved him passionately those fourteen years, he had caused her far more suffering than happiness. She had been trembling and fainting with terror almost every day, afraid he would fall ill, would catch cold, do something naughty, climb on a chair and fall off it, and so on and so on. When Kolya began going to school, the mother devoted herself to studying all the sciences with him so as to help him, and go through his lessons with him. She hastened to make the acquaintance of the teachers and their wives, even made up to Kolya’s schoolfellows, and fawned upon them in the hope of thus saving Kolya from being teased, laughed at, or beaten by them. She went so far that the boys actually began to mock at him on her account and taunt him with being a “mother’s darling.” But the boy could take his own part. He was a resolute boy, “tremendously strong,” as was rumored in his class, and soon proved to be the fact; he was agile, strong‐willed, and of an audacious and enterprising temper. He was good at lessons, and there was a rumor in the school that he could beat the teacher, Dardanelov, at arithmetic and universal history. Though he looked down upon every one, he was a good comrade and not supercilious. He accepted his schoolfellows’ respect as his due, but was friendly with them. Above all, he knew where to draw the line. He could restrain himself on occasion, and in his relations with the teachers he never overstepped that last mystic limit beyond which a prank becomes an unpardonable breach of discipline. But he was as fond of mischief on every possible occasion as the smallest boy in the school, and not so much for the sake of mischief as for creating a sensation, inventing something, something effective and conspicuous. He was extremely vain. He knew how to make even his mother give way to him; he was almost despotic in his control of her. She gave way to him, oh, she had given way to him for years. The one thought unendurable to her was that her boy had no great love for her. She was always fancying that Kolya was “unfeeling” to her, and at times, dissolving into hysterical tears, she used to reproach him with his coldness. The boy disliked this, and the more demonstrations of feeling were demanded of him the more he seemed intentionally to avoid them. Yet it was not intentional on his part but instinctive—it was his character. His mother was mistaken; he was very fond of her. He only disliked “sheepish sentimentality,” as he expressed it in his schoolboy language. There was a bookcase in the house containing a few books that had been his father’s. Kolya was fond of reading, and had read several of them by himself. His mother did not mind that and only wondered sometimes at seeing the boy stand for hours by the bookcase poring over a book instead of going to play. And in that way Kolya read some things unsuitable for his age. Though the boy, as a rule, knew where to draw the line in his mischief, he had of late begun to play pranks that caused his mother serious alarm. It is true there was nothing vicious in what he did, but a wild mad recklessness. It happened that July, during the summer holidays, that the mother and son went to another district, forty‐five miles away, to spend a week with a distant relation, whose husband was an official at the railway station (the very station, the nearest one to our town, from which a month later Ivan Fyodorovitch Karamazov set off for Moscow). There Kolya began by carefully investigating every detail connected with the railways, knowing that he could impress his schoolfellows when he got home with his newly acquired knowledge. But there happened to be some other boys in the place with whom he soon made friends. Some of them were living at the station, others in the neighborhood; there were six or seven of them, all between twelve and fifteen, and two of them came from our town. The boys played together, and on the fourth or fifth day of Kolya’s stay at the station, a mad bet was made by the foolish boys. Kolya, who was almost the youngest of the party and rather looked down upon by the others in consequence, was moved by vanity or by reckless bravado to bet them two roubles that he would lie down between the rails at night when the eleven o’clock train was due, and would lie there without moving while the train rolled over him at full speed. It is true they made a preliminary investigation, from which it appeared that it was possible to lie so flat between the rails that the train could pass over without touching, but to lie there was no joke! Kolya maintained stoutly that he would. At first they laughed at him, called him a little liar, a braggart, but that only egged him on. What piqued him most was that these boys of fifteen turned up their noses at him too superciliously, and were at first disposed to treat him as “a small boy,” not fit to associate with them, and that was an unendurable insult. And so it was resolved to go in the evening, half a mile from the station, so that the train might have time to get up full speed after leaving the station. The boys assembled. It was a pitch‐dark night without a moon. At the time fixed, Kolya lay down between the rails. The five others who had taken the bet waited among the bushes below the embankment, their hearts beating with suspense, which was followed by alarm and remorse. At last they heard in the distance the rumble of the train leaving the station. Two red lights gleamed out of the darkness; the monster roared as it approached. “Run, run away from the rails,” the boys cried to Kolya from the bushes, breathless with terror. But it was too late: the train darted up and flew past. The boys rushed to Kolya. He lay without moving. They began pulling at him, lifting him up. He suddenly got up and walked away without a word. Then he explained that he had lain there as though he were insensible to frighten them, but the fact was that he really had lost consciousness, as he confessed long after to his mother. In this way his reputation as “a desperate character,” was established for ever. He returned home to the station as white as a sheet. Next day he had a slight attack of nervous fever, but he was in high spirits and well pleased with himself. The incident did not become known at once, but when they came back to the town it penetrated to the school and even reached the ears of the masters. But then Kolya’s mother hastened to entreat the masters on her boy’s behalf, and in the end Dardanelov, a respected and influential teacher, exerted himself in his favor, and the affair was ignored. Dardanelov was a middle‐aged bachelor, who had been passionately in love with Madame Krassotkin for many years past, and had once already, about a year previously, ventured, trembling with fear and the delicacy of his sentiments, to offer her most respectfully his hand in marriage. But she refused him resolutely, feeling that to accept him would be an act of treachery to her son, though Dardanelov had, to judge from certain mysterious symptoms, reason for believing that he was not an object of aversion to the charming but too chaste and tender‐hearted widow. Kolya’s mad prank seemed to have broken the ice, and Dardanelov was rewarded for his intercession by a suggestion of hope. The suggestion, it is true, was a faint one, but then Dardanelov was such a paragon of purity and delicacy that it was enough for the time being to make him perfectly happy. He was fond of the boy, though he would have felt it beneath him to try and win him over, and was severe and strict with him in class. Kolya, too, kept him at a respectful distance. He learned his lessons perfectly; he was second in his class, was reserved with Dardanelov, and the whole class firmly believed that Kolya was so good at universal history that he could “beat” even Dardanelov. Kolya did indeed ask him the question, “Who founded Troy?” to which Dardanelov had made a very vague reply, referring to the movements and migrations of races, to the remoteness of the period, to the mythical legends. But the question, “Who had founded Troy?” that is, what individuals, he could not answer, and even for some reason regarded the question as idle and frivolous. But the boys remained convinced that Dardanelov did not know who founded Troy. Kolya had read of the founders of Troy in Smaragdov, whose history was among the books in his father’s bookcase. In the end all the boys became interested in the question, who it was that had founded Troy, but Krassotkin would not tell his secret, and his reputation for knowledge remained unshaken. After the incident on the railway a certain change came over Kolya’s attitude to his mother. When Anna Fyodorovna (Madame Krassotkin) heard of her son’s exploit, she almost went out of her mind with horror. She had such terrible attacks of hysterics, lasting with intervals for several days, that Kolya, seriously alarmed at last, promised on his honor that such pranks should never be repeated. He swore on his knees before the holy image, and swore by the memory of his father, at Madame Krassotkin’s instance, and the “manly” Kolya burst into tears like a boy of six. And all that day the mother and son were constantly rushing into each other’s arms sobbing. Next day Kolya woke up as “unfeeling” as before, but he had become more silent, more modest, sterner, and more thoughtful. Six weeks later, it is true, he got into another scrape, which even brought his name to the ears of our Justice of the Peace, but it was a scrape of quite another kind, amusing, foolish, and he did not, as it turned out, take the leading part in it, but was only implicated in it. But of this later. His mother still fretted and trembled, but the more uneasy she became, the greater were the hopes of Dardanelov. It must be noted that Kolya understood and divined what was in Dardanelov’s heart and, of course, despised him profoundly for his “feelings”; he had in the past been so tactless as to show this contempt before his mother, hinting vaguely that he knew what Dardanelov was after. But from the time of the railway incident his behavior in this respect also was changed; he did not allow himself the remotest allusion to the subject and began to speak more respectfully of Dardanelov before his mother, which the sensitive woman at once appreciated with boundless gratitude. But at the slightest mention of Dardanelov by a visitor in Kolya’s presence, she would flush as pink as a rose. At such moments Kolya would either stare out of the window scowling, or would investigate the state of his boots, or would shout angrily for “Perezvon,” the big, shaggy, mangy dog, which he had picked up a month before, brought home, and kept for some reason secretly indoors, not showing him to any of his schoolfellows. He bullied him frightfully, teaching him all sorts of tricks, so that the poor dog howled for him whenever he was absent at school, and when he came in, whined with delight, rushed about as if he were crazy, begged, lay down on the ground pretending to be dead, and so on; in fact, showed all the tricks he had taught him, not at the word of command, but simply from the zeal of his excited and grateful heart. I have forgotten, by the way, to mention that Kolya Krassotkin was the boy stabbed with a penknife by the boy already known to the reader as the son of Captain Snegiryov. Ilusha had been defending his father when the schoolboys jeered at him, shouting the nickname “wisp of tow.” About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Fyodor Dostoyevsky (2009). The Brothers Karamazov. Urbana, Illinois: Project Gutenberg. Retrieved October https://www.gutenberg.org/cache/epub/28054/pg28054-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org , located at https://www.gutenberg.org/policy/license.html .

ANY.RUN

any.run

-- employees

Since 2015

COMPANY RANKING

ANY.RUN

EVERGREEN INDEX #113

Representing this company? Click here to claim and customize this page!

ANY.RUN WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

ANY.RUN

any.run

-- employees

Since 2015

COMPANY RANKING

ANY.RUN

EVERGREEN INDEX #113

RELATED COMPANIESRANK

STORIES ABOUT ANY.RUN

Representing this company? Click here to claim and customize this page!

ANY.RUN WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!