Hyperquery

hyperquery

Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 5 Experiments Setup To evaluate MEME, several experiments were conducted in different configurations. First, there was a selection of four different malware detection solutions as targets to evade. Second, there was a comparison of MEME and four other evasion techniques on these four targets. 5.1 Targets The selection of targets was made to include three highly cited malware detection models together with a real implementation of a popular free Antivirus solution. 1. Ember. A LightGBM [20] model that was released as part of the Ember dataset [3] which was used for training that same model. The decision threshold was set to 0.8336, which corresponds to a 1% false positive rate (FPR) on the Ember 2018 test set. 2. Sorel-LGB. A LightGBM model that was distributed as part of the Sorel20M [15] dataset, which was used for training that same model. The decision threshold was set to 0.5, which corresponds to a 0.2% false positive rate (FPR) on the Sorel-20M test set. 3. Sorel-FFNN. A feed-forward neural network (FFNN) that was also released as part of the Sorel 20M dataset and was trained using the same data. The decision threshold was set to 0.5, which corresponds to 0.6% false positive rate (FPR) on the Sorel-20M test set. 4. Microsoft Defender. An antivirus product that comes pre-installed with the Windows operating system. According to [36], it is the most used free antivirus product for personal computers. All tests were performed using a virtual machine (VM) running an updated version of the product. The VM had no internet connectivity during the binary file scanning. 5.2 Datasets Our experiments required the use of the following datasets: 1. Ember 2018. A dataset that consists of features extracted from one million Windows Portable Executable (PE) files [3]. The dataset is split into training, testing, and ”unlabeled sets”. The training set consists of 300,000 clean samples, 300,000 malicious samples, and 200,000 “unlabeled” samples. The so-called unlabeled part of the dataset was truly unlabeled in the first version of the dataset, however, in the 2018 release, the authors provided an avclass label for all malicious samples, including those in the unlabeled set. Each sample has 2,381 static features related to byte and entropy histograms, PE header information, strings, imports, data directories, etc. 2. Sorel-20M. The Sorel dataset [15] was released in 2020 and contains the extracted features of 20 million binary files (malicious and benign). The feature set used was the same as the one from the Ember dataset. 3. Malware Binary Files. In addition to the Ember features used for training the surrogate, we also obtained 1,000 malicious binary files whose hashes were part of Ember 2018 and we use them for generating the evasive malware with all the methods. 4. Benign Binary Files. All methods require a benign set of data from where they extract benign strings, sections, and other elements that are used for the binary modifications. The same set of 100 benign binaries were used in all experiments. The files were obtained from a Windows 10 virtual machine after installing known benign software. MEME created two versions of the Daux dataset to train the surrogate models. For the Ember and AV surrogates, Daux contains the unlabeled part of the Ember dataset. For the Sorel surrogates, Daux contains 200,000 samples from the Sorel-20M validation set. These datasets were chosen to create the surrogate because they were not used to train the corresponding targets. During the evaluation, a subset of the Sorel-20M test set was used to evaluate the performance of the Sorel surrogate models and the Ember test set was used to evaluate the Ember and AV models. The 1,000 malware binaries were split into training and test sets with a 70-30% ratio using five different seeds. The test sets were used to test all the methods, while the 700 binaries in the training set were used to train each of the RL policies for PPO and MEME (these were the binaries to which modifier actions were applied). 5.3 Adversarial Malware Generation Comparison With regard to the generation of adversarial malware, MEME is compared with four algorithms in total. Two baseline reinforcement learning algorithms that use the Malware-Gym environment: a random agent and an agent that learns a policy using the vanilla Proximal Policy Optimization (PPO) algorithm [35], and two state-of-the-art (SOTA) algorithms: MAB [38] and GAMMA [9]. The two SOTA algorithms were selected based on the fact that they are relatively recently released and the fact that they seem to perform well in the malware evasion task. In addition, their source code is available. The detailed setup used for each of the algorithms, as well as any modifications, are presented below: 1. Random Agent The random agent is the simplest baseline used in our experiments. It uses the Malware-Gym environment and randomly samples the next modification action from the available action space. The agent is evaluated in the test environments using the 300 test malware samples. 2. PPO This is an agent that uses the PPO [35] algorithm as implemented in the Stable-Baselines3 software package. The agent was trained for 2,048 steps on the malware training set and evaluated on the malware test set. To select the hyper-parameters related to PPO training, we used the Tree-structured Parzen Estimator (TPE) method [4] as implemented in the software package Optuna [1]. The TPE algorithm was executed with the Ember dataset, but the settings performed well in the other targets. The tuned hyper-parameters were γ, the learning rate, the maximum gradient norm, the activation function, and the neural network size for the actor and critic models. The search space of each parameter and the final values are presented in the Appendix. 3. MAB RL algorithm that treats the evasive malware generation as a multiarmed bandit problem2. It operates in two stages: evasion and minimization. It samples the action space, which includes generic actions (similar to Malware-Gym) and any successful evasive actions along with the specific modifiers, e.g., appending a specific benign section. MAB directly manipulates each binary without generating a learned policy. Therefore, all experiments were conducted directly on the malicious binary test set. 4. GAMMA An algorithm that injects benign binary sections into malicious PE files while preserving their functionality. It modifies features like section count, byte histograms, and strings, leaving features related to, e.g., certificates and debugging data unaffected. By employing genetic algorithms, GAMMA searches for optimal benign sections to reduce the target model’s confidence by minimizing the content and location of injected sections. The attack is implemented in the SecML library3. Though effective, it has a significantly longer runtime than other tested methods. The attack uses a restricted set of 30 available benign sections and a population size of 20. The λ parameter, impacting the injected data size, was set to 10−6. GAMMA directly operates on each binary and does not generate a learned policy. Hence, all experiments were conducted on the malicious binary test set. 5.4 MEME Experimental Setup The initial training steps n in Algorithm 1 were set to 1,024, and the total number of loops k was two. For evaluation, the test set of 300 malware binaries was used. The surrogate training steps m were set to 2,048 (step 6 of Algorithm 1). MEME utilizes PPO for training and updating the policy (πθ). The PPO settings remained the same as in the baseline experiments, enabling a comparison of the impact of using a surrogate model for additional PPO training. In total, there were a total of 2,048 queries to the target and 4,096 training steps using the surrogate environment. The surrogate was always a LightGBM model. Surrogate training involved two datasets: Daux from an external dataset (e.g., Ember 2018 or Sorel-20M) and Dsur generated during lines 3 and 7 of Algorithm 1. These datasets were mixed with a ratio α, a hyperparameter for LGB surrogate tuning. Other hyperparameters, such as the number of boosting trees, learning rate, tree depth, minimum child samples, and feature fraction, were also tuned separately for each target using TPE and Optuna. Appendix A provides the detailed search space and selected values. Surrogate models were evaluated using the respective target test sets, and a decision threshold matching target FPR levels was calculated. For the AV target without a representative dataset, the surrogate’s decision threshold was set to 0.5. 5.5 General Experiment Settings All the algorithms were tested under a common set of constraints. The maximum allowed modifications to a binary file were set to 15 for the RL-based algorithms. Similarly, for GAMMA, the number of iterations was set to 15; for MAB, the number of ”pulls” was also 15. The second constraint was to set the maximum running time for all experiments to 4 hours. For MAB and GAMMA, this setting means that the algorithms must manage to handle as many malicious binaries as possible in that time. At the same time, for PPO and MEME, this time included both the policy training time and the evaluation time. For PPO and MEME, we set the query budget to 2,048. This budget does not include the final evaluation queries on the test set. MAB and GAMMA were not constrained in the total number of queries because it is not supported by the respective frameworks. Finally, all experiments were run with five different seeds. The random seeds controlled the split of the 1,000 malicious binaries into train and test, and therefore, all methods were tested in the same files. This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

MEME Algorithm's Impact on Adversarial Malware Generation and Model Evasion

It's been nearly 2 years since Seth Rosen's trashboard tweet, and yet the trashboard epidemic is still quite real. https://twitter.com/sethrosen/status/1407019976469397514?lang=en&embedable=true We still encounter companies that are curiously happy with having thousands of , their analysts left to spelunk deep caverns of ad hoc requests with naught but an dashboards IDE. There are good intentions all around, of course -- ad hoc work can be painful, and an automated solution sounds appealing. What's more, dashboarding is a safe bet - -we still live in a world where the dashboard is still accepted as The Default Tool For Analytics, after all. Let's talk about how we got here, why this needs to change, and where we should go from here. In short, in what follows I'll argue that: Decades of heritage have brought us to a weird place where dashboards have primacy. But dashboards are not always the best way to provide value to your org, especially after the first few. If you believe the two statements above, your beliefs, your processes, your tooling need to fundamentally change. The problem: dashboard primacy in a Pareto world Dashboards vs. analyses, by way of cooking So don't misunderstand - dashboards are fantastic, they're just blunt. They can serve endless needs, but in doing so, rarely do they serve particular needs perfectly. It's the for data: the more data you expose, the less sharp your insight is. Heisenberg uncertainty principle Dashboards, by nature, prioritize data exposure. They offer data by itself, and interpretation is left to the consumer. A certain number of core dashboards are certainly required, of course. Every business unit has core metrics and events that it tracks, and these need to be surfaced somewhere -- the dashboard here acts as the heartbeat of the business. But beyond this, value greatly diminishes after each . nth dashboard Analyses, on the other hand, are sharp. They expose interpretation, enabling more precise relevance to business problems. If dashboards are like salad bars, analyses are made-to-order meals. And just as specific meals fit cravings better, so too do will analyses always fit specific business needs better. On the other hand, incremental salad bar items (more dashboards) have diminishing returns -- they'll never quite satisfy a particular craving, but they'll crudely solve most hunger-related problems. You can benefit greatly from having a few, but you don't need to continue building them endlessly. Then why do dashboards still have primacy? Although I've found folks generally agree with that sentiment, we live in a peculiar time in history where dashboarding still retains its primacy. It made sense when data was slow, data was hard to handle, and dashboards were cumbersome to make -- core dashboard maintenance was the entirety of an analyst's job. But as the warehouse sped up, as analytics needs changed, and the returns on additional dashboards diminished, our mindset never adjusted accordingly. We've somehow become trapped by some perversion of the sunk cost fallacy: we'd allocated all incremental resources to dashboarding, and so we started to believe dashboards were of the utmost importance. I'll concede that there's an inherent attractiveness to dashboards: dashboards allow us to build with high coverage in mind. They sell the dream that our ad hoc request volume will drop to 0 one day. One day, given enough dashboards, we'll have addressed all questions and all follow-up questions that are to come. But unfortunately, running a business is rarely so formulaic. Power law distributions proliferate. begets pockets of value and worthlessness, change and stagnation, accumulation and void. Not all things are equal. It's the world of the Pareto principle, where avalanches drive change. Where 20% of the effort drives 80% of the outcomes. Preferential attachment And in this world, analyses are no exception. If you were to rank order all the work you've done by impact, I suspect you'll find most of your value add came from just a few key analyses. And in this world, the dashboard flounders -- what we gain in exhaustiveness, we rob from the avalanche. Where do we go from here? I've long been a proponent of . But I seldom do analytics work these days, and when I recently got my first ad hoc request in months, I'll admit I jumped directly into the data. I had a strong urge to answer the question as quickly as possible- - I viewed the work as a hindrance, not an opportunity. It was an attempt to , but in doing so, I fell from Co-founder to human API. I fell victim to the bad loop ruining analytics, even as a self-proclaimed curate of this borderline analytics religion. elevating the role of the analyst diminish the work required of me "Everything is a self portrait. Everything is a diary. When you care about something it becomes fairly obvious to everyone else. When you care, your devotion or discipline is directional. Most people can intuitively feel it." Chuck Palahniuk Ergo: we need to change our defaults -- - or risk falling back into the same old habits. our beliefs, our processes, our tools- Belief: value beyond data pulls. The first step in driving any sort of behavior change is to merely internalize that there's a clear reason why you're doing what you're doing. We need to believe that our value add extends beyond getting data. By all means, if you believe that your organization's primary need is a core set of dashboards, then set those up and focus your efforts on that. But if you have an inkling that you can provide leverage outside of this - by helping folks make better decisions, or by even keeping others intellectually honest in the conversions- - then internalize this and proceed under the assumption that it is true. Processes: analysis-by-default, not dashboard-by-default. To some extent, I believe we all want to have more impactful roles, drive more business value, and participate in higher levels of strategic problem-solving. But beyond simply the remaining hurdles are still substantial. In my estimation, the most debilitating hindrance henceforth is that it's far too easy to fall into bad habits. We need to build systems that make it easy to do the right things and harder to do the wrong ones. recognizing that "Goals are good for setting a direction, but systems are best for making progress." James Clear, Atomic Habits I'd start by considering a move to an analysis-by-default world, particularly if you've already invested heavily into self-service systems and a set of core dashboards. If we spend our IC time trying to (analysis), rather than trying to (dashboards), it follows then that our work will end up being more useful, by definition. maximize the usefulness of our work maximize the scalability of our work Beyond that, build templates (here's ), and standards around how work should be done. Write up hardened operating principles for your team. Hell, set borderline toxic policies to enforce better behavior: analyses don't count unless you put them in our centralized, agreed-upon place, for instance. Adopt rituals that reinforce good behavior: have daily stand-ups where you present this work; bake ritual compliance into your performance reviews. Do whatever it takes to make sure folks do the things that you know will make them better. mine Tools: notebook? At minimum, an analysis-first workflow If we believe in analysis by default, we need a set of tools that inverts common working patterns and pushes us to build analyses first, then dashboards, not vice versa. Certainly, I'm biased - this has been the philosophy behind building Hyperquery- - but even if you're not sold on our vision, even ensuring you don't jump into Tableau for every analysis is a worthy first step. At a minimum, narrative-first is a must. By simply forcing yourself to default to words as deliverables, rather than data, you're already breaking out of a ruinous habit. Narrative-first forces alignment, relevance, and value over immediacy. Narrative-first elevates the conversation beyond technical contributions to strategic contributions. Narrative-first nudges your stakeholders that your value lies in narratives, not code. Final comments Analytics is ripe for change. Even if you don't buy the arguments I've laid out, it's undeniable that our industry has changed in the last decade. And with new cars come new roads, and new laws. At a minimum, I hope this post pushes you to think about what that new world ought to look like. astronomically This post was originally published on . 🙂 Win With Data 👋 Hello! I’m Robert, CPO of and a former data scientist + analyst. Find me on or — I’m always happy to chat. Hyperquery LinkedIn Twitter

Moving Beyond Dashboards: Rethinking Analytics in the Era of Ad Hoc Requests

Last year, Emilie Schario and Taylor Murphy proposed this wonderful idea of . The key premise of the article was this: product teams have a lot of great practices that data teams would benefit from adopting. But somewhere along the way, we lost sight of this point and happily replaced it with strawmen: , , or painstakingly defining what means in service of hardening . All of these are certainly worth consideration, but they’re concerned more with the proper handling of data and data assets rather than the data teams that actually drive the impact. “running your data team like a product team” maintaining production-grade systems for our data assets building more data products production data contracts The central idea here was never to quibble over the definition and boundaries of the “Data Product” or set SLAs for data producers, but rather to compel us to reconsider how data teams operate by using product teams as a model. I’d like to spend some time discussing precisely how to run your data team like a product team. Two core ideas: user-centricity and proactivity There are two core principles that product teams embody — user-centricity and proactivity. Let’s discuss each in turn. User-centricity The best product teams are user-centric. They speak with their customers regularly and let direct user feedback directly influence their roadmap. This flywheel is the lifeblood of any good product — it ensures that they are not just shipping features but solving problems. Data teams need to operate the same way. We’ve been too enamored by how technically interesting our work can be, and we’ve forgotten that we are not standalone havens for scientific/engineering pursuits — we are a business unit hired to provide business value. And if we, like product teams, do not solve business problems with data, our metaphorical “data product” (all data work we do) is failing. This does not mean responding reactively to ad hoc requests. Nor does this mean completely shunning scientific endeavors. It simply means staying attuned to what the business needs and pursuing opportunities to that end. While Taylor and Emilie suggest your coworkers are your customers, I don’t think this is quite enough — the business is your customer. You need to know it, understand it, and orient everything you do around it. Proactivity Secondly, the best product teams have proactive processes set up to support the product-building process. They provide themselves with intentional space to set the vision, to brainstorm, to pursue passion projects that lie outside the scope of directly fielding customer requests. Analytics teams, on the other hand, seldom operate like this. At a minimum, we should spend some time exploring data outside of inbound requests. And at the team level, we should be on the lookout for patterns so we can intentionally craft our roadmap and do high-leverage work. That said, reactive work certainly still has its place — analysts are the primary means through which the business can explore data, so we will often find ourselves necessarily in a supporting role. But the key is constantly pushing to understand the context behind this work, and letting this context motivate strategic, high-leverage projects. But how do you get started? The original LO article has some great organization-level suggestions to make this all possible: have sufficient headcount so you have enough bandwidth to be strategic; gather a multidisciplinary team to draw inspiration from. To add onto this, here are a couple of concrete -level changes that you can immediately make:‍ process 1. Establish a process for knowledge consolidation.‍ Putting your team’s work in one place is a prerequisite to being user-centric. For your work to be user-centric, you need a view of all the work you’re being asked to do. Organizing your work in a shared space can enable pattern-matching across your team’s work — equivalent to a product team studying UX research findings before brainstorming. This will be your biggest hurdle because non-compliance is a huge issue. People strive to stick with the status quo, and too often I’ve seen documentation/knowledge-sharing initiatives fail. Publishing work in a wiki workspace like Notion, Confluence, or Dropbox paper (and for an analytics-specific solution, hyperquery) can break down this barrier. The key components here to ensure widespread adoption: As tempting as it might be to use git and a set up a peer review process, layers of process won’t ensure rigor — they’ll reduce adoption. Do something easy, light, and focus on ensuring your team will put their work in the tool. Lower the friction to usage. . Again, use a tool like hyperquery, Notion, or Confluence, with a wiki structure that’ll enable your team to establish practices around not only centralization, but . Agree on reasonable, functional categories, and create a central “Start here” doc that onboards new analysts onto your practices. Set up organizational scaffolding organization 2. Understand the business needs intimately. We’re not just technical workers. We are a bridge between data and the rest of the business. And if we just immerse ourselves in the data — which is just one side of this conversation — we won’t be nearly as effective as we should be. We pride ourselves on our technical prowess, but we are only effective insofar as we know why we’re doing our work. Without keen business acumen, we’ll write up pointless analysis after analysis until we’re moved to , our interactions relegated to data pulls. Storage B What this looks like, practically speaking: Before you dumpster dive into SQL, make sure you’re aligned with your stakeholders on business objectives. Write this down, agree on an approach, and only then, do the work. This sets a precedent that your involvement in the decision-making process is not limited to the technical work — at minimum, you’ll be at least seen as a translator, and at best, a thought partner. . Always ask why It sounds obvious, but too often I’ve seen analysts and data scientists put blinders on to the business and immerse themselves in the technical aspects of their work. This behavior is a harbinger of a truly dysfunctional, low-impact analytics org. Higher impact does not generally come from better analyses, but from data-driven impact at higher levels of strategic execution. Care about the business. Conclusion The nature of data analysis has drastically evolved in the last decade. We have access to more data, more compute power, and more tools than ever before. But we haven’t yet figured out we should operate within an organization with our newfound powers. It can be helpful here to carry successful practices over from other domains. From product teams, in particular, a focus on user-centricity and proactivity can mean the difference between a help desk analytics team and one that truly drives strategy. And user-centricity and proactivity come from an acute awareness of business needs and better knowledge-sharing practices. Original blog post on hyperquery. published

How Data Teams Can Benefit From Running Like a Product Team

Cómo se pueden beneficiar los equipos de datos al funcionar como un equipo de producto

Comment les équipes de données peuvent bénéficier d'un fonctionnement comme une équipe produit

उत्पाद टीम की तरह चलने से डेटा टीमें कैसे लाभ उठा सकती हैं

データ チームが製品チームのように機能することで得られるメリット

Como as equipes de dados podem se beneficiar da execução como uma equipe de produto

Nhóm dữ liệu có thể thu lợi như thế nào khi hoạt động giống như một nhóm sản phẩm

数据团队如何从像产品团队一样运行中受益

Improving your experience with Criteria API using Builder pattern and JPA Static Metamodel - Part I

How To Build A Time-Traveling Contacts App with SirixDB

Aggregating into data lakes is the solution of today — but are Federated Sources the solution of tomorrow? TL;DR are considered to be a popular solution for uniting insights from ALL the organization’s data sources. Data lakes An emerging alternative is the approach, which generates the same output with a fraction of the effort and cost — without the data duplication. In this post, we’ll explore the advantages and disadvantages of both options and where it may make sense for your team. federated data sources A perfect database. What If? Imagine the perfect database. The best aspects of all databases types, combined into one — without limitations: Unlimited data storage and processing you get with Hadoop, the fast query execution enjoyed with Redis or DynamoDB; lightning-fast CRUD (create, read, update delete); RDBMS capabilities like Joins, Unions and aggregation; support for structured unstructured data; querying as if you’re using a data warehouse; with permissions configurable per table, column and row. and I’m not sure that even if such a super computing database existed, all of these capabilities could logically reside in one database. Anyway, it’ll be awhile before we get there. Until then, organizations are using multiple databases with different capabilities for different use cases. But that comes with it’s own issues. The problem Two main issues arise from juggling data in multiple databases and database types. Because each database uses a different API/language, having different database types requires having experts at hand for each DB: IT/admin, DevOps, Application Developers. Backups and maintenance are different for each DB as well. More importantly is that the main resource of the organization — — is distributed, duplicated, and cannot be aggregated for global insights in a clean, unified manner. data The data lake solution The first solution, which has become a common buzzword over the past few years, is to build a . The rationale for having a data lake is to bring all data into one place, where it can be queried using a single query engine. data lake However, the process to the lake requires many ETL (Extract, Transform and Load) processes using tools like Airflow, AWS Glue, Google Cloud Dataflow, among others. During this process, data is often duplicated and not updated as frequently as needed. Every organization has its data graveyards — huge stores of data that no one knows about or doesn’t have the courage to delete. With data lakes, the graveyard’s size will multiply. get it into Lastly, ETL processes can raise security and regulation concerns, they have high maintenance and costs associated with them, and current lake solutions are provider-locked, allowing you access to data for of their services only. some own The Federation solution brings a new approach to the table. Each database processes and storing its own data as they were meant to. Data is not transferred out. Instead, there is one engine that can query multiple types of databases and seamlessly merge the results. Federated Sources What are the important features of an engine capable of querying federated sources? This engine must have three important characteristics: One common API or language to query the database (SQL?) A mechanism that can unify all the data from disparate sources so it can be queried and aggregated. The security, governance, and auditing controls one has with a data lake solution. Two important notes about federation: Performance isn’t the main benefit of this approach. If your output requires specific capabilities, load the data into the relevant data engine (BigQuery, DynamoDB, etc.) and use it there. But if you need to get business logic from multiple database types, federation will do the job for you.Business logic output will always be relatively small. For example, the DWH will do what it does best and respond with the aggregated output to this new engine, which will be able to join/merge/union the results with other database outputs in order to help with important business insights. This federation engine will also have to have integrations to each database which implements each API, translate data formats so they can be merged/joined with other results, and expose one common (SQL) syntax to query all these databases. Who is investing in federated solutions? is investing some effort in the direction of federated sources, with a caveat — Google-only services (and only three, as of now). They sometimes refer to it as and sometimes “external data sources”. In their case, acts as the federation engine and SQL is the common syntax used. You can query external resources: , and The data sources can be or file types like and Google BigQuery federation, BigQuery Cloud BigTable Cloud Storage, Google Drive. MySQL tables ORC Parquet. also invests in the federation approach with a new project called ”. It enables unified query access across multiple data stores and data formats by separating the syntax and semantics of a query from the underlying format of the data or the data store that is being accessed — and it is open source. Amazon AWS “ PartiQL Wrapping up While federated querying has its obvious advantages over data lakes, the solutions out there today aren’t complete. Once we can use SQL syntax to query data source while being able to Join, Union, and aggregate the outputs in a single command line, we’ll have a complete solution. any then It’ll look something like this: * bigqueryTable bqt dynamoTable dt facebookApi fbt lt …. … select from as join as join as join logfiles as where group by At , we believe that this “complete” engine is possible, and are building for it — we call it “ ”. superQuery Data Alloy

Going From Data Lakes to Oceans

Photo by on Markus Spiske Unsplash The Problem I have a serverless process that runs in a row — one Lambda runs and before it ends, it invokes another Lambda. Due to rate limits and other errors, I needed to know if the invocation was successful, but I didn’t want to wait until the new Lambda ended. This was challenging. The other annoying thing was that the number of running Lambdas was increasing out of control, even though I should have one Lambda alive at a time (maybe two, because I start a new Lambda and only then close the current one — an overlap of few milliseconds). The Solution There are two types of lambda invocations: Sync (“RequestResponse”) and Async (“Event”). Initially, I started with the option as I didn’t want to wait for the new Lambda to end. It turns out that adds this invocation request to a queue and you have no idea whether it was invoked successfully. Furthermore, if the invocation failed then AWS may re-run it again.. Async AWS My next attempt was with the Sync invocation. In order to know if the Lambda invocation succeeded, I needed to wait for the new Lambda to end. So I used a little trick: Invoke Sync Lambda Wait for a second After one second, of three things happened: I got a successful invocation from AWS within a second (meaning that the new lambda ran for less than a second — not likely to happen) I got an error from AWS A timeout occurred, so I assume that invocation was successful. The new Lambda is running, but I no longer wait for it. Important notes from my experience: If a timeout occurred, I assumed that invoke was successful and continued with my code at some point in time the new function will end and reply with a real success or failure response. You must prepare your code to handle it. You may get two flows instead of one — watch out! BUT Same scenario as the previous, but this time the first Lambda is over before the invoked one replies with an answer — AWS will consider the invocation as failed and re-run the Lambda! Lambda socket timeout: If Lambda A invokes a new Lambda B in a synchronous way ( ), it will wait for it to end according to the Lambda socket timeout — even if your code is over and you replied with 200 or 500 response! “RequestResponse” (lambda.Config.httpOptions = {timeout: 1500};) You can force Lambda to end and not wait for the socket timeout with: . = ;” — note that you will get an error message when the invoked lambda ends (B) and it will re-invoke in this case too. “context callbackWaitsForEmptyEventLoop false Conclusion I wanted to know whether Lambda invoked successfully, but I could not find a straightforward solution. You need to use a Sync invocation and “hack” the system. Be aware of additional invocations when AWS considers your Lambda as failed just because you didn’t wait for its response. Your Lambda function will be invoked multiple times — your code should be idempotent — code logic should know how to handle multiple invocation with the same input (event). Other solutions that not discussed in this post: Lambda Dead Letter Queue — where a failed lambda is inserted to a queue that needs to be managed, the retry invocation on error is still happening in this case. Use an external DB to manage your invocation. For example, increment a dynamo counter when Lambda starts and decrease it when it ends. Amos Shahar is Director of DevOps & IT at superQuery . Via our SQL IDE for Google BigQuery, superQuery uses AI and smart caching to enable data teams to run the most efficient and cost-effective queries on Google BigQuery.

Stories, News & Mentions about HYPERQUERY

Hyperquery’s Stories on HackerNoon