Datanomik

datanomik

Production systems often need to store sensitive data, including personally-identifiable information (PII). Developers often need their test systems to have data that is as close to that in the production systems as is reasonably possible. Whilst it was always best-practice, legal data protection regimes such as HIPAA, HITECH, CPRA and GDPR means it’s ever more important to ensure that any personal data remains only where it’s strictly needed, and is properly masked or anonymized when being transferred elsewhere. There are a number of different ways to bridge this gap, such as designing a strict separation between database tables which hold PII and those which don’t, allowing the PII tables to be skipped on export and replaced with synthetic data on the development systems. This approach can certainly work, but it relies on the system adhering to this design pattern, and the synthetic data being kept closely enough in step with the production equivalents to not cause problems. An alternative might be to generate a special kind of “cleansed” dump on bthe production system, with PII already masked or replaced with synthetic data, ready for developers to import, keeping the risk of any sensitive data ever leaving the production environment low. This is the approach takes. Datanymizer Fakers, anonymizers, and obfuscators — there are various free and open-source data anonymization tools that have been around for a long time and work pretty well, so why did we create a new one The one that supports globals, uniqueness constraints, inline rules, and other cool features. We had some particular requirements we wanted our tool to meet. We didn't want the anonymizer to take a "raw" dump and mutate it. Instead, we needed to provide an already anonymized dump, without access to real data. The configuration that determined how the real system data would be anonymized should have been kept separate from that data. We also wanted a tool that was flexible about how the anonymization itself takes place, ideally allowing the use of templates to populate field contents. Enter Datanymizer: your flexible privacy-preserving friend Datanymizer does all of these things: you define a configuration which specifies what to do (and not do), and it then dumps data directly from your database, applying the rules that you define, and it even integrates the Tera templating engine to enable complex values to be synthesized. The output is an anonymized SQL dump, written either to a file or directly to standard output, ready to be imported into a database using your normal tools. Getting started There are several ways to install pg_datanymizer. Choose a more convenient option for you: a pre-compiled binary, Homebrew / Linuxbrew, Docker. The README contains an example configuration that you can use as a starting point. Now you can invoke Datanymizer to generate a cleansed dump of your data. It creates a new dump file /tmp/dump.sql with a native SQL dump for PostgreSQL database. Table filters You can specify a list of tables which should never be included in a dump: for dumping only public.markets and public.users data. For ignoring those tables and dump data from others. You can also specify data and schema filters separately. Global variables You can specify global variables available from any template rule. Built-in rules Datanymizer includes built-in support (“rules”) for certain types of value, including a pipeline filter which allows multiple rules to be executed in sequence. Other filters include email, ip, words, first_name, last_name, city, phone, capitalize, template, digit, random_number, password, datetime and more. Uniqueness constraints Uniqueness is supported by the email, ip, phone, and random_number rules. Uniqueness is ensured by keeping track of values that have been generated where uniqueness is required, and re-generating any which are duplicates of those in the list. You can customize the number of attempts with try_count. This is an optional field, the default number of tries depends on the rule. Future development We plan to implement the following additional features soon: for example, if it is necessary to dump not all users but those matching specific criteria (e.g., 100 users, aged 27 years old or more, named Alexander), supporting arbitrary SQL queries for filtering. Pre-filtering: when you don’t need to anonymize existing data, but instead generate synthetic data based upon certain rules.RDBMS support Data generation: Datanymizer currently supports PostgreSQL databases, although MySQL (and so also MariaDB) support is planned. Contributions are of course very welcome! . Previously published at https://evrone.com/datanymizer. The author of the story is the Chief Editor at Evrone

We Open Sourced Datanymizer: in-Flight Template-Driven Data Anonymization Tool

121 Stories To Learn About Data Protection

Let's learn about via these 79 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Rust /Learn Repo 1. Top 10 Programming Languages for Web Devs to Learn During Quarantine If you're just sitting at home most nights, wondering what the hell you should be doing, rejoice! I've got a list of 10 of the hottest, coolest programming languages you can stuff in a single sack (or...blog post). 2. An Intro to Rust for JavaScript Developers Learn Rust in 5 minutes with your existing JavaScript knowledge. Here, I list differences Rust has compared to JS that is crucial to understanding it. 3. How to Build a To-Do List in Rust Using bb8 and Tokio_Postgres Good news! It’s easy to create a simple to-do list using bb8 and tokio_postgres for the data layer side. 4. How Rust and Elixir Read Ethereum and other EVM Smart Contracts: Functional Programming & Blockchain I am going to show the function of reading Ethereum Smart Contract by Elixir&Rust in this article 5. Debugging Rust Cargo Issues in Gitlab Straightforward moral of the story: always first check environment variables in CI settings. They can overwrite your configs and cause unexpected issues. 6. Let's Build a Telegram Bot in Rust. Part 1 Building telegram bot using Rust. 7. Raspberry Pi Wireless Thermostat - in Rust! I created a fully working wireless thermostat that runs on a Raspberry Pi, written in Rust, as a practical learning project. 8. Lets Graph Simple Moving Averages Using Rust Simple Moving Averages are calculated by getting the mean closing price over a period of time. Lets see on how we can program that idea into Rust. 9. A Comprehensive Guide for Handling Errors in Rust This article address how errors are handled with the Rust programming language. 10. Why Rust is Meant to Replace C The Rust programming language is an ambitious project of the Mozilla Foundation – a language that claims to be the next step in evolution of C and C++. Over the years of existence of these languages some of their basic flaws still haven’t been fixed, like segmentation errors, manual memory control, risks of memory leaks and unpredictable compiler behavior. Rust was created to solve these problems while improving security and performance along the way. 11. "Solidity is Not Ideal For Writing Foolproof Smart Contracts" - Patrick Dai This article discusses smart contracts, ethereum, blockchain, decentralization, and the state of DeFi. 12. A Brief Guide to Rust for Embedded Development Rust can be used for embedded development on RT-Thread operating system. 13. Rust Is On A Roll I plan to use Rust as a replacement for Python for writing automation, especially any complex-calculable tasks. Here are the reasons why. 14. Everybody Loves Rust: How Rust is Being Used in Production Who uses Rust, and what are the benefits of choosing this programming language for your stack? Find out the answer in stories from 9 successful companies. 15. We open sourced dotenv-linter: a lightning-fast linter for .env files, written in rust Find dotenv-linter on Github! 16. The Tools and Resources You Need to Become a Web3 Developer in 2022 In this article, I'm going to give you a roadmap and some of the best resources on the internet that will definitely help you get your first job in Web3. 17. After 20 Years Of Web Development, I Am Ready For Disruption I have been working in web technology for more than 20 years. I spent the first five years of my career as a full-stack developer. Back then, we used the term webmaster. I would set up servers (which included e-mail and FTP), order domain names, create databases and order SSL certificates. I would code the backend and the frontend, plus open up Photoshop to do a little bit of design and UX from time to time. I was the scrum master and the business analyst plus I would support customers and project manage their websites. These tasks and job titles were all just part of being a webmaster. You crafted the web by yourself, and it lived in a box in the room next door, occasionally you would have to go in and check on it, maybe even restart it. Later in my career, I saw an opportunity to specialise. Living and working in London, I saw the increasing need for the expert. Recruiters started to ring me, asking about specialist job titles. I very much enjoyed the visual element of working in the browser and JavaScript was growing in power and clearly here to stay. 18. Rust Programming Language: Everything You Should Know With this post, we aim at spreading the word about Rust language for web development by shedding light on the features and benefits. 19. The History of Hackathons: A Digital Evolution The term hackathon is a compound word, a fusion of hacking and marathon, where hack is used in the sense of exploratory programming. 20. An In-Depth Guide to IC Programming and Best Practices to Follow The integration of Internet Identity (II for short) needs to be distinguished between the development environment and the main network environment. 21. The Noonification: The Wall of Death (12/26/2022) 12/26/2022: Top 5 stories on the Hackernoon homepage! 22. Rust Coming to PS4 and Xbox: Rust Console Date Details Rust is finally on its way to Xbox and Playstation. Here's details on the Rust console date. 23. The Noonification: How to Move Away From Twitter (12/15/2022) 12/15/2022: Top 5 stories on the Hackernoon homepage! 24. How Rust Developers Can Succeed with the Aleo Ecosystem How the first builders of the Aleo ecosystem will be able to successfully execute their startups in the shortest amount of time? 25. Creating a P2P, Messaging App on Web3 Using Substrate & Ionic - Introducing Uke - BOG#002 Creating a peer to peer messaging app and protocol. 26. Implementing Multi-Threaded Shared Memory in Rust Hide the messy details of dealing with Mutexes by encapsulating them inside a struct. 27. Learn How to Build an HTTP API with Rust In this article, I'll write about the developer experience of developing the above application in Rust, compared to what I'm used to with Spring Boot. 28. Lilan Anjana Fernando is Excited About New And Upcoming Programming Languages Lilan Anjana Fernando from United States has been nominated for a 2020#Noonie in the Future Heroes and Technology categories. 29. Languages, Frameworks and Tools to Become a Robust Web3 Dev Web3, blockchain, NFTs?! It’s all getting a bit year 3000 out there… We’ve pulled this apart to better understand what the future of work looks like for develop 30. How to Build a Parser in Rust for Fun and Profit Learn how to build a  parser in Rust in an afternoon with memes 31. Running A Tech Community in Your Company: An Ex-Principal Engineer’s Guide 5 easy steps to create a sustainable tech community in your company that everyone loves. 32. Find and Fix Problems in .env Files With Dotenv-linter v2.2.0 We have developed dotenv-linter — a useful tool for checking .env files. It helps to find problems in .env files that you might miss at first, but which later may result in incorrect work of applications. 33. Building A Blockchain Social Media Module Using Substrate - WEB3 101 Everything you need to know to build your first custom blockchain using Substrate.  Learn exactly how to build, test, and deploy your own pallets on Substrate. 34. Cross Compiling Rust on MacOS to Run as a Unikernel Most programs on Linux are dynamically linked. So when you are creating a unikernel with OPS out of a linux application OPS goes out and finds all the libraries it's dynamically linked to and throws it onto the disk image. This works well if you are on linux - 99% of everything on linux is dynamically linked. 35. How to Move Away From Twitter Some people I know planned or already had moved away. The target seems to be Mastodon, an alternate decentralized Open Source using the ActivityPub protocol. 36. Second State Releases Scalable Privacy Service at Mozilla Open Labs Build a scalable and privacy-first Internet on Second State’s serverless infrastructure. Get the email newsletter on Rust, WebAssembly, serverless, blockchain, and AI. 37. How I Rebuilt the Classic Snake Game on the CENNZnet Blockchain What goes into building a blockchain using Rust? Our newest dev at the CENNZnet project provides an insight into his latest project. 38. The Goals and Principles of The DevX Initiative Sponsorship Program DevX Initiative Sponsorship program is here and focuses on the individuals who have worked passionately to make Rust what it is today. Read to know more. 39. How Using AWS Lambda with Rust Saved 3x the Cost Compared to Using Python or .NET When building for large-scale use, using Rust certainly feels worth it both from a cost perspective (factor x3 saved) and from a performance perspective. 40. Deep Vs. Shallow Copying Let's go back one moment. A little further down to our data structures. The dear heaps and stacks of them.Quite literally. 41. How to Go About Using WASM Semantic Search in Rust Introducing Voy, a portable WebAssembly semantic search engine with vector similarity search in Rust. 42. My Journey to Running a Workflow Automation Engine in a Browser About a week ago I got seriously bothered to figure out whether it is possible to run BPXE (a workflow automation engine I am developing) in a browser. [43. Rust Cross Compiling to the Raspberry Pi](https://hackernoon.com/building-a-wireless-thermostat-in-rust-for-raspberry-pi-part-2) Efficiently set up a development environment for cross-compiling Rust code for other platforms, including Raspberry Pi and AWS Lambda. 44. How to Create A Smart Contract to Mint NFTs on NEAR Protocol Blockchain technology is perpetually becoming a game-changer for most enthusiasts. It came to the limelight with the advent of cryptocurrency. 45. Using Go Defer and Rust Drop to Defer the Call to Rollback Forgetting to close things off in Go can potentially come back to bite you. The most basic and straightforward method is to call rollback or commit. 46. Including Files and Deeply Directories in Rust One of the things that's been criticized by newbies in Rust is the file include mechanism. About 2 days ago, I spent around 5 hours on how I was supposed to include a file that was referenced deep down in a directory tree. The docs didn't help, as they were simple structures. Here I'll show you how you can include your code in different parts of the application, even when the structure is complex, and save you hours on how to do this supposedly trivial task. 47. Learning Rust in Lockdown The COVID19 lockdown has been a pretty challenging time for everyone.  Staying sane without the social interactions we normally have can be difficult.  I took the time to learn a new programming language.  This served several purposes. 48. Building an API in Rust with Rocket.rs and Diesel.rs Following Clean Architecture In this guide I’m going to walk you through the process of building a simple CRUD API from scratch in Rust using Rocket.rs. 49. Don't Learn Alone, Take ChatGPT With You You may have noticed that, in the last few years, the Rust language has become a true Internet darling. 50. Rust’s Ownership and Borrowing Enforce Memory Safety How does Rust guarantee memory safety? 51. Improving React App Performance with SSR and Rust: Rust vs Node.js [Part III] In theory, a new technology or a modern approach should have a lot of benefits, but the main question that matters is that - what is actually practical advantages of that in numbers? 52. Using Cloud-ZK for Developing ZKP Acceleration in the Cloud Cloud-ZK: A Toolkit for developing ZKP acceleration in the cloud 53. 6 Best Rust Programming Books Ranked by Reviews This list will cover books that primarily focus on Rust. It also won’t take into account separate editions of books. 54. We Open Sourced Datanymizer: in-Flight Template-Driven Data Anonymization Tool Datanymizer is an open-source, GDPR-compliant, privacy-preserving data anonymization tool flexible about how the anonymization takes place 55. How To Improve React App Performance with SSR and Rust [Part II: Rust Web Server] Continue creating boosted Server Side Rendering implementation. In the first part we prepared ReactJS Application with SSR script that lets choose the best technologies for our Web Server. 56. Using Rust For Monitoring 30k API Calls Per Minute At Bearer, we are a polyglot engineering team. Both in spoken languages and programming languages. Our stack is made up of services written in Node.js, Ruby, Elixir, and a handful of others in addition to all the languages our agent library supports. Like most teams, we balance using the right tool for the job with using the right tool for the time. 57. Tutorial: Extending Sonic With Limit Orders Using ic-cron Library In this tutorial we build a crypto trading bot canister on Internet Computer (Dfinity) which can execute limit orders on AMM-based exchange - Sonic. 58. How to Insert Binary Tree in Rust Let's learn interesting Rust concepts like smart pointers and ownership with a classic data structure and algorithm together. 59. A Procedural Landscape Experiment Quick and dirty(?) procedural generation in 99 LoC of Rust 60. Retired Software Richard Eng Discusses Rust, Golang, Smalltalk, China and More! Richard Eng 2020 Noonie Nominee for Blockchain from India has been nominated for a 2020#Noonie in the Future Heroes and Technology categories.. The Noonies are Hacker Noon’s way of getting to know — from a community perspective —  what matters in tech today. So, we asked our Noonie Nominees to tell us. Here’s what Richard had to share. 61. Why Rust Is So Popular? If you are looking for some kind of metal panel business idea, allow me to be clear: the Rust I am referring to is a programming language. 62. How to Execute Background Tasks on Particular Weekdays with IC-Cron and Chrono In this tutorial we use ic-cron library in order to execute tasks on Internet Computer (Dfinity) at a specific days of week. 63. Parsing HTML with Rust: A Simple Tutorial Using Tokio, Reqwest, and Scraper Learn to parse websites with Rust using Tokio, Reqwest, Scraper. 64. Make Your Rust Code More Rustic By Breaking Some Rules Warp engineer and resident Rust expert @ChuckAPierce  writes about when to break the rules of Rust's infamous borrow checker. 65. Do Modern Technologies Like Serverless Really Offer Better Benefits? Remember mainframes? Serverless is pretty much that: we own the machine and you rent out time on our big iron. We went full circle on progress! 66. "Rust Isn’t Afraid to be Imperfect as Long as we Ship something Useful" - Steve Klabnik Steve Klabnik is a member of the Rust core team, an active open-source contributor, and author of The Rust Programming Language, Rails 4 in Action, and Designing Hypermedia APIs books. In 2012 and 2016, we invited Steve to speak at the RailsClub (now RubyRussia) conference. Since then, Steve has been working on Rust a lot, did a lot of interesting things and we realized that we should definitely interview him once again! 67. Static Full-Text Search in Next.js with WebAssembly, Rust, and Xor Filters I built a full-text search engine in Rust and WebAssembly. Fast, lightweight, and tons of fun to build! 68. How To Create Small, Fast And Cool Desktop Apps With Tauri And Ember.js I recently played around with Tauri, a toolkit for development desktop apps with web technologies. Here's how I got it working with an Ember.js application. 69. Faster than Linux FTL usually refers to "faster than light". A theoretical particle known as a tachyon that powers certain spaceships in the Star Trek universe keeps the plot going for decades through multiple series and and movie franchises. 70. Topological Sorting of a Directed Acyclic Graph in Rust Using DFS Topological sorting refers to the linear ordering of vertices in a graph such that for every directed edge ab, of vertex a to b, a,  always comes before b. 71. An Introduction to Building on the Solana Network This tutorial will teach you how to build and deploy your own programs on the Solana blockchain! 72. How I Used Rust + Lunatic to Build a TelNet Chat Server With WebAssembly This post describes how I leveraged Rust, WebAssembly and Lunatic to build a telnet chat server. 73. 10 Most Sought-After Programming Languages You Should Learn In 2021 If you are working on any technology and want to know which programming language is best for you then you can follow this article. 74. Variable Shadowing: "Let" Keyword Is Immutable But Not Constant Variable Let's take a look at some of the common pitfalls with the keywords let and mut. Then, we will learn how immutable != constant by using variable shadowing. 75. Interview: Flask Creator Armin Ronacher Introduction 76. Recap: 2020 WebAssembly (Wasm) Summit in Silicon Valley [incl. Links and Resources] The WebAssembly Summit was held in Silicon Valley, on the 10th February 2020. The venue, Google’s newest building at 1625 Plymouth St Mountain View CA. 77. Top 10 Programming Languages To Learn in 2022 The craze for learning programming languages is very high. These days people have one question in their mind and that is which programming language should one prefer in 20201 and why? 78. Building A Blockchain in Rust & Substrate: [A Step-by-Step Guide for Developers] A beginner-friendly tutorial on how to build hardcore blockchain infrastructure in Substrate, an open source framework. 79. Rusty Chains: A Basic Blockchain Implementation Written in Pure Rust A hands-on tutorial on blockchain basics, taxonomy and Rust. Thank you for checking out the 79 most read stories about Rust on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

79 Stories To Learn About Rust

On a scale from 1 to 10 how good are your data warehousing skills? Want to go above 7/10? This article is for you then. How good is your SQL? Want to get ready for a job interview asap? This blog post explains the most intricate data warehouse SQL techniques in detail. I will use BigQuery standard SQL dialect to scribble down a few thoughts on this topic. 1. Incremental tables and MERGE Updating table is important. It is important indeed. Ideal situation is when you have transactions that are a PRIMARY key, unique integers and auto increment. Table update in this case is simple: https://gist.github.com/mshakhomirov/18775cbbe8288af864ad79247c0de63d?embedable=true#file-1-1-sql That is not always the case when working with denormalized star-schema datasets in modern data warehouses. you might be tasked to create with SQL and/or incrementally update datasets with just a portion of data. might not exist but instead you will have to deal with data model where unique key depends on the latest (or timestamp) known. For example, in dataset depends on the latest known connection timestamp. In this case you would want to existing users and the new ones. sessions transaction_id transaction_id user_id last_online update insert MERGE and incremental updates You can use or you can split the operation into two actions. One to update existing records with new ones and one to insert completely new ones that don't exits (LEFT JOIN situation). MERGE is a statement that is generally used in relational databases. Google BigQuery MERGE Command is one of the Data Manipulation Language (DML) statements. It is often used to perform three main functions atomically in one single statement. These functions are UPDATE, INSERT, and DELETE. MERGE UPDATE or DELETE clause can be used when two or more data match. INSERT clause can be used when two or more data are different and do not match. The UPDATE or DELETE clause can also be used when the given data does not match the source. This means that the Google BigQuery MERGE Command enables you to merge Google BigQuery data by updating, inserting, and deleting data from your Google BigQuery tables. Consider this SQL: https://gist.github.com/mshakhomirov/5ad1a7518c54bc030d1c78b56fe3cf82?embedable=true#file-1-2-sql 2. Counting words Doing UNNEST() and check if the word you need is in the list you need migth be useful in many situation, i.e. data warehouse sentiment analysis: https://gist.github.com/mshakhomirov/694e040539b0d1b556f8e053d315a3bf?embedable=true#file-2-sql 3. Using IF() statement outside of the SELECT statement This gives us an opportunity to save some lines of code and be more eloquent code-wise. Normally you would want to put this into a sub-query, and add a filter in the clause but you can do instead: where this https://gist.github.com/mshakhomirov/933e6a358e49dcccd4e547a5509c8fda?embedable=true#file-3-sql Another example how to use it with tables. . This is bad example because since the matching table suffixes are probably determined dynamically (based on something in your table) you NOT partitioned Don't do this will be charged for a full table scan. https://gist.github.com/mshakhomirov/1c62d79cd9690140c569cd047b9d491f?embedable=true#file-3-2-sql You can also use it in clause and functions. HAVING AGGREGATE 4. Using GROUP BY ROLLUP The ROLLUP function is used to perform aggregation at multiple levels. This is useful when you have to work with dimension graphs. The following query returns the total credit spend per day by the transaction type (is_gift) specified in the clause, and it also shows the total spend for each day and the total spend in all the dates available. where https://gist.github.com/mshakhomirov/4cf738aaad967fe92c4fb7192874fadf?embedable=true#file-4-sql 5. Convert table to JSON Imagine you are required to convert your table into JSON object where each record is an element of nested array. This is where function becomes useful: to_json_string() https://gist.github.com/mshakhomirov/aac1f93312ae305ba80c915fe4a2a386?embedable=true#file-5-sql Then you can use it anywhere: dates, marketing funnels, indices, histogram graphs, etc. 6. Using PARTITION BY Given , and columns. For EACH date, how do you show the total revenue value for EACH customer while keeping all the rows? You can achieve this like so: user_id date total_cost https://gist.github.com/mshakhomirov/e4f11721eb5a3182150df08f25b70d64?embedable=true#file-6-sql 7. Moving average Very often BI developers are tasked to add a moving average to their reports and fantastic dashboards. This might be 7, 14, 30 day/month or even year MA line graph. So how do we do it? https://gist.github.com/mshakhomirov/ebf5488d0036bc9b84ae05889346d986?embedable=true#file-7-sql 8. Date arrays Becomes really handy when you work with or want to check some dataset for missing values, i.e. dates. BigQuery has a function called : user retention GENERATE_DATE_ARRAY https://gist.github.com/mshakhomirov/2ba5a67053f85794462dab98e56ad74d?embedable=true#file-8-sql 9. Row_number() This is useful to get something latest from your data, i.e. latest updated record, etc. or even to remove duplicates: https://gist.github.com/mshakhomirov/05d0c04c5975207d98552ffd436add8b?embedable=true#file-9-sql 10. NTILE() Another numbering function. Really useful to monitor things like if you have a mobile app. For example, I have my App connected to Firebase and when users I can see how long it took for them. Login duration in seconds login This function divides the rows into buckets based on row ordering and returns the 1-based bucket number that is assigned to each row. The number of rows in the buckets can differ by at most 1. The remainder values (the remainder of number of rows divided by buckets) are distributed one for each bucket, starting with bucket 1. If evaluates to NULL, 0 or negative, an error is provided. constant_integer_expression constant_integer_expression https://gist.github.com/mshakhomirov/16fe941aa8c4ed79e4aad8b7049b307a?embedable=true#file-10-sql 11. Rank / dense_rank They are also called functions. I tend to use as it doesn't skip the next available ranking whereas would. It returns consecutive rank values. You can use it with a partition which divides the results into distinct buckets. Rows in each partition receive the same ranks if they have the same values. numbering DENSE_RANK as default ranking function RANK Example: https://gist.github.com/mshakhomirov/459b68c5f3d1e8284c01e516db1d8dcb?embedable=true#file-11-1-sql Another example with product prices: https://gist.github.com/mshakhomirov/4c90a6fc8516d8264e172676a83a1048?embedable=true#file-11-2-sql 12. Pivot / unpivot Pivot changes rows to columns. It's all it does. Unpivot does the . opposite https://gist.github.com/mshakhomirov/f90b035ba259e672d4d51a669e0cd1fc?embedable=true#file-12-sql 13. First_value / last_value That's another useful function which helps to get a delta for each row against the first / last value in that particular partition. https://gist.github.com/mshakhomirov/ea4de9144b97bf8c196cab07609c309e?embedable=true#file-13-sql 14. Convert a table into Array of structs and pass them to UDF This is useful when you need to apply a user defined function (UDF) with some complex logic to each row or a table. You can always consider your table as an array of TYPE STRUCT objects and then pass each one of them to UDF. It depends on your logic. For example, I use it to calculate purchase expire times: https://gist.github.com/mshakhomirov/35d956fa9db86b12b44ab62c00f42a40?embedable=true#file-14-sql In a similar way you can create tables with no need to use . For example, I use it to mock some test data for unit tests. This way you can do it very fast just by using + + in your editor. UNION ALL Alt Shift Down https://gist.github.com/mshakhomirov/6ea226c1b5b789d4a31691ce065c20d7?embedable=true#file-14-2-sql 15. Creating event funnels using FOLLOWING AND UNBOUNDED FOLLOWING Good example might be marketing funnels. Your dataset might contain continiously repeating events of the same type but ideally you would want to chain each event with next one of a different type. This might be useful when you need to get a list of something, i.e. events, purchases, etc. in order to build a funnels dataset. Working with PARTITION BY it gives you the opportunity to group all the follwoing events no matter how many of them exists ineach partition. https://gist.github.com/mshakhomirov/05fd7d79d8acf3b173181a5d950ab6e7?embedable=true#file-15-sql 16. Regexp You would to use it if you need to extract something from unstructured data, i.e. fx rates, custom groupings, etc. Working with currency exchange rates using regexp Consider this example with exchange rates data: https://gist.github.com/mshakhomirov/9ca6e153da19c491034bd57995875308?embedable=true#file-16-1-sql Working with App versions using regexp Sometimes you might want to use to get , or versions for your app and a create a custom report: regexp major release mod https://gist.github.com/mshakhomirov/b1f442a296ffef52c7baa1245e1dc316?embedable=true#file-16-2-sql Conclusion SQL is a powerful tool that helps to manipulate data. Hopefuly these SQL use cases from digital marketing will be useful for you. It's a handy skill indeed and can help you with many projects. These SQL snippets made my life a lot easier and I use at work alomost every day. More, SQL and modern data warehouses are essentials tools for data science. Its robust dialect features allow to model and visualize data with ease. Because SQL is the language that data warehouses and business intelligence professionals use, it's an excellent selection if you want to share data with them. It is the most common way to communicate with almost every data warehouse / lake solution in the market. Originally published in by mydataschool.com datamike Mike is a passionate and digitally focussed individual with an abundance of drive and enthusiasm, loving the challenges the full mix of digital marketing throws up. Lives in the UK, completed MBA from Newcastle University in 2015.

16 SQL Techniques Every Beginner Needs to Know

16 এসকিউএল কৌশল প্রতিটি শিক্ষানবিস জানা প্রয়োজন

16 SQL-Techniken, die jeder Anfänger kennen muss

16 técnicas de SQL que todo principiante necesita saber

16 techniques SQL que tout débutant doit connaître

16 SQL तकनीकें प्रत्येक नौसिखिए को जानना आवश्यक है

すべての初心者が知っておくべき 16 の SQL テクニック

모든 초보자가 알아야 할 16가지 SQL 기술

16 técnicas de SQL que todo iniciante precisa saber

Yeni Başlayanların Bilmesi Gereken 16 SQL Tekniği

16 kỹ thuật SQL mà mọi người mới bắt đầu cần biết

每个初学者都需要了解的 16 种 SQL 技术

World Economic Forum | How much data is generated each day? According to the World Economic Forum, in 2020 the entire digital universe has reached 44 Zetabytes of Data. For perspective, that is 40 times more bytes than there are stars in the observable universe. And it is just at the beginning. The Internet of Things (IoT) and an explosion in mobile devices, smart connected devices from smartphones to smartwatches and autonomous vehicles as well as social networks and digital transformation is creating a world in which people can know anything, anywhere, anytime. This on-demand culture is truly mindboggling and insane. We live like Gods today compared to our ancestors. And the technology we possess is godlike. The world is changing so fast that people no longer understand the world we live in anymore and the world we are rapidly entering. Software is not just eating the world, it’s swallowing entire industries whole and creating different industries overnight. Understanding the accelerating rate of change itself in digital technologies and the information technology revolutions is SO Important. Because it has been, is and will continue to be responsible for rapid societal, political and economic change for better or worse. Today’s millennials and younger generations are used to technology and are digital natives. But even many of them can’t keep up. We are living in a fully hyperconnected and digital world and the implications of this change is MASSIVE. This is why industry experts don’t even know what the future looks like. Because things are changing so fast that there are terms being thrown around such as the 4th Industrial Revolution and the Technological Singularity. The 4th Industrial Revolution is basically the technological revolution with all the digital and exponential technologies that are transforming business, the economy and society rapidly. Industry 4.0 The Future is Now & The Singularity is Near The implications of billions of people having access to the world’s information at their fingertips is massive. The implications are multi-dimensional and the older ways of politics, society, economics stop functioning in the face of this type of exponential change. When things change so fast, so hard, it becomes impossible to figure out the rules and even to agree on how society or business or consumerism or entrepreneurship or regulation should work in the digital age. To understand exponential growth vs linear growth, please read Peter Diamandis’s tech blog posts on having an exponential mindset and understanding exponential growth in digital technologies, as well as Michael Simmons article on Ray Kurzweil’s explanations of exponential growth and a video of Moore’s Law Graphed over 5 decades. Peter Diamandis Tech Blog | Why an Exponential Mindset? Peter Diamandis Tech Blog | Proof of Exponential Tech Growth Google Director Of Engineering: This is how fast the world will change in ten years One thing is for sure, however. The hyperconnected and hyper-technological world that is being built by the second has MASSIVE implications and the future is being created faster than ever. Check out what Arm CEO Simon Segars had to say at Arm TechCon 2019 about 5G, the 5th Wave of Computing and the Trillion Device World. Simon Segars predicts that by 2035, 1 Trillion Smart Devices will be connected through 5G Networks and the IoT and Marcelo Claure, COO of SoftBank says that these hyperspeed networks and new computing world will create $11 Trillion of Economic Value. The applications of computing are significant in regards to the IoT (Internet of Things), 5G, Artificial Intelligence, Self-Driving Cars, Cloud, Data Centers, SaaS and digital transformation and digital disruption of every single industry. What happened with Amazon, Facebook, Google, Netflix, Microsoft, Oracle, Apple and other tech giant’s rise in the last few decades will pale in comparison to what happens in the next few decades. Entire markets, industries and companies will be built and wiped out OVERNIGHT. The competition has not even started yet. Because as we move from a scarcity-based economy to an abundance-based economy where we can have anything and everything we want on-demand for basically free, the transition has tremendous implications across the board. The rules of how EVERYTHING worked in the past now CEASE to function as the pace of technological change explodes at an exponential rate. Exponential Technologies and Our Technologically Disruptive and Digitally Transformed Future Dawn of a New Computing Era, The Fifth Wave of Computing, Technological Singularity, Nvidia’s Arm Acquisition Thematic Technology & Innovation Investing for the 2020s It’s not just that there are technologies that exist separately but that many exponential technologies are starting to converge and many societal, economic and political conversations are also starting to converge. Globalization, Technological Disruption and Capitalism are now on Steroids as our society ironically shifts to something completely unimaginable. However, because there are so many exponential technologies, and because the rate of change happening in innovation is itself accelerating so quickly, it is very hard if not impossible to predict the future and the near future. That’s why even futurists say that a combination of human intuition and creativity must be combined with data in order to generate better quality insights and make higher quality decisions. In a world that’s changing faster than ever it’s not enough to rely on one or the other. But when we combine humans and machines, we unlock tremendous synergies and potential. Furthermore, using the past is a terrible benchmark for trying to predict the future, in a world that is changing faster than ever, and so using foresight (which can be gained by using the combination of human intuition, creativity as well as data from algorithms and computers) to navigate the future is important to trying to not just anticipate outcomes, but to actively navigate and create better outcomes. It’s not just technologies, it’s also generational shifts in tastes (which are largely driven by new technologies - younger generations such as Millennials and Gen Z are used to on-demand networking, culture, economies, and society and are more comfortable doing things in the digital world - gathering data and making purchasing and other decisions). And it’s also changing expectations about corporate responsibilities, climate change, and the welfare of entire societies and not just those in the top 1%, especially in the post-covid world we are entering, where we have seen what happens when we don’t behave with ethics in terms of climate, or in terms of economic equality. This is even more true in 2021 and going into the next decade as Covid19 accelerated a bunch of previous trends that were already moving lightning fast in regards to the pace of change in technology and digital transformation and will have immense and increasingly powerful effects and disruptions on individuals, to communities, nations, corporations, governments and societies for better and for worse. The importance and role of big data, artificial intelligence, digital technologies, automation will only become increasingly important but how humans use foresight thinking, creativity, and intuition along with machine capabilities and data will be crucial in how the future is shaped as the world moves more and more online and our world, society and economy becomes increasingly networked. Datanami | 2021 Predictions: Data Science How the Coronavirus Pandemic is Turbocharging the Future of Work, Hyperautomation, Remote Working, Digital Transformation and Exponential Tech

How The 5th Wave Of Computing And IoT Are Changing Our Lives

History, Waves and Winters in AI

Stories, News & Mentions about DATANOMIK

Datanomik’s Stories on HackerNoon