No data, no AI: Data is essential to solving our toughest challenges in business, science, and society

Datarade

By: Jon Keegan and Alfred Ng A huge but little-known industry has cropped up around monetizing people’s movements. Companies that you likely have never heard of are hawking access to the location history on your mobile phone. An estimated , the location data industry has many players: collectors, aggregators, marketplaces, and location intelligence firms, all of which boast about the scale and precision of the data that they’ve amassed. $12 billion market Location firm Near describes itself as “The World's Largest Dataset of People's Behavior in the Real-World,” with data representing “1.6B people across 44 countries.” Mobilewalla boasts “40+ Countries, 1.9B+ Devices, 50B Mobile Signals Daily, 5+ Years of Data.” X-Mode’s website claims its data covers “25%+ of the Adult U.S. population monthly.” In an effort to shed light on this little-monitored industry, The Markup has identified 47 companies that harvest, sell, or trade in mobile phone location data. While hardly comprehensive, the list begins to paint a picture of the interconnected players that do everything from providing code to app developers to monetize user data to offering analytics from “ ” and on hundreds of millions of people. Six companies claimed more than a billion devices in their data, and at least four claimed their data was the “most accurate” in the industry. 1.9 billion devices access to datasets “There isn't a lot of transparency and there is a really, really complex shadowy web of interactions between these companies that’s hard to untangle,” Justin Sherman, a cyber policy fellow at the Duke Tech Policy Lab, said. “They operate on the fact that the general public and people in Washington and other regulatory centers aren't paying attention to what they're doing.” Occasionally, stories illuminate just how invasive this industry can be. In 2020, Motherboard reported that a company that collects location data through apps, was collecting data from Muslim prayer apps and selling it to military contractors. X-Mode, The Wall Street Journal also reported in 2020 that for immigration enforcement. Venntel, a location data provider, was selling location data to federal agencies A Catholic news outlet also </a> to out a priest who had frequented gay bars, though it’s still unknown what company sold that information. used location data from a data vendor Many firms promise that privacy is at the center of their businesses and that they’re careful to never sell information that can be traced back to a person. But researchers studying anonymized location data just how misleading that claim can be. have shown The truth is, it’s hard to know all the ways in which your movements are being tracked and traded. Companies often reveal little about what apps serve as the sources of data they collect, what exactly that data consists of, and how far it travels. To piece together a picture of the ecosystem, The Markup reviewed the websites and marketing language of each of the 47 companies we identified as operating in the location data industry, as well as any information they revealed about how the data got to them. . See our methodology here How the Data Leaves Your Phone Most times, the location data pipeline starts off in your hands, when an app sends a notification asking for permission to access your location data. Apps have all kinds of reasons for using your location. Map apps need to know where you are in order to give you directions to where you’re going. A weather, waves, or wind app checks your location to give you relevant meteorological information. A video streaming app checks where you are to ensure you’re in a country where it’s licensed to stream certain shows. But unbeknownst to most users, some of those apps sell or share location data about their users with companies that analyze the data and sell their insights, like Advan Research. Other companies, like Adsquare, buy or obtain location data from apps for the purpose of aggregating it with other data sources. Companies like real estate firms, hedge funds and retail businesses might then turn and use the data for their own advertising, analytics, investment strategy, or marketing purposes. Serge Egelman, a researcher at UC Berkeley’s ​​International Computer Science Institute and CTO of AppCensus, who has , said it’s hard to tell which apps on your phone simply use the data for their own functional purposes and which ones release your data into the economic ether. researched sensitive data permissions on mobile apps “When the app asks for location, in the moment, because maybe you click the button to find stuff near you and you get a permission dialog, you might reasonably infer that ‘Oh, that's to service that request to provide that functionality,’ but there's no guarantee of that,” Egelman said. “And there's certainly usually never a disclosure that says that the data is going to be limited to that purpose.” Companies that trade in this data are reluctant to share which apps they get data from. The Markup asked spokespeople from . all the companies on our list</a> where they get the location data they obtain Companies like Adsquare and Cuebiq told The Markup that they don’t publicly disclose what apps they get location data from to keep a competitive advantage but maintained that their process of obtaining location data was transparent and with clear consent from app users. “It is all extremely transparent,” said Bill Daddi, a spokesperson for Cuebiq. He added that consumers must know what the apps are doing with their data because so few consent to share it. “The opt-in rates clearly confirm that the users are fully aware of what is happening because the opt-in rates can be as low as less than 20%, depending on the app,” Daddi said in an email. Yiannis Tsiounis, the CEO of the location analytics firm Advan Research, said his company buys from location data aggregators, who collect the data from thousands of apps—but would not say which ones. Tsiounis said the apps he works with do explicitly say that they share location data with third parties somewhere in the privacy policies, though he acknowledged that most people don’t read privacy policies. “There’s only so much you can squeeze into the notification message. You get one line, right? So you can’t say all of that in the notification message,” Tsiounis said. “You only get to explain to the user, ‘I need your location data for X, Y, and Z.’ What you have to do is, there has to be a link to the privacy policy.” Only one company spokesperson, Foursquare’s Ashley Dawkins, actually named any specific apps—Foursquare's own products, like Swarm, CityGuide, and Rewards—as sources for its location data trove. But Foursquare also produces a free software development kit (SDK)—a set of prebuilt tools developers can use in their own apps—that can potentially track location through any app that uses it. is used in apps like , a service that compares prices at nearby gas stations, , a shopping app for coupons, and , another location-based discount app. Foursquare’s Pilgrim SDK GasBuddy Flipp Checkout 51 GasBuddy, Flipp, and Checkout 51 didn’t respond to requests for comment. A search on l a site that analyzes and tracks SDKs in apps, found Foursquare’s Pilgrim SDK in 26 Android apps. Mighty Signa While not every app with Foursquare’s SDK sends location data back to the company, the privacy policies for Flipp, Checkout 51, and GasBuddy all disclose that they share location data with the company. Foursquare’s method of obtaining location data through an embedded SDK is a common practice. Of the 47 companies that The Markup identified, 12 of them advertised SDKs to app developers that could send them location data in exchange for money or services. i says in its marketing that it does foot traffic analysis and that its SDK is installed in more than 500 apps and has insights on more than 20 million devices. Placer.a “We partner with mobile apps providing location services and receive anonymized aggregated data. Very critically, all data is anonymized and stripped of personal identifiers before it reaches us,” Ethan Chernofsky, ’s vice president of marketing, said in an email. Placer.ai Into the Location Data Marketplace Once a person’s location data has been collected from an app and it has entered the location data marketplace, it can be sold over and over again, from the data providers to an aggregator that resells data from multiple sources. It could end up in the hands of a “location intelligence” firm that uses the raw data to analyze foot traffic for retail shopping areas and the demographics associated with its visitors. Or with a hedge fund that wants insights on how many people are going to a certain store. “There are the data aggregators that collect the data from multiple applications and sell in bulk. And then there are analytics companies which buy data either from aggregators or from applications and perform the analytics,” said Tsiounis of Advan Research. “And everybody sells to everybody else.” Some data marketplaces are part of well-known companies, like or , which sell all types of data, not just location data. Oracle boasts its listing as the “world’s largest third-party data marketplace” for targeted advertising, while Amazon claims to “make it easy to find, subscribe to, and use third-party data in the cloud.” Both marketplaces feature listings for several of the location data companies that we examined. Amazon’s AWS Data Exchange Oracle’s Data Marketplace Amazon spokesperson Claude Shy said that data providers have to explain how they gain consent for data and how they monitor people using the data they purchase. “Only qualified data providers will have access to the AWS Data Exchange. Potential data providers are put through a rigorous application process,” Shy said. Oracle declined to comment. Other companies, like Narrative, say they are simply connecting data buyers and sellers by providing a platform. lists location data providers like SafeGraph and Complementics among its 17 providers with more than two billion mobile advertising IDs to buy from. Narrative’s website, for instance But Narrative CEO Nick Jordan said the company doesn’t even look at the data itself. “There’s a number of companies that are using our platform to acquire and/or monetize geolocation data, but we actually don’t have any rights to the data,” he said. “We’re not buying it, we’re not selling it.” To give a sense of how massive the industry is, Amass Insights has 320 location data providers listed on its directory, Jordan Hauer, the company’s CEO, said. While the company doesn’t directly collect or sell any of the data, hedge funds will pay it to guide them through the myriad of location data companies, he said. “The most inefficient part of the whole process is actually not delivering the data,” Hauer said. “It’s actually finding what you’re looking for and making sure that it’s compliant, making sure that it has value and that it is exactly what the provider says it is.” Oh, the Places Your Data Will Go There are a whole slew of potential buyers for location data: investors looking for intel on market trends or what their competitors are up to, political campaigns, stores keeping tabs on customers, and law enforcement agencies, among others. Data from location intelligence firm Thasos Group has been used to measure the l . number of workers pulling extra shifts</a> at Tesla plants. Political campaigns on both sides of the aisle have also used ocation data from people who were at rallies</a> for targeted advertising Fast food restaurants and other businesses have been known to buy location data for advertising purposes down to a person’s steps. For example, in 2018, Burger King ran a promotion in which, if a customer’s phone was within , the Burger King app would let the user buy a Whopper for one cent. 600 feet of a McDonalds The Wall Street Journal and Motherboard have also written extensively about how federal agencies including the , , and bought location data from companies tracking phones. Internal Revenue Service Customs and Border Protection the U.S. military Of the location data firms The Markup examined, the offerings are diverse. Advan Research, for instance, uses historical location data to tell its customers, largely retail businesses or their private equity firm owners, where their visitors came from, and makes guesses about their income, race, and interests based on where they’ve been. “For example, we know that the average income in this neighborhood by census data is $50,000. But then there are two devices—one went to Dollar General, McDonald’s, and Walmart, and the other went to a BMW dealer and Tiffany’s ... so they probably make more money,” Advan Research’s Tsiounis said. Others combine the location data they obtain with other pieces of data gathered from your online activities. Complementics, which boasts data on “more than a billion mobile device IDs,” offers for mobile ad targeting. location data in tandem with cross-device data The prices can be steep. Outlogic (formerly known as X-Mode) offers a license for a titled “Cyber Security Location data” on Datarade for $240,000 per year. The listing says “Outlogic’s accurate and granular location data is collected directly from a mobile device’s GPS.” location dataset At the moment, there are few if any rules limiting who can buy your data. Sherman, of the Duke Tech Policy Lab, </a> finding that data brokers were advertising location information on people based on their political beliefs, as well as data on U.S. government employees and military personnel. published a report in August “There is virtually nothing in U.S. law preventing an American company from selling data on two million service members, let's say, to some Russian company that's just a front for the Russian government,” Sherman said. Existing privacy laws in the U.S., like California’s Consumer Privacy Act, do not limit who can purchase data, though California residents can request that their data not be “sold”—which can be n. Instead, the law focuses on allowing people to opt out of sharing their location in the first place. a tricky definitio The European Union’s General Data Protection Regulation has stricter requirements for when their data is being processed or transferred. notifying users But Ashkan Soltani, a privacy expert and former chief technologist for the Federal Trade Commission, said it’s unrealistic to expect customers to hunt down companies and insist they delete their personal data. “We know in practice that consumers don't take action,” he said. “It's incredibly taxing to opt out of hundreds of data brokers you've never even heard of." Companies like Apple and Google, who control access to the app stores, are in the best position to control the location data market, AppCensus’s Egelman said. “The real danger is the app gets booted from the Google Play store or the iOS app store,” he said.” As a result, your company loses money.” </a> from using location reporting SDKs from several data companies. Google and Apple both recently banned app developers , however, that the companies’ SDKs were still making their way into Google’s app store.  Apple didn’t respond to a request for comment. Researchers found "The Google Play team is always working to strengthen privacy protections through both product and policy improvements. When we find apps or SDK providers that violate our policies, we take action,” Google spokesperson Scott Westover said in an email. Digital privacy has been a key policy issue for U.S. senator Ron Wyden, a Democrat from Oregon, who told The Markup that the big app stores needed to do more. “This is the right move by Google, but they and Apple need to do more than play whack-a-mole with apps that sell Americans’ location information. These companies need a real plan to protect users’ privacy and safety from these malicious apps,” Wyden said in an email. This article was license. originally published on The Markup</a> and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives

Companies Pay Big Money to Have the Location Data From Your Phone

Most of the time when I’m developing applications, I default to using some form of SQL. Whether it’s a simple application using SQlite or something a bit more robust that I plan to scale out with MySQL, it’s inevitably some flavor of SQL. However, when starting new projects over the past few years I’ve been trying to make sure I have experience with document databases and have spent more time using in C#. MongoDB In this article, I’ll guide you through some basics for being able to insert data using MongoDB in C#. We’ll keep it light and to the point, so let’s dive into a brief overview of MongoDB in C# followed by some code examples in C#! Overview of MongoDB in C# MongoDB is a widely used NoSQL database that provides a flexible and scalable solution for data storage. One of the key benefits of MongoDB is its document-oriented nature so instead of using tables with fixed schemas like in traditional relational databases, MongoDB stores data in collections of JSON-like documents. This can allow for a more flexible and dynamic data model, especially in scenarios where data structures can vary. In C#, you can seamlessly integrate MongoDB into your applications because MongoDB provides a native driver for C#. This simplifies the process of interacting with the database and provides a comfortable development experience for C# developers. Well, mostly. If you’re used to using SQL connections, SQL command objects, and classes to access your data, it’s a little bit more cumbersome. But this is coming from the perspective of someone who prefers to write raw SQL queries in their data access layer. I suspect if you didn’t have this bias then working with their API, especially for filtering, would feel pretty intuitive. DataReader One big thing you’ll want to keep in mind if you’re coming from a relational database background is understanding the primary differences between relational and document databases. In the former, we generally denormalize data across tables and, in the latter, we write the data as a document that we’d like to read back without having to join. These types of databases aren’t designed to join data like we do with relational databases, so it’s something to be aware of! Follow Along with MongoDB in C# Content I always try to make sure I provide multiple modes of content when I’m able to. Below, you’ll find a YouTube video walking through examples of MongoDB in C# for inserting records: https://youtu.be/0fB9qg-oR04?embedable=true Of course, if you’d rather play around with some working code (aside from ensuring you have your own database), you can find the example code on my . You should be able to clone it, set up your connection string properly, change your database and collection names as needed, and then rock and roll! Dev Leader GitHub repository here Inserting Documents into MongoDB in C# In this section, I’ll explain how to insert documents into MongoDB using C#. We’ll look at code examples using the and methods. We’ll see these methods in their synchronous and asynchronous forms! Remember, you will need to ensure you have the “MongoDB.Driver” NuGet package installed in your project for the following code examples to work. InsertOne InsertMany Using the InsertOne and InsertOneAsync Methods To insert a single document into a MongoDB collection synchronously, use the method. This method is straightforward and is executed immediately, blocking the current thread until the operation completes. Here’s a basic example: InsertOne using MongoDB.Bson;
using MongoDB.Driver;

// Connect to MongoDB instance
var client = new MongoClient("mongodb://localhost:27017");

// Select database
var database = client.GetDatabase("testDatabase");

// Select collection (sort of like a SQL "table")
var collection = database.GetCollection<BsonDocument>("myCollection");

var document = new BsonDocument
{
    { "name", "John" },
    { "age", 30 }
};

// Insert document into collection
collection.InsertOne(document); In this example: We first establish a connection to the MongoDB instance and select our database and collection. Then, we create a object, which represents the document we want to insert. BsonDocument Finally, we call on our collection object, passing in the document. This inserts the document into the specified collection. InsertOne For asynchronous operations, you can use . This method is especially useful in applications that require non-blocking operations, such as web applications or services, to maintain responsiveness. Here’s how to use it: InsertOneAsync using MongoDB.Bson;
using MongoDB.Driver;
using System.Threading.Tasks;

// just like the former example
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("testDatabase");
var collection = database.GetCollection<BsonDocument>("myCollection");

var document = new BsonDocument
{
    { "name", "Jane" },
    { "age", 28 }
};

// Asynchronously insert document
await collection.InsertOneAsync(document, cancellationToken: CancellationToken.None); In this asynchronous example: The setup is similar to the synchronous version, but we use with . This tells the program to continue with other work that doesn’t depend on the completion of the insert operation. await InsertOneAsync Take note that you can pass in a cancellation token here which is a best practice for async code Note that both and methods throw exceptions if the insert operation fails, so you should use try-catch blocks to handle potential errors gracefully. InsertOne InsertOneAsync Using the InsertMany and InsertManyAsync Methods The method synchronously inserts a list of document objects into the specified collection. This is useful when you have multiple documents ready to be stored and want to execute the operation in a single call. Here’s an example: InsertMany using MongoDB.Bson;
using MongoDB.Driver;
using System.Threading.Tasks;

// just like the former example
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("testDatabase");
var collection = database.GetCollection<BsonDocument>("myCollection");

var documents = new List<BsonDocument>
{
    new BsonDocument("name", "John").Add("age", 30),
    new BsonDocument("name", "Jane").Add("age", 25),
    new BsonDocument("name", "Doe").Add("age", 28)
};

collection.InsertMany(documents); In this example, we first get a reference to our collection. Then, we of objects, each representing a document to insert. Finally, we call with our list of documents. This method will insert all documents in the list into the MongoDB collection in one operation. create a list BsonDocument InsertMany For asynchronous operations, particularly useful in applications where responsiveness is crucial or when dealing with I/O-bound tasks, you can use the method. This method works similarly to , but it performs the operation asynchronously. Here’s an example: InsertManyAsync InsertMany using MongoDB.Bson;
using MongoDB.Driver;
using System.Threading.Tasks;

// just like the former example
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("testDatabase");
var collection = database.GetCollection<BsonDocument>("myCollection");

var documents = new List<BsonDocument>
{
    new BsonDocument("name", "John").Add("age", 30),
    new BsonDocument("name", "Jane").Add("age", 25),
    new BsonDocument("name", "Doe").Add("age", 28)
};

await collection.InsertManyAsync(documents); In this asynchronous version, we use before calling , ensuring that the operation completes before moving on to the next line of code. This is especially important in web applications or services where blocking the main thread could lead to poor performance or user experience. await InsertManyAsync Best Practices for Efficiently Inserting Data This article is focused on the API usage of inserting data into MongoDB using C#, but I wanted to touch on a few other points. To improve the efficiency of data insertion in MongoDB, you can consider the following best practices: Batch Inserts: Instead of inserting one document at a time, you can batch multiple documents together and insert them in a single operation using InsertMany. This reduces the overhead of multiple round-trips to the database. Use Indexes: Indexes can significantly improve the insert performance by speeding up the search for the correct position to insert the new document. Ensure that you have appropriate indexes defined on the fields commonly used for insertion. Consider Sharding: If you have a large amount of data to insert, sharding can distribute the data across multiple servers, improving insertion performance. Write Concern: The default write concern in MongoDB is Acknowledged, which waits for an acknowledgment from the server. If you are performing bulk insertions and do not require immediate acknowledgment, you can set a lower write concern to improve performance. Consider Asynchronous Operations: Asynchronous operations can improve the responsiveness of your application by allowing multiple insert operations to be executed concurrently. Aside from asynchronous versions of these APIs which were mentioned earlier, I won’t be covering these other topics in more detail in this article. Stay tuned for upcoming articles where I cover these topics along with other functionality, such as updating documents in MongoDB and deleting records! Wrapping Up MongoDB in C# To recap, in this article, we explored the topic of writing data to MongoDB in C# using InsertOne and InsertMany APIs. We saw both the synchronous and asynchronous versions of these APIs in functional code examples. Some of the key takeaways: Understanding relational database vs document database characteristics Best practice considerations for performance Synchronous vs asynchronous APIs for inserting data to MongoDB in C# You should continuously improve your skills and stay updated with the latest technologies! If you found this useful and you’re looking for more learning opportunities, consider and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube

Simplified Guide For Inserting Data Into MongoDB Using C#

In this article, I aim to share my experience standardizing the ETL process within frameworks like Spark and Flink, particularly for those not using data processing frameworks like dbt. Context Data processing is a comprehensive field that encompasses the regular processing of substantial information volumes in both batch and stream modes. These procedures, often referred to as ETL processes, are developed using a variety of programming languages and frameworks. Typically, the main stages of data processing are as follows: Data is collected from various sources for subsequent processing. Reading data: The data undergoes various transformations and manipulations to attain the required structure and format. Transformation: The transformed data is then stored in the target storage. Writing: Schematically, it is represented as shown in the diagram: Challenges After the main stages are implemented, several other stages can be added to enhance and refine the process: This stage involves verifying data compliance with requirements and expectations, including data cleaning to ensure accuracy and completeness. Data quality check: The registration process tracks and documents available data sets. It may also involve data and metadata management for effective data organization. Data registration in the data catalog: This allows visualization of relationships between data assets and ETL processes to understand dependencies. It can help identify potential data quality issues and improve overall data quality. Building data lineage: Additional processing may define the level of data confidentiality and criticality, ensuring proper data protection and security. Data classification: Each of the stages mentioned can be implemented using various technologies and tools, allowing flexibility in choosing the right solution for each specific stage of data processing. Solution Reading data The first step in the ETL process involves reading data from the source. Various types of sources may be used at this stage, such as files, databases, message brokers, and more. The data may also come in different formats, such as CSV, JSON, Parquet, Delta, Hudi, etc. Here is some boilerplate code for reading data from these different sources (implemented as Strategy pattern): from abc import ABC, abstractmethod
from pyspark.sql import SparkSession, DataFrame

# Define the strategy interface
class ReadStrategy(ABC):
    @abstractmethod
    def read_data(self) -> DataFrame:
        pass

# File System Read Strategy - represents a file system reading    
class FileSystemReadStrategy(ReadStrategy):
    def __init__(self, spark: SparkSession, path: str):
        self.spark = spark
        self.path = path
    
    @abstractmethod
    def read_data(self):
        pass
    
# Implement the strategy for CSV
class CSVReadStrategy(FileSystemReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("csv").option("header", "true").load(self.path)


# Implement the strategy for Delta
class DeltaReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("delta").load(self.path)

# Implement the strategy for Parquet
class ParquetReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("parquet").load(self.path)

# Context class
class DataReader:
    def __init__(self, read_strategy: ReadStrategy):
        self.read_strategy = read_strategy

    def read(self) -> DataFrame:
        return self.read_strategy.read_data() Usage example: spark = SparkSession.builder.appName("our-ETL-job").getOrCreate()
path = "path-to-our-data"
src_df = DataReader(DeltaReadStrategy(spark, path)).read() This approach offers several benefits: It enables the alteration of the reading strategy (source format). It facilitates the addition of new reading strategies without modifying existing code. Each strategy can have its unique parameters. For example, CSV may have a delimiter, while Delta may have a version, etc. Instead of providing a path, we can create a dedicated class, like , which can hold all necessary data reading parameters. ReadConfig It allows the incorporation of additional logic into strategies without affecting the main code. For instance: We can call by default for a file source. input_file_name() If we have a data catalog, we can check whether the table exists there and register it if not. Each strategy can be tested individually. Transformation After reading the data, it must be transformed. This can include various operations like filtering, aggregating, and joining. Since the transformation process can be complex and time-consuming, it's crucial to standardize it for ease of understanding. From my experience, using the method of the DataFrame API is an effective way to standardize this process. This method lets us chain multiple transformations, enhancing code readability. Each transformation can be declared as a separate pure function, performing a specific transformation, and can be tested independently. transform import pyspark.sql.functions as F
from pyspark.sql import DataFrame

desired_columns = ["name", "age", "country"]

def with_country_code(df: DataFrame) -> DataFrame:
    return df.withColumn("country_code", F.when(F.col("country") == "USA", "US").otherwise("Other"))

def with_age_category(df: DataFrame) -> DataFrame:
    return df.withColumn("age_category", F.when(F.col("age") < 18, "child").when((F.col("age") >= 18) & (F.col("age") < 65), "adult").otherwise("senior"))

def filter_adult(df: DataFrame) -> DataFrame:
    return df.filter(F.col("age_category") == "adult")

def select_columns(columns: list) -> callable[[DataFrame], DataFrame]:
    def inner(df: DataFrame) -> DataFrame:
        return df.select(columns)
    return inner

transformed_df = (src_df
                  .transform(with_country_code)
                  .transform(with_age_category)
                  .transform(filter_adult)
                  .transform(select_columns(desired_columns))
                  ) Writing The last stage in the ETL process is writing the transformed data into the target storage, which could be a file system, a database, or a message broker. The loading phase may also incorporate extra steps like data validation, data registration, and data lineage. For this stage, we can apply the same methods as reading data, but with writing strategies. from abc import ABC, abstractmethod
from pyspark.sql import DataFrame, SparkSession

class WriteStrategy(ABC):
    @abstractmethod
    def write_data(self, df: DataFrame):
        pass

class FileSystemWriteStrategy(WriteStrategy):
    def __init__(self, spark: SparkSession, path: str, format: str, mode: str = "overwrite"):
        self.spark = spark
        self.path = path
        self.format = format
        self.mode = mode

    def write_data(self, df: DataFrame):
        df.write.format(self.format).mode(self.mode).save(self.path)

class DataWriter:
    def __init__(self, write_strategy: WriteStrategy):
        self.write_strategy = write_strategy

    def write(self, df: DataFrame):
        self.write_strategy.write_data(df) Usage example: path = "path-to-our-data"
writer = DataWriter(FileSystemWriteStrategy(spark, path, "delta"))
writer.write(transformed_df) Conclusion This implementation standardizes the ETL process, making it more comprehensible and maintainable. It facilitates the addition of new reading and writing strategies without modifying existing code. Each strategy can possess unique parameters, and additional logic can be incorporated into strategies without disrupting the main code. The approach also enables separate testing of each strategy. Moreover, we can introduce additional stages to the ETL process, such as data quality checks, data registration in the data catalog, building data lineage, and data classification.

Dev Standards for Spark-jobs

স্পার্ক-জবসের জন্য ডেভ স্ট্যান্ডার্ড

Entwicklungsstandards für Spark-Jobs

Estándares de desarrollo para trabajos Spark

Normes de développement pour les tâches Spark

स्पार्क-नौकरियों के लिए विकास मानक

Spark ジョブの開発標準

Spark 작업을 위한 개발 표준

Padrões de desenvolvimento para trabalhos Spark

Стандарты разработки для Spark-заданий

Spark işleri için Geliştirme Standartları

Tiêu chuẩn dành cho nhà phát triển cho công việc Spark

Spark 作业的开发标准

Auto-Increment Columns in Databases: A Simple Trick That Makes a Big Difference

Top 10 Websites to Learn SQL and Database Online

Step 1: Install the necessary libraries To connect to a database and query data, you need to install the following two libraries: pandas: A Python library for data analysis, including functions for reading and writing data.
sqlalchemy: A Python library for interacting with relational databases, allowing you to use Python with various databases. You can install these libraries using the following command in the command prompt or terminal: pip install pandas
pip install sqlalchemy Step 2: Connect to the database Connecting to a database requires the following information: Database Type: The type of database you want to connect to, such as MySQL, PostgreSQL, etc.
Host Name: The host name or IP address where the database is located.
Port Number: The port number of the database, usually the default port number.
Username: The username required to connect to the database.
Password: The password required to connect to the database.
Database Name: The name of the database you want to connect to. You can use the following Python code to connect to the database: from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') Step 3: Query data using the read_sql function Use the read_sql function from pandas to query data from the database. The read_sql function requires two parameters: SQL Query: The SQL query you want to execute.
Database Connection: The database connection you created earlier. Here is an example of querying data: import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X(@Luca_DataTeam). Happy exploring!👋 Step 1: Install the necessary libraries To connect to a database and query data, you need to install the following two libraries: pandas: A Python library for data analysis, including functions for reading and writing data. sqlalchemy: A Python library for interacting with relational databases, allowing you to use Python with various databases. pandas : A Python library for data analysis, including functions for reading and writing data. pandas sqlalchemy : A Python library for interacting with relational databases, allowing you to use Python with various databases. sqlalchemy You can install these libraries using the following command in the command prompt or terminal: pip install pandas
pip install sqlalchemy pip install pandas
pip install sqlalchemy Step 2: Connect to the database Connecting to a database requires the following information: Database Type: The type of database you want to connect to, such as MySQL, PostgreSQL, etc. Host Name: The host name or IP address where the database is located. Port Number: The port number of the database, usually the default port number. Username: The username required to connect to the database. Password: The password required to connect to the database. Database Name: The name of the database you want to connect to. Database Type : The type of database you want to connect to, such as MySQL, PostgreSQL, etc. Database Type Host Name : The host name or IP address where the database is located. Host Name Port Number : The port number of the database, usually the default port number. Port Number Username : The username required to connect to the database. Username Password : The password required to connect to the database. Password Database Name : The name of the database you want to connect to. Database Name You can use the following Python code to connect to the database: from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') from sqlalchemy import create_engine

# Connect to a MySQL database
engine = create_engine('mysql://username:password@hostname:port/databasename')

# Connect to a PostgreSQL database
engine = create_engine('postgresql://username:password@hostname:port/databasename')

# Connect to an Oracle database, requires cx_Oracle library to be installed
engine = create_engine('oracle+cx_oracle://username:password@hostname:port/databasename') Step 3: Query data using the read_sql function Use the read_sql function from pandas to query data from the database. The read_sql function requires two parameters: SQL Query: The SQL query you want to execute. Database Connection: The database connection you created earlier. SQL Query : The SQL query you want to execute. SQL Query Database Connection : The database connection you created earlier. Database Connection Here is an example of querying data: import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) import pandas as pd

# Execute SQL query and store the result in a DataFrame
df = pd.read_sql('SELECT * FROM mytable', engine)

# Print the DataFrame
print(df) Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on LinkedIn or X (@Luca_DataTeam). Happy exploring!👋 LinkedIn X

How to Connect to Oracle, MySql and PostgreSQL Databases Using Python

Storing Passwords In A Database: A Better Way to Do So

In this article, I’ll guide you through C# Testcontainers for MongoDB! I love using Testcontainers for local database work when I don’t want to think about managing the overhead of spinning up and tearing down a database, or managing something in the cloud. MongoDB is no exception here as the C# Testcontainers NuGet package that we have for MongoDB is a dream to work with. In this article, I’ll introduce you to the basics of Testcontainers. From there, I’ll demonstrate how simple it is to interact with your newly created database via the C# MongoDB.Driver NuGet package — almost like you wouldn’t even know there was a container spun up! Let’s dive into it. Overview of C# Testcontainers Testcontainers is an awesome NuGet package that can greatly enhance your testing and local development workflow when building .NET applications. It provides a simple and effective way to manage Docker containers, allowing you to spin up lightweight and isolated instances and have them tear down like nothing ever happened! That means with Testcontainers, you no longer have to worry about setting up and tearing down test databases or dealing with complex configuration files. It handles all the details of container management, making this process feel trivial — I sure wouldn’t want to handle this manually. Some of the key benefits of Testcontainers include: Its ability to provide a consistent and reproducible test environment. You can define the exact service version and configuration that you require, ensuring that your test results are reliable and consistent across different environments. The speed and flexibility it brings to your testing and development workflow. By using lightweight Docker containers, you can quickly spin up and tear down test environments on demand. No configuring cloud or shared resources ahead of time… Just on-the-fly awesomeness. Seamless integration with existing testing frameworks and tools in the DotNet ecosystem, such as Xunit, NUnit, and BenchmarkDotNet. This allows you to easily incorporate containerized service instances into your existing test suites without any major modifications. For this article, we’ll be focusing on how we can use C# Testcontainers for MongoDB in particular. Much of the advice covered here will be applicable to other services, especially databases, but we’ll keep our sights set on MongoDB. Setting Up Testcontainers in C# To get started with Testcontainers in C#, you’ll need to install and configure the necessary dependencies in your project. Follow the steps below to set up Testcontainers for MongoDB in C#. 1. Installing Testcontainers Generally, the first step would be to install the Testcontainers NuGet package in your C# project. You would open the NuGet Package Manager Console, and run the following command: Install-Package Testcontainers However, we need the MongoDB flavor of this package (I am using version 3.8.0 at the time of writing) which includes the base Testcontainers package. You can install it via the package manager user interface or with this command: Install-Package Testcontainers.MonogoDB 2. Configuring Testcontainers for MongoDB Once you have installed Testcontainers, you need to configure it for MongoDB. This involves setting up a MongoDB container with the required settings. Here is an example of how you can configure Testcontainers for MongoDB in C#: using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
    .WithImage("mongo:latest")
    .Build(); In the above example, we create a new instance of the , and specify the MongoDB image and port binding. The method sets the image for the MongoDB container, so using “mongo:latest” will always pull the latest image. MongoDbBuilder WithImage Optionally, we could use port binding to configure which ports we want to be using explicitly: using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
    .WithImage("mongo:latest")
    .WithPortBinding(1337, 27017)
    .Build(); The code above uses method binds the container’s port 27017 to the host machine’s port 1337. We’ll see in the upcoming sections that unless you have a specific need to bind ports, you can potentially just leave this be. We can get a dynamic connection string, which is very convenient! WithPortBinding 3. Starting and Stopping the MongoDB Container After configuring Testcontainers for MongoDB, you can start and stop the container as needed. Here are code examples for starting and stopping the container: To start the MongoDB container: await container.StartAsync(); To stop the MongoDB container: await _container.StopAsync(); Remember the MongoDbContainer type can be disposed of, so call to clean up when you’re ready. DisposeAsync Using C# Testcontainers for MongoDB in Practice Now that we’ve seen how to spin up and tear down containers, what are we supposed to do with them? We have plenty of options, and truly, the limit is your own imagination (and time, money, and other resources…)! Most people leverage C# Testcontainers for writing integration or . You can match up Testcontainers with xUnit or NUnit, for example, and write tests that interact with a real database. functional tests For me, I needed to go wire C# Testcontainers for MongoDB up to BenchmarkDotNet so I could ! A similar use case where I needed transient local databases, but not necessarily for a test framework. benchmark inserting records into MongoDB Regardless, we need to be able to connect to these MongoDB database containers from C#, and these next subsections will cover that. Connecting to C# Testcontainers for MongoDB with MongoDB.Driver Let’s assume you read the earlier sections and you have the correct MongoDB package installed for Testcontainers. If you skimmed ahead to this section, go back, read it, and get the right package. Otherwise, you’ll be scratching your head about why you can’t find the right dependencies! You’ll also need to get the MongoDB.Driver NuGet package installed. This is what we’re going to be using to establish a connection to the database container that we spun up. I have other articles you can read if you want more explanation on how MongoDB.Driver works in general: MongoDB in C#: Simplified Guide For Inserting Data MongoDB Filtering in C# – Beginner’s Guide For Easy Filters How To Delete Documents From MongoDB In C# – What You Need To Know How To Update MongoDB Documents in C# – What You Need To Know With the right packages ready to go, we can combine the code we saw earlier with some MongoDB driver code to get things glued together: using MongoDB.Bson;
using MongoDB.Driver;

using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
    .WithImage("mongo:latest")
    .Build();
await container.StartAsync();
string connectionString = container.GetConnectionString();

MongoClient mongoClient = new MongoClient(connectionString);
IMongoDatabase database = mongoClient.GetDatabase("test");
IMongoCollection<BsonDocument> collection = database.GetCollection<BsonDocument>("test"); In the example above, we can call on the newly spun-up MongoDB container from Testcontainers. The awesome part about this is that no matter how you configure your Testcontainer for MongoDB using the builder pattern we see in this code and earlier in the article, will get you what you need to connect. GetConnectionString() GetConnectionString() Because takes in a connection string as a single argument, it’s trivial to get access to the MongoDB database to start working with it right away! MongoClient Performing CRUD Operations on the MongoDB Testcontainer Now that we have our MongoDB Testcontainer set up and we have a connected to it, we can start performing CRUD operations on it. The connection details provided by the container instance made it a breeze to combine these two things, and truly, we can focus on just the MongoDB.Driver method calls for CRUD. MongoClient Here’s an example of performing a simple CRUD operation: // using code from earlier examples...

// Create
await collection.InsertOneAsync(new BsonDocument()
{
    ["Name"] = "Nick Cosentino",
});

// Read
var filterBuilder = Builders<BsonDocument>.Filter;
var filter = filterBuilder.Eq("Name", "Nick Cosentino");
var results = collection.Find(filter);

// Update
var updateBuilder = Builders<BsonDocument>.Update;
var update = updateBuilder.Set("Name", "Dev Leader");
collection.UpdateOne(filter, update);

// Delete
filter = filterBuilder.Eq("Name", "Dev Leader");
collection.DeleteOne(filter); Using the previous code snippets, we were already up and running with our MongoDB container instance and a . As a result, you can see in the code snippet above that we can directly work with our that we obtained from the prior setup. MongoClient IMongoCollection<BsonDocument> This helps illustrate that once you have the MongoDB docker container up and running via Testcontainers, you don’t have to treat things any special way once you connect to it! Wrapping Up C# Testcontainers for MongoDB To conclude, C# Testcontainers for MongoDB is an incredibly simple way to get a transient MongoDB data store up and running for your testing and development. While this article didn’t focus on specific use cases, hopefully, you have a better understanding of how transparent it feels to layer a MongoDB driver on top of Testcontainers to interact with MongoDB. Given that C# Testcontainers takes the headache out of the setup, teardown, and even connection string management, it’s an easy recommendation from me if you want to get up and running with MongoDB for local usage. If you found this useful and you’re looking for more learning opportunities, consider , and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube Also published here

Easily Run Local Databases Using C# Testcontainers for MongoDB

MongoDB-এর জন্য C# টেস্ট কনটেইনার ব্যবহার করে সহজেই স্থানীয় ডেটাবেস চালান

Führen Sie ganz einfach lokale Datenbanken mit C#-Testcontainern für MongoDB aus

Ejecute fácilmente bases de datos locales utilizando C# Testcontainers para MongoDB

Exécutez facilement des bases de données locales à l'aide de C# Testcontainers pour MongoDB

MongoDB के लिए C# टेस्टकंटेनर का उपयोग करके आसानी से स्थानीय डेटाबेस चलाएँ

MongoDB 用の C# Testcontainers を使用してローカル データベースを簡単に実行する

MongoDB용 C# 테스트 컨테이너를 사용하여 로컬 데이터베이스를 쉽게 실행

Execute facilmente bancos de dados locais usando C# Testcontainers para MongoDB

Легко запускайте локальные базы данных с помощью тестовых контейнеров C# для MongoDB

MongoDB için C# Testcontainer'larını Kullanarak Yerel Veritabanlarını Kolayca Çalıştırın

Dễ dàng chạy cơ sở dữ liệu cục bộ bằng C# Testcontainer cho MongoDB

使用 MongoDB 的 C# 测试容器轻松运行本地数据库

DATARADE

datarade.ai

-- employees

Since n.d.

COMPANY RANKING

DATARADE

EVERGREEN INDEX #3545

Representing this company? Click here to claim and customize this page!

Datarade WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DATARADE

datarade.ai

-- employees

Since n.d.

COMPANY RANKING

DATARADE

EVERGREEN INDEX #3545

RELATED COMPANIESRANK

STORIES ABOUT DATARADE

Representing this company? Click here to claim and customize this page!

Datarade WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!