Don’t just sell to your customers, build a relationship with them. Our Omnichannel sales tool - Smart Label generator and Custom document creation tool - <strong>Sourcewiz Studio helps companies provide delightful experiences to their customers and reduce revenue leakage by multiple folds</strong>.

Sourcewiz | LinkedIn

Sourcewiz is <strong>a SAAS platform built for manufacturing, distribution, retail, warehousing, export import and many other such businesses for digital transformation</strong>. We are a B2B digital market for businesses providing digital solutions, visual merchandising, manufacturing processes, wholesalers ...

About Sourcewiz

We started using <strong>Sourcewiz</strong> internally first. Over time, they added many useful features to the product that would save both time and cost in many areas of our business. This has made it an integral part of our <strong>company</strong>&#x27;s operations.

Sourcewiz: Sales automation app for B2B manufacturing

<strong>Sourcewiz</strong> is a platform for exporters to digitize their workflows and help them grow and manage their business.

Sourcewiz - Crunchbase Company Profile & Funding

Information on valuation, funding, cap tables, investors, and executives for <strong>Sourcewiz</strong>. Use the PitchBook Platform to explore the full profile.

Sourcewiz Company Profile: Valuation & Investors | PitchBook

<strong>Sourcewiz</strong> has raised $3M in a seed funding round co-led by Blume Ventures and Alpha Wave Incubation and Angel investors from renowed <strong>companies</strong> like Gokaldas Exports, Wayfair US, Transworld Group of <strong>Companies</strong>, Classplus &amp; Posist. Some of their clients are Sangat Overseas, Seet Kamal International, ...

Sourcewiz Company Profile, information, investors, valuation & Funding

Find contact information for <strong>Sourcewiz</strong>. Learn about their Business Intelligence (BI) Software, Software market share, competitors, and <strong>Sourcewiz</strong>&#x27;s email format.

Sourcewiz Company Profile | Management and Employees List

<strong>Sourcewiz</strong> - Tech enabler platform offering operation management solutions to businesses. Raised a total funding of $5.5M over 2 rounds from 37 investors. Valued at $23.5M. Founded by Divyanshu Makkar, Mayur Chandrakant Bhangale and 1 other in the year 2020. <strong>Sourcewiz</strong> has 549 competitors.

Sourcewiz - Company Profile - Tracxn

Sourcewiz - Funding, Financials, Valuation & Investors

Small and medium-sized exporters in India struggle to communicate their designs with buyers. They can’t list those designs publicly on their website or

Blume Ventures and AWI back Indian exporter SaaS startup SourceWiz ...

Apply to jobs at <strong>Sourcewiz</strong> today. Learn about careers at <strong>Sourcewiz</strong> in software, marketing, sales and operations. Let <strong>Sourcewiz</strong> and other <strong>companies</strong> compete to hire you!

Sourcewiz Careers | Jobs at Sourcewiz - Instahyre

<strong>Sourcewiz</strong> on the 2023 30 Under 30 - Asia - Enterprise Technology - Makkar, Garg and Bhangale cofounded the Bangalore-based SaaS <strong>company</strong> <strong>Sourcewiz</strong> in 2021.

Sourcewiz

Using the <strong>Sourcewiz</strong> platform, exporters have been able to cater to more buyers, increase sales conversion rate and reduce sampling cost, the <strong>company</strong> said. &quot;We are already seeing some of our early customers double the number of buyers they engage with,” Makkar said.

sourcewiz funding: Sourcewiz gets $3 million seed funding from ...

How much do <strong>Sourcewiz</strong> employees make? Glassdoor has salaries, wages, tips, bonuses, and hourly pay based upon employee reports and estimates. ... 50LPA (43 fixed) in India vs 110k in Texas. Which one is worth? A 12 years experienced full stack developer. Microsoft Stack with Azure and AWS cloud ... I am software engineer i have 20 years experience if change <strong>company</strong> ...

How Much Does Sourcewiz Pay in 2023? (49 Salaries) | Glassdoor

Sourcewiz - Contacts, Employees, Board Members, Advisors & Alumni

You need to enable JavaScript to run this app

Sourcewiz | Company Page | IndieFolio

Our first product is a vertical sales software built for exporters, which allows them to market their unique creations to more buyers, generate more inquiries and increase their sales conversion

RxJS for Beginners:  Mastering Observables, Subjects, and Operators for Reactive Programming

Using MinIO and Python to Create Scalable and Efficient Event-driven Applications

স্কেলযোগ্য এবং দক্ষ ইভেন্ট-চালিত অ্যাপ্লিকেশন তৈরি করতে MinIO এবং Python ব্যবহার করে

Verwenden von MinIO und Python zum Erstellen skalierbarer und effizienter ereignisgesteuerter Anwendungen

Uso de MinIO y Python para crear aplicaciones basadas en eventos escalables y eficientes

Utiliser MinIO et Python pour créer des applications événementielles évolutives et efficaces

स्केलेबल और कुशल इवेंट-संचालित एप्लिकेशन बनाने के लिए मिनियो और पायथन का उपयोग करना

MinIO と Python を使用してスケーラブルで効率的なイベント駆動型アプリケーションを作成する

MinIO 및 Python을 사용하여 확장 가능하고 효율적인 이벤트 기반 애플리케이션 만들기

Usando MinIO e Python para criar aplicativos orientados a eventos escalonáveis e eficientes

Использование MinIO и Python для создания масштабируемых и эффективных приложений, управляемых событиями

Ölçeklenebilir ve Verimli Olay Odaklı Uygulamalar Oluşturmak için MinIO ve Python'u Kullanmak

Sử dụng MinIO và Python để tạo các ứng dụng hướng sự kiện có khả năng mở rộng và hiệu quả

使用 MinIO 和 Python 创建可扩展且高效的事件驱动应用程序

Benchmarking C# Regular Expressions to Measure Performance

is an open-source search and analytics engine based on Apache Lucene. When building applications on change data capture (CDC) data using Elasticsearch, you’ll want to architect the system to handle frequent updates or modifications to the existing documents in an index. Elasticsearch In this blog, we’ll walk through the different options available for updates including full updates, partial updates, and scripted updates. We’ll also discuss what happens under the hood in when modifying a document and how frequent updates impact CPU utilization in the system. Elasticsearch Example Application with Frequent Updates To better understand use cases that have , let’s look at a search application for a video streaming service like Netflix. When a user searches for a show, ie “political thriller”, they are returned a set of relevant results based on keywords and other metadata. frequent updates Let’s look at an example document in Elasticsearch of the show “House of Cards”: {
	"name": "House of Cards",
	"description": "Frank Underwood is a Democrat appointed as the Secretary of State. Along with his wife, he sets out on a quest to seek revenge from the people who betrayed him while successfully rising to supremacy.",
	"genres": ["drama", "thriller"],
	"views": 100,
} The search can be configured in Elasticsearch to use and as full-text search fields. The field, which stores the number of views per title, can be used to boost content, ranking more popular shows higher. The field is incremented every time a user watches an episode of a show or a movie. name description views views When using this search configuration in an application the scale of , the number of updates performed can easily cross millions per minute as determined by the . According to the Report, users watched ~100 billion hours of content from January to July. Assuming an average watch time of 15 minutes per episode or a movie, the number of views per minute reaches 1.3 million on average. With the search configuration specified above, each view would require an update in the millions scale. Netflix Netflix Engagement Report Many search and analytics applications can experience frequent updates, especially when built on CDC data. Performing updates in Elasticsearch Let’s delve into a general example of how to perform an update in Elasticsearch with the code below: -
from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID you want to update
index_name = 'movies'
document_id = 'your_document_id'

# Retrieve the current document to get the current 'views' value
try:
    current_doc = es.get(index=index_name, id=document_id)
    current_views = current_doc['_source']['views']
except Exception as e:
    print(f"Error retrieving current document: {e}")
    current_views = 0  # Set a default value if there's an error

# Define the update body to increment 'views' by 1
update_body = {
    "doc": {
        "views": current_views + 1  # Increment 'views' by 1
    }
}

# Perform the update
try:
    es.update(index=index_name, id=document_id, body=update_body)
    print("Document updated successfully!")
except Exception as e:
    print(f"Error updating document: {e}") Full Updates versus Partial Updates in Elasticsearch When performing an update in Elasticsearch, you can use the to replace an existing document or the to make a partial update to a document. index API update API The index API retrieves the entire document, makes changes to the document, and then reindexes the document. With the update API, you simply send the fields you wish to modify, instead of the entire document. This still results in the document being reindexed but minimizes the amount of data sent over the network. The update API is especially useful in cases where the document size is large and sending the entire document over the network will be time-consuming. Let’s see how both the index API and the update API work using Python code. Full updates using the index API in Elasticsearch from elasticsearch import Elasticsearch

# Connect to Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID
index_name = "your_index"
document_id = "1"

# Retrieve the existing document
existing_document = es.get(index=index_name, id=document_id)

# Make your changes to the document
existing_document["_source"]["field1"] = "new_value1"
existing_document["_source"]["field2"] = "new_value2"

# Call the index API to perform the full update
es.index(index=index_name, id=document_id, body=existing_document["_source"]) As you can see in the code above, the index API requires two separate calls to Elasticsearch which can result in slower performance and higher load on your cluster. Partial updates using the update API in Elasticsearch Partial updates internally use the , but have been configured to only require a single network call for better performance. reindex API from elasticsearch import Elasticsearch

# Connect to Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID
index_name = "your_index"
document_id = "1"

# Specify the fields to be updated
update_fields = {
    "field1": "new_value1",
    "field2": "new_value2"
}

# Use the update API to perform a partial update
es.update(index=index_name, id=document_id, body={"doc": update_fields}) You can use the update API in Elasticsearch to update the view count but, by itself, the update API cannot be used to increment the view count based on the previous value. That is because we need the older view count to set the new view count value. Let’s see how we can fix this using a powerful scripting language, Painless. Partial updates using Painless scripts in Elasticsearch is a scripting language designed for Elasticsearch and can be used for query and aggregation calculations, complex conditionals, data transformations and more. Painless also enables the use of scripts in update queries to modify documents based on complex logic. Painless In the example below, we use a Painless script to perform an update in a single API call and increment the new view count based on the value of the old view count. from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID you want to update
index_name = 'movies'
document_id = 'your_document_id'

# Define the Painless script for the update
update_script = {
    "script": {
        "lang": "painless",
        "source": "ctx._source.views += 1"  # Increment 'views' by 1
    }
}

# Perform the update using the Painless script
try:
    es.update(index=index_name, id=document_id, body=update_script)
    print("Document updated successfully!")
except Exception as e:
    print(f"Error updating document: {e}") The Painless script is pretty intuitive to understand, it is simply incrementing the view count by 1 for every document. Updating a Nested Object in Elasticsearch in Elasticsearch are a data structure that allows for the indexing of arrays of objects as separate documents within a single parent document. Nested objects are useful when dealing with complex data that naturally form a nested structure, like objects within objects. In a typical Elasticsearch document, arrays of objects are flattened, but using the nested data type allows each object in the array to be indexed and queried independently. Nested objects Painless scripts can also be used to update nested objects in Elasticsearch. from elasticsearch import Elasticsearch

# Connect to your Elasticsearch instance
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Index name and document ID for the example
index_name = 'your_index'
document_id = 'your_document_id'

# Specify the nested field and the updated value
nested_field = "nested_field_name"
updated_value = "new_value"

# Define the Painless script for the update
update_script = {
    "script": {
        "lang": "painless",
        "source": "ctx._source.nested_field_name = params.updated_value",
        "params": {
            "updated_value": updated_value
        }
    }
}

# Perform the update using the Update API and the Painless script
try:
    es.update(index=index_name, id=document_id, body=update_script)
    print("Nested object updated successfully!")
except Exception as e:
    print(f"Error updating nested object: {e}") Adding a New Field in Elasticsearch Adding a new field to a document in Elasticsearch can be accomplished through an index operation. You can partially update an existing document with the new field using the Update API. When dynamic mapping on the index is enabled, introducing a new field is straightforward. Simply index a document containing that field and Elasticsearch will automatically figure out the suitable mapping and add the new field to the mapping. With dynamic mapping on the index disabled, you will need to use the update mapping API. You can see an example below of how to update the index mapping by adding a “category” field to the movies index. PUT /movies/_mapping
{
  "properties": {
    "category": {
      "type": "keyword"
    }
  }
} Updates in Elasticsearch Under the Hood While the code is simple, Elasticsearch internally is doing a lot of heavy lifting to perform these updates because data is stored in immutable segments. As a result, Elasticsearch cannot simply make an in-place update to a document. The only way to perform an update is to reindex the entire document, regardless of which API is used. Elasticsearch uses Apache Lucene under the hood. A Lucene index is composed of one or more segments. A segment is a self-contained, immutable index structure that represents a subset of the overall index. When documents are added or updated, new Lucene segments are created and older documents are marked for soft deletion. Over time, as new documents are added or existing ones are updated, multiple segments may accumulate. To optimize the index structure, Lucene periodically merges smaller segments into larger ones. Updates are essentially inserts in Elasticsearch Since each update operation is a reindex operation, all updates are essentially inserts with soft deletes. There are cost implications for treating an update as an insert operation. On one hand, the soft deletion of data means that old data is still being retained for some time, bloating the storage and memory of the index. Performing soft deletes, reindexing and garbage collection operations also take a heavy toll on the CPU, a toll that is exacerbated by repeating these operations on all replicas. Updates can get more tricky as your product grows and your data changes over time. To keep Elasticsearch performant, you will need to update the shards, analyzers, and tokenizers in your cluster, requiring a reindexing of the entire cluster. For production applications, this will require setting up a new cluster and migrating all of the data over. Migrating clusters is both time-intensive and error prone so it's not an operation to take lightly. Updates in Elasticsearch The simplicity of the update operations in Elasticsearch can mask the heavy operational tasks happening under the hood of the system. Elasticsearch treats each update as an insert, requiring the full document to be recreated and reindexed. For applications with frequent updates, this can quickly become expensive as we saw in the Netflix example where millions of updates happen every minute. We recommend either batching updates using the , which adds latency to your workload, or looking at alternative solutions when faced with frequent updates in Elasticsearch. Bulk API Rockset, a search and analytics database built in the cloud, is a mutable alternative to Elasticsearch. Being built on , a key-value store popularized for its mutability, Rockset can make in-place updates to documents. This results in only the value of individual fields being updated and reindexed rather than the entire document. RocksDB If you’d like to compare the performance of Elasticsearch and Rockset for update-heavy workloads, you can start a with $300 in credits. free trial of Rockset

A Simple Guide for Updating Documents in Elasticsearch

ইলাস্টিকসার্চে ডকুমেন্ট আপডেট করার জন্য একটি সহজ গাইড

Eine einfache Anleitung zum Aktualisieren von Dokumenten in Elasticsearch

Una guía sencilla para actualizar documentos en Elasticsearch

Un guide simple pour mettre à jour des documents dans Elasticsearch

इलास्टिकसर्च में दस्तावेज़ों को अपडेट करने के लिए एक सरल गाइड

Elasticsearch でドキュメントを更新するための簡単なガイド

Elasticsearch에서 문서 업데이트를 위한 간단한 가이드

Um guia simples para atualização de documentos no Elasticsearch

Простое руководство по обновлению документов в Elasticsearch

Elasticsearch'te Belgeleri Güncellemek İçin Basit Bir Kılavuz

Hướng dẫn đơn giản để cập nhật tài liệu trong Elaticsearch

Elasticsearch 中更新文档的简单指南

When working with local files, you might need to open, create, or update Zip files. In this article, we will learn how to work with Zip files in C#. We will learn how to perform basic operations such as opening, extracting, and creating a Zip file. The main class we will use is named and comes from the namespace. It’s been present in since .NET Framework 4.5, so we can say it’s pretty stable 😉 Nevertheless, . Let’s learn! ZipFile System.IO.Compression C# there are some tricky points that you need to know before using this class Using C# to list all items in a Zip file Once you have a Zip file, you can access the internal items without extracting the whole Zip. You can use the method. ZipFile.Open using ZipArchive archive = ZipFile.Open(zipFilePath, ZipArchiveMode.Read);
System.Collections.ObjectModel.ReadOnlyCollection<ZipArchiveEntry> entries = archive.Entries; Notice that I specified the . This is an Enum whose values are , , and . ZipArchiveMode Read Create Update Using the property of the , you can access the whole list of files stored within the Zip folder, each represented by a instance. Entries ZipArchive ZipArchiveEntry The object contains several fields, like the file’s name and the full path from the root archive. ZipArchiveEntry There are a few about the entries listed in the . key points to remember ZipArchiveEntry It is a : it means that even if you find a way to add or update the items in memory, the changes are not applied to the actual files; ReadOnlyCollection<ZipArchiveEntry> , not only those at the root level. As you can see from the image above, it lists both the files at the root level, like , and those in inner folders, such as ; It lists all files and folders File.txt TestZip/InnerFolder/presentation.pptx Each file is characterized by two similar but different properties: is the actual file name (like ), while contains the path from the root of the archive (e.g. ); Name presentation.pptx FullName TestZip/InnerFolder/presentation.pptx It lists : in the image above, you can see . You can recognize them because their property is empty, and their is 0; folders as if they were files TestZip/InnerFolder Name Length Lastly, remember that returns an , so you should place the operations within a statement. ZipFile.Open IDisposable using ❓❓A question for you! Why do we see an item for the folder, but there is no reference to the folder? Drop a comment below 📩 TestZip/InnerFolder TestZip Using C# to extract the Zip to a local path Extracting a Zip folder is easy but not obvious. We have only one way to do that: by calling the method. ZipFile.ExtractToDirectory It accepts as mandatory parameters the path of the Zip file to be extracted and the path to the destination: var zipPath = @"C:\Users\d.bellone\Desktop\TestZip.zip";
var destinationPath = @"C:\Users\d.bellone\Desktop\MyDestination";
ZipFile.ExtractToDirectory(zipPath, destinationPath); Once you run it, you will see the content of the Zip copied and extracted to the folder. MyDestination Note that if it does not exist. this method creates the destination folder This method accepts two more parameters: , by which you can specify the encoding. . entryNameEncoding The default value is UTF-8 allows you to specify whether it must overwrite existing files. The default value is . If set to false and the destination files already exist, this method throws a saying that the file already exists. overwriteFiles false System.IO.IOException Using C# to create a Zip from a folder The key method here is , which allows you to create Zip files in a flexible way. ZipFile.CreateFromDirectory The first mandatory value is, of course, the . source directory path The second mandatory parameter is the destination of the resulting Zip file. It can be the to the file: local path string sourceFolderPath = @"\Desktop\myFolder";
string destinationZipPath = @"\Desktop\destinationFile.zip";

ZipFile.CreateFromDirectory(sourceFolderPath, destinationZipPath); Or it can be a that you can use later for other operations: Stream using (MemoryStream memStream = new MemoryStream())
{
    string sourceFolderPath = @"\Desktop\myFolder";
    ZipFile.CreateFromDirectory(sourceFolderPath, memStream);

    var lenght = memStream.Length;// here the Stream is populated
} You can finally add some optional parameters: , whose values are , , , . compressionLevel Optimal Fastest NoCompression SmallestSize : a flag that defines if you have to copy only the first-level files or also the root folder. includeBaseDirectory A quick comparison of the four Compression Levels As we just saw, we have four compression levels: , , , and . Optimal Fastest NoCompression SmallestSize What happens if I use the different values to zip all the photos and videos of my latest trip? The source folder’s size is 16.2 GB. Let me zip it with the four compression levels: private long CreateAndTrack(string sourcePath, string destinationPath, CompressionLevel compression)
 {
     Stopwatch stopwatch = Stopwatch.StartNew();

     ZipFile.CreateFromDirectory(
         sourceDirectoryName: sourcePath,
         destinationArchiveFileName: destinationPath,
         compressionLevel: compression,
         includeBaseDirectory: true
         );
     stopwatch.Stop();

     return stopwatch.ElapsedMilliseconds;
 }

// in Main...

var smallestTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Smallest.zip"),
    CompressionLevel.SmallestSize);

var noCompressionTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "NoCompression.zip"),
    CompressionLevel.NoCompression);

var fastestTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Fastest.zip"),
    CompressionLevel.Fastest);

var optimalTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Optimal.zip"),
    CompressionLevel.Optimal); By executing this operation, we have this table: We can see a bunch of weird things: Fastest compression generates a smaller file than Smallest compression. Fastest compression is way slower than Smallest compression. Optimal lies in the middle. This is to say: don’t trust the names; , even with a test as simple as this. remember to benchmark the parts where you need performance Wrapping up This was a quick article about one specific class in the .NET ecosystem. As we saw, even though the class is simple and it’s all about three methods, there are some things you should keep in mind before using this class in your code. I hope you enjoyed this article! Let’s keep in touch on or ! 🤜🤛 Twitter LinkedIn Happy coding! 🐧 Also published . here

Zip Files in C#: How to Extract, Create, or Update Your Archives

Web applications have greatly benefited from the inclusion of maps, providing users with valuable location-based information. Maps have transformed our interaction with the world, from navigating unfamiliar places to discovering nearby eateries. As a result, the integration of maps into websites has become increasingly popular in recent times. Nevertheless, designing maps that are both functional and user-friendly can present a challenge, especially for those lacking experience in this domain. In this article, I will share useful tips on how to create effective maps within your browser. Tech stack Let’s discuss technologies. When working with maps, we typically use three layers: Rendering the user interface, which includes buttons and forms. In our stack, React performs this role; Rendering the map and enabling user interaction. We use Mapbox for this; Fetching data from the server, such as information about markers or polygons. To retrieve data, we use the browser’s built-in fetch feature. Let’s review each item to gain a better understanding of our technology stack when working with maps. React The library enables you to work with page elements conveniently and efficiently. It was developed by Facebook for their own use, and includes numerous components such as buttons, forms, and other interactive elements on the pages. For example, Facebook created a to quickly compare different states. React reconciliation algorithm As changing elements on a page is the most expensive operation for a browser, it’s essential to do it as efficiently as possible. To address this issue, Facebook engineers developed the React library, which enables fast and straightforward element changes on a page. Besides providing rapid state changes on a page, React allows us to do this declaratively without working with DOM elements directly. Instead, we use an abstraction, usually , which looks like HTML. Let’s consider an example: JSX // It's our state. Is the user our friend or not?
// false by default
const [isFriend, setIsFriend] = useState(false)

// Depending on the state, we show the text on the button
const buttonText = isFriend ? 'Your my Friend' : 'Add as Friend'

// There is JSX, syntax for UI
// In this case, we display a button, when clicked, we change the state
return (
  <button onClick={() => setIsFriend(true)}>{buttonText}</button>
) It is possible to nest components with ordinary DOM elements like forms, buttons, and inputs at the bottom of the hierarchy. By assembling these simple elements, we can create more complex ones, such as a complete form: const Form = () => (
  <form>
      <input name="Email"/>
      <input name="Password"/>
  </form>
)

const App = () => (
  <main>
      <h1>My form!</h1>
      <Form />
  </main>
) How does React assist us in the context of maps? As the map on the page is interactive, similar to a button or form, we aim to optimize its rendering and interaction through events like clicks on the map. React can help achieve this optimization. Here is an example of how it works: // Use React to render the map with different event handlers
// and render markers
return (
  <BaseMap
    onInitMap={() => console.log('I am alive!')}
    onClickMap={() => console.log('Click!')}
    onDestroyMap={() => console.log('Oh no!')}
  >
    <ClustersMarkers />
    <PostsMarkers />
    <ListingsMarkers />
  </BaseMap>
) When working with React, it is essential to remember that it allows for efficient manipulation of elements on the page, quick changes, and interaction with them through events. This is achieved through an abstraction that resembles HTML, making it easy to create complex components from simpler ones. Mapbox Now, let’s discuss the map itself. Creating and using maps can be challenging, and only a few product companies can design maps from scratch. Typically, most people rely on pre-made libraries with a user-friendly API that has been tried and tested. Numerous dynamic map providers are available, including Google Maps, Leaflet, Bing Maps, Mapbox, and more. However, we will focus on due to its extensive features and fair pricing policy. For instance, Google Maps charges for 100k map views per month, while Mapbox only charges . Moreover, Mapbox offers free access for up to 50k map views per month. Mapbox $700 $250 Mapbox offers , a tool often compared to Photoshop for maps. With this tool, designers can create custom styles, hide unnecessary map elements, and remove house displays. One of its benefits is the ability to enhance the visibility of . Another advantage is the opportunity to experiment with map styling in corporate colors. However, it’s essential to consider the user experience and maintain a familiar color palette, including green grass and blue water. The best part about Mapbox Studio is that it simplifies the map design process by eliminating the need for developing and transferring requirements, ultimately reducing the cost of working with maps. Mapbox Studio Points of Interest Additionally, Mapbox provides a that can convert addresses to coordinates or vice versa, simplifying the process of locating points on a map. However, the tool’s may be insufficient, leading to difficulties for certain projects. Mapbox gathers data from , such as governments, open data initiatives like OpenStreetMap, and private companies. geocoding tool worldwide coverage numerous sources Mapbox provides a variety of , although the could benefit from some improvement. For instance, when incorporating marker functionality, examples can be more informative than the documentation. Additionally, challenges may arise when merging multiple components, such as marker clusters, custom data loading, normalization, and marker state changes. practical examples documentation Let’s revisit Mapbox maps. What does the do? Mapbox API It initializes the map in an HTML element on the page; It loads and renders images that make up the map; It draws additional elements, such as markers, using GeoJson as input data; It generates events, such as clicks or zoom changes, that can be handled. Let’s take a closer look at each of these items. Mapbox specializes in map rendering using tiles. Tiles are small square images that make up the larger map. The default size of a tile is 512x512 pixels, and it can be either . tiles are used to display roads, buildings, Points of Interest (PoI), and more. They can be dynamically styled, are lightweight, and allow for smooth interaction with the map. tiles, on the other hand, are used for displaying satellite imagery. vector or raster Vector Raster Just so you know, Mapbox Studio allows us to choose the specific data we want to include in the map tiles. These tiles are then placed onto a , which is a special DOM element in the browser that displays images and other graphical elements on the web. To give you an idea, it’s similar to how documents are displayed on a canvas in Google Docs. canvas <canvas width="100" height="100" /> Mapbox handles the loading, insertion, and updating of tiles. All we have to do is specify where we want the map to be displayed and the initial conditions, like the zoom level or the map’s coordinates. To use Mapbox, you’ll need an , which is a unique key that can be found in your Mapbox account. To set up a basic map, here’s a quick example, but for more information, check out the provided: access token link mapboxgl.accessToken = 'YOUR_MAPBOX_ACCESS_TOKEN';

const map = new mapboxgl.Map({
  container: 'map', // we can use an Id or an element
  style: 'mapbox://styles/mapbox/streets-v11', // URL for styles
  center: [-74.5, 40], // initial coordinates [lng, lat]
  zoom: 9, // initial zoom
}); After this, we will get a map on the page in an element with the id ‘map.’ To provide users with more information on the map, we often display the location of a certain establishment or the boundaries of a specific area. To achieve this, we use a specific data format called and instruct Mapbox on how to showcase this data. GeoJSON GeoJSON is the standard format for storing geographic structures on maps. It can store various primitive types that describe geographic objects such as addresses, locations, streets, highways, borders, countries, states, and combinations of these, known as multipart. GeoJSON was introduced in 2008 and is represented like this: {
  "type": "Feature", // also can be FeatureCollection, it's collection of Feature
  "geometry": {
    "type": "Point", // also can be LineString, Polygon, MultiPolygon
    "coordinates": [125.6, 10.1] // for other types you can use Array with coordinates
  },
  "properties": { // it's metadata, we can you that to show something on the map
    "name": "Dinagat Islands"
  }
} Let’s talk about the coordinate system used in Mapbox. By default, Mapbox employs , which is a way of projecting the Earth onto a flat surface to work with tiles. This is known as Web Mercator and is the standard for online maps. However, when dealing with data about markers and polygons, it uses a different coordinate system called . This system relies on latitude and longitude to describe coordinates on the Earth’s ellipsoid. Mapbox and other map providers automatically convert coordinates from EPSG 4326 to EPSG 3857. If we need to work with different , we can use the method. EPSG 3857 EPSG 4326 projections map.setProjection Now, we will discuss how to display GeoJSON on the map. Mapbox offers two entities that we will find helpful: — This is the data seed. We can create a new source with GeoJSON data and configure it, for example, to generate an ID for each feature in the feature collection. Source — This is the data representation. We can display the data from the source in different ways, such as showing boundaries. Layer To display polygons or markers on the map, we must retrieve the data in GeoJson format from the server. Then, we create a source, input the data into it, and connect it to the required layer. const geoJsonFeature = {
  'type': 'Feature',
  'geometry': {
      'type': 'Polygon',
      'coordinates': [
          [-67.13734, 45.13745],
          [-66.96466, 44.8097],
          [-68.03252, 44.3252],
          [-67.13734, 45.13745]
      ]
  }
}

// Create source with our data
map.addSource('ourSource', {
  'type': 'geojson',
  'data': geoJsonFeature
});

// Add layer for background
map.addLayer({
  'id': 'background',
  'type': 'fill',
  'source': 'ourSource', // название нашего source
  'layout': {},
  'paint': {
      'fill-color': '#0080ff',
      'fill-opacity': 0.5
  }
});

// Add layer for border
map.addLayer({
  'id': 'border',
  'type': 'line',
  'source': 'ourSource',
  'layout': {},
  'paint': {
      'line-color': '#000',
      'line-width': 3
  }
}); After running this code, we get the result: To learn more about this topic, you can refer to the provided by Mapbox. In this tutorial, we have covered the process of initializing the map and presenting our data. Next, we will explore how to handle events like clicking, dragging, and zooming. For instance, we can use Mapbox to monitor events and display the coordinates in the console as the user moves their cursor on the map. To achieve this, we simply call the method with the event type we want, similar to how we work with DOM elements. documentation on map.on('mousemove', (e) => {
  console.log(JSON.stringify(e.point));
});

// Result: {"x":330,"y":49} In summary, what do we need to remember? Mapbox allows us to display a map, draw our data on top of it, and process map events. At the same time, Mapbox takes care of loading and displaying images (tiles). Fetch A word about . We have already seen how to render data on a map, but first, we must retrieve it from the server. When we dynamically request data from the server in the background without reloading the page, we call this approach (“ synchronous avaScript nd ML”). There are many tools for loading data asynchronously from the server, such as Axios or XMLHttpRequest (native). fetch AJAX A J a X What to remember? We retrieve data from the server, and there are many libraries for this, but we will use fetch. Next, we will look at how we do this specifically when working with maps, as there are nuances. React + Mapbox Now let’s see how the technologies described above work together. First, we will retrieve the data for displaying the polygon using fetch. Then we will declare the initialization of the map, and after it loads, we will add the polygon to the map. You can also find a working example at the provided. link const useFetch = () => {
  /*
  Our data
  {
      'type': 'Feature',
      'geometry': {
          'type': 'Polygon',
          'coordinates': [
              [
                  [-67.13734, 45.13745],
                  [-68.03252, 44.3252],
                  [-68.90478, 47.18479],
                  [-67.13734, 45.13745],
              ]
          ]
      }
  }
  */
  const [data, setData] = useState(null)

  useEffect(() => {
    fetch('https://our-api.com/polygon')
      .then(response => response.json())
      .then(setData)
      .catch(e => {
        console.error(e)
      })
  }, [setData])

  return { data }
}

const BaseMap = () => {
  // Use the hook to fetch data
  const { data } = useFetch(GET_REGION);

  // Map instance
  const map = useRef(null);
  // DOM element
  const mapContainer = useRef(null);

  // Main logic - init the map and add the event
  useEffect(() => {
    if (map.current) {
      return; // initialize map only once
    }

    mapboxgl.accessToken = 'YOUR_MAPBOX_ACCESS_TOKEN';
    map.current = new mapboxgl.Map({
      container: mapContainer.current,
      style: 'mapbox://styles/mapbox/light-v10', // style URL (it's Mapbox's core style)
      center: [-68.137343, 45.137451], // starting position
      zoom: 5 // starting zoom
    });

    // Handle event
    map.on('load', () => {
      const sourceId = 'source-region'

      // Add a data source containing GeoJSON data
      map.addSource(sourceId, {
        'type': 'geojson',
        'data': data.region // our data from Apollo
      });

      // Add a new layer to visualize the polygon
      map.addLayer({
        'id': 'background',
        'type': 'fill',
        'source': sourceId, // reference the data source
        'paint': {
            'fill-color': '#0080ff', // blue color fill
            'fill-opacity': 0.5
        }
      });
      // Add a black outline around the polygon
      map.addLayer({
        'id': 'outline',
        'type': 'line',
        'source': sourceId,
        'paint': {
            'line-color': '#000',
            'line-width': 3
        }
      });
    });
  });

  return <div ref={mapContainer} />;
} Conclusion We looked at the technology stack that underpins our future architecture. In the following article, we will discuss the principles that help design a map architecture, how to achieve maximum low coupling and high cohesion of modules, and how to maintain and develop a scalable map system. Thank you very much for your attention! Have a great day.

Maps on the Web Using React and Mapbox

প্রতিক্রিয়া এবং ম্যাপবক্স ব্যবহার করে ওয়েবে মানচিত্র

Karten im Web mit React und Mapbox

Mapas en la Web usando React y Mapbox

Cartes sur le Web à l'aide de React et Mapbox

रिएक्ट और मैपबॉक्स का उपयोग करके वेब पर मानचित्र

React と Mapbox を使用した Web 上の地図

React와 Mapbox를 사용하여 웹에서 지도 만들기

Mapas na Web usando React e Mapbox

Карты в Интернете с использованием React и Mapbox

React ve Mapbox Kullanarak Web'deki Haritalar

Bản đồ trên web bằng React và Mapbox

使用 React 和 Mapbox 制作网络地图

Form manipulation is a typical task for a Front-End engineer. They are used everywhere - from simple feedback forms to a complex ones for creating something big. While you are using any of the UI frameworks or libraries, they offer us a convenient way to create forms of any complexity easily and quickly. With Angular, we have two options for form building: template-driven and reactive forms. Both of them can be applied in different scenarios. For example, if we need to create some simple form with primitive (or even without) validation, we can consider a template-driven form as a good candidate. Another case is when we need to build something more complex than a login form, with tons of validation, field dependencies from another field, etc. In this way, reactive forms come on the stage. But even with this powerful tool, we sometimes need to build some functionality that is not available from the box. In this article, I will show one of the possible cases, and you can apply this approach to your project. Let's cut to the chase and move to the problem: we need to update a form field based on a value from another form field. Moreover, to enhance the application experience and not confuse a user, we want to highlight (put some message and color the field) the updated field. The fields can be any kind of element (input, dropdown, date picker, etc.), and any combination of field values can lead to the updation of another field. How can we solve this problem? There are some possible ways: We can write a directive for it. It will be used in any control you want despite your choice of form type. But it's quite a complicated task to support this directive in the future, and possibly, it will go for an awkward legacy code that nobody would like to extend or fix some issue related to this; Use EventEmitter. We can observe the expected condition for updating a target field and emit a value. When we do it - update the target field with the emitted value (or you can use this like a trigger and set any value you want). This option is better than the previous one, but it still requires a lot of manual work, and it won't be easy to reuse it somewhere - we will have to implement this mechanism for each case by hand; Extend the FormControl class. In functionality, which is available from the box, we won't find any methods or fields (or even a combination of them) to check that the form control was updated by another one. Let's create it! First things first, we need to create a class for our new control, which extends the FormControl class: import { FormControl } from '@angular/forms';

export class AutoUpdateFormControl<TValue> extends FormControl {
  constructor() {
    super();
  }
} Then, to add a new functionality, we need to write our method for the control updating and mark the inner state as auto-updated: import { FormControl } from '@angular/forms';

export class AutoUpdateFormControl<TValue> extends FormControl {
    public isAutoUpdated: boolean = false;

    constructor() {
        super();
    }

    public autoUpdateValue(value: any, options?: { emitEvent: boolean }): void {
        super.setValue(value, options);
        this.markAsAutoUpdated();
    }

    public setValue(
        value: TValue,
        options?: {
            onlySelf?: boolean;
            emitEvent?: boolean;
            emitModelToViewChange?: boolean;
            emitViewToModelChange?: boolean;
        },
    ): void {
        super.setValue(value, options);
        this.markAsNotAutoUpdated();
    }

    public markAsAutoUpdated(): void {
        this.isAutoUpdated = true;
    }

    public markAsNotAutoUpdated(): void {
        this.isAutoUpdated = false;
    }
} An example of usage: Create a form: interface IForm {
  sourceField: AbstractControl<string>;
  targetField: AutoUpdateFormControl<string>;
}

const form = new FormGroup<IForm>({
    sourceField: new FormControl(''),
    targetField: new AutoUpdateFormControl(),
}); Subscribe to the control and check the required value. If matches - call the autoUpdateValue method. It sets the control value and updates the control field: sourceField form.controls.sourceField.valueChanges.subscribe((value) => {
    if (value === 'test') {
        this.form.controls.testField.autoUpdateValue('updated field');
    }
}); We can add mark-up for handling field highlighting and showing the text depending on the control field: <span *ngIf="form.controls.targetField.isAutoUpdated">Auto-updated</span> If you use some custom components or the wrappers around the third-party ones, you can directly add this kind of template into the component to not repeat it again and again. I found this solution perfect for my needs and decided to share it. Perhaps, it will help someone. But if you want to go further, remember: Your custom form control should remain compatible with Angular's forms API. Avoid altering the existing behavior of FormControl unless necessary; Test your custom form control thoroughly, especially if you're also overriding any of the existing methods; Document your custom methods and any changes to standard behavior to help other developers in your team understand your custom control. With these bits of advice, you can create your custom controls safely and with pleasure.

How to Extend the FormControl Functionality in Angular

Data is widely considered the lifeblood of an organization; however, it is worthless—and useless—in its raw form. Help is needed to turn this data into life-giving information. Part of this help is to move this data from its source to destination via data pipelines. But managing the end-to-end data pipeline lifecycle is not easy: ones are challenging to scale on demand, others can result in delays in detecting and addressing issues or be difficult to control. These and the other problems may be solved in the declarative programming paradigm, and in this article we will see it on Terraform. Where Does This Data Come From? The data in these extensive datasets come from multiple disparate sources and are typically loaded into datasets in a data storage like ClickHouse via data pipelines. These pipelines play a critical role in this process by efficiently moving, transforming, and processing the data. For instance, imagine you own a data analytics startup that provides organizations with real-time data analytics. Your client owns an eBike hire startup that wants to integrate weather data into its existing data to inform its marketing messaging strategies. Undoubtedly, your data engineers will set up many different data pipelines, pulling data from many sources. The two we are concerned with are designed to extract data from your client’s CRM and the local weather station. The Layers of the Modern Data Stack A data stack is a collection of different technologies, logically stacked on top of each other, that provides end-to-end data processing capabilities. As described here, a data stack can have many different layers. The modern data stack consists of ingestion (data pipelines), storage (OLAP database), and business intelligence (data analytics) layers. Lastly, there is an orchestration layer—for example, Kubernetes or Docker Compose—that sits on top of these layers, orchestrating everything together so that the stack does what it is supposed to do; ergo, ingest raw data and egest advanced predictive analytics. Data Pipelines are typically structured either as Extract, Transform, Load (ETL) or Extract, Load Transform (ELT) pipelines. However, developing, monitoring, and managing these data pipelines, especially in international enterprises with many branches worldwide, can be highly challenging, especially if the end-to-end data pipeline monitoring process is manual. Developing Data Pipelines, the Traditional Way Data pipelines are broadly categorized into two groups: code-based and deployed in a Direct Acyclic Graph or DAG-like tools such as Airflow and Apache Fink or non-code-based, which are  often developed in a SaaS-based application via a drag-and-drop UI. Both have their own problems. Code-based data pipelines are challenging to scale on demand. The development/test/release cycle is typically too long, especially in critical environments with thousands of pipelines, loading petabytes of data into data stores daily. Additionally, they are typically manually implemented, deployed, and monitored by human data engineers, allowing the risk of errors creeping in and significant downtime. With non-code-based data pipelines engineers don’t have access to the backend code, there can be limited visibility and monitoring, resulting in delays in detecting and addressing issues and failures. Moreover, it is challenging to reproduce errors, as making an exact copy of the pipeline is impossible. Lastly, version control using a tool like Git is complex. It is harder to control the evolution of a pipeline as the code is not stored in a repository such as a GitHub repo. The Solution: Improving Your Data Pipelines with Declarative, Reproducible, Modular Engineering The good news is that there is a sustainable solution to these challenges. It is important to remind ourselves of the difference between imperative and declarative programming. In imperative programming, you control how things happen. In declarative programming, you express the logic without specifying the control flow. The answer to the end-to-end data pipeline development, deployment, monitoring, and maintenance challenges is to utilize the declarative programming paradigm, that is, abstracting the computation logic. In practice, this is best achieved by using Terraform by HashiCorp — an open-source project that is an infrastructure-as-code development tool. Because modern world data-intensive application architecture is very similar to the data stack the applications run on, Terraform provides the framework that will make your data stack declarative, reproducible, and modular. Let’s return to our data analytics startup example described above to elucidate these concepts further. Imagine your data analytics app is a web application that executes inside Terraform. As expected, the most significant part of the application is the ClickHouse database. We must build two data pipelines to ETL data from the CRM system—stored in a PostgreSQL database—into the ClickHouse database, transforming and enriching the data before loading it into ClickHouse. The second pipeline extracts real-time weather data from the local weather service—through an API and transforms it—ensuring there are no errors and enriching it—before loading it into ClickHouse. Using Terraform to Code the Data Pipelines Classic infrastructure for modern cloud-based applications looks like: This is an oversimplification, but a more or less solid way of building applications. You have a public subnet with a user facing API / UI, which is connected to a private subnet where data is stored. To simplify things, let’s look at how to use Terraform to build the first data pipeline—to ETL data from your RDBMS (in this case PostgreSQL) system into the ClickHouse database. This diagram visualizes the data pipeline that ELTs the data from the RDBMS database into the ClickHouse database, and then exposed as a connection into Visualization service. This workflow must be reproducible as your company’s information technology architecture has at least three environments: dev, QA, and prod. Enter Terraform. Let’s look at how to implement this data pipeline in Terraform: It is best practice to develop modular code, distilled down to the smallest unit, which can be reused many times. The first step is to start with a , containing nothing more than the provider’s definition. main.tf file provider "doublecloud" {
  endpoint       = "api.double.cloud:443"
  authorized_key = file(var.dc-token)
}
provider "aws" {
  profile = var.profile
} The second step is to create a BYOA netwok: data "aws_caller_identity" "self" {}

data "aws_region" "self" {}

# Prepare BYOC VPC and IAM Role
module "doublecloud_byoc" {
  source  = "doublecloud/doublecloud-byoc/aws"
  version = "1.0.2"
  providers = {
    aws = aws
  }
  ipv4_cidr = var.vpc_cidr_block
}

# Create VPC to peer with
resource "aws_vpc" "peered" {
  cidr_block                       = var.dwh_ipv4_cidr
  provider                         = aws
}

# Get account ID to peer with
data "aws_caller_identity" "peered" {
  provider = aws
}

# Create DoubleCloud BYOC Network
resource "doublecloud_network" "aws" {
  project_id = var.dc_project_id
  name       = "alpha-network"
  region_id  = module.doublecloud_byoc.region_id
  cloud_type = "aws"
  aws = {
    vpc_id          = module.doublecloud_byoc.vpc_id
    account_id      = module.doublecloud_byoc.account_id
    iam_role_arn    = module.doublecloud_byoc.iam_role_arn
    private_subnets = true
  }
}

# Create VPC Peering from DoubleCloud Network to AWS VPC
resource "doublecloud_network_connection" "example" {
  network_id = doublecloud_network.aws.id
  aws = {
    peering = {
      vpc_id          = aws_vpc.peered.id
      account_id      = data.aws_caller_identity.peered.account_id
      region_id       = var.aws_region
      ipv4_cidr_block = aws_vpc.peered.cidr_block
      ipv6_cidr_block = aws_vpc.peered.ipv6_cidr_block
    }
  }
}

# Accept Peering Request on AWS side
resource "aws_vpc_peering_connection_accepter" "own" {
  provider                  = aws
  vpc_peering_connection_id = time_sleep.avoid_aws_race.triggers["peering_connection_id"]
  auto_accept               = true
}

# Confirm Peering creation
resource "doublecloud_network_connection_accepter" "accept" {
  id = doublecloud_network_connection.example.id

  depends_on = [
    aws_vpc_peering_connection_accepter.own,
  ]
}

# Create ipv4 routes to DoubleCloud Network
resource "aws_route" "ipv4" {
  provider                  = aws
  route_table_id            = aws_vpc.peered.main_route_table_id
  destination_cidr_block    = doublecloud_network_connection.example.aws.peering.managed_ipv4_cidr_block
  vpc_peering_connection_id = time_sleep.avoid_aws_race.triggers["peering_connection_id"]
}

# Sleep to avoid AWS InvalidVpcPeeringConnectionID.NotFound error
resource "time_sleep" "avoid_aws_race" {
  create_duration = "30s"

  triggers = {
    peering_connection_id = doublecloud_network_connection.example.aws.peering.peering_connection_id
  }
} This is preparing our Stage for later clusters. Architecture after terraform apply looks follows: Once we prepare a stage, we can create a cluster in this private-subnet: resource "doublecloud_clickhouse_cluster" "alpha-clickhouse" {
  project_id = var.dc_project_id
  name       = "alpha-clickhouse"
  region_id  = var.aws_region
  cloud_type = "aws"
  network_id = doublecloud_network.aws.id

  resources {
    clickhouse {
      resource_preset_id = "s1-c2-m4"
      disk_size          = 34359738368
      replica_count      = 1
    }
  }

  config {
    log_level       = "LOG_LEVEL_TRACE"
    max_connections = 120
  }

  access {
    ipv4_cidr_blocks = [
      {
        value       = doublecloud_network.aws.ipv4_cidr_block
        description = "DC Network interconnection"
      },
      {
        value       = aws_vpc.tutorial_vpc.cidr_block
        description = "Peered VPC"
      },
      {
        value       = "${var.my_ip}/32"
        description = "My IP"
      }
    ]
    ipv6_cidr_blocks = [
      {
        value       = "${var.my_ipv6}/128"
        description = "My IPv6"
      }
    ]
  }
} This is added to our Stage for simple Clickhouse Cluster. But this cluster still empty, so we must enable transfer between PostgreSQL and ClickHouse: resource "doublecloud_transfer_endpoint" "pg-source" {
  name       = "chinook-pg-source"
  project_id = var.dc_project_id
  settings {
    postgres_source {
      connection {
        on_premise {
          tls_mode {
            ca_certificate = file("global-bundle.pem")
          }
          hosts = [
            aws_db_instance.tutorial_database.address
          ]
          port = 5432
        }
      }
      database = aws_db_instance.tutorial_database.db_name
      user     = aws_db_instance.tutorial_database.username
      password = var.db_password
    }
  }
}

data "doublecloud_clickhouse" "dwh" {
  name       = doublecloud_clickhouse_cluster.alpha-clickhouse.name
  project_id = var.dc_project_id
}

resource "doublecloud_transfer_endpoint" "dwh-target" {
  name       = "alpha-clickhouse-target"
  project_id = var.dc_project_id
  settings {
    clickhouse_target {
      connection {
        address {
          cluster_id = doublecloud_clickhouse_cluster.alpha-clickhouse.id
        }
        database = "default"
        user     = data.doublecloud_clickhouse.dwh.connection_info.user
        password = data.doublecloud_clickhouse.dwh.connection_info.password
      }
    }
  }
}

resource "doublecloud_transfer" "pg2ch" {
  name       = "postgres-to-clickhouse-snapshot"
  project_id = var.dc_project_id
  source     = doublecloud_transfer_endpoint.pg-source.id
  target     = doublecloud_transfer_endpoint.dwh-target.id
  type       = "SNAPSHOT_ONLY"
  activated  = false
} This is adding a ELT snapshot process between PostgreSQL and ClickHouse. As a last piece we can add a connection to this newly created cluster to visualization service: resource "doublecloud_workbook" "dwh-viewer" {
  project_id = var.dc_project_id
  title      = "DWH Viewer"

  config = jsonencode({
    "datasets" : [],
    "charts" : [],
    "dashboards" : []
  })

  connect {
    name = "main"
    config = jsonencode({
      kind          = "clickhouse"
      cache_ttl_sec = 600
      host          = data.doublecloud_clickhouse.dwh.connection_info.host
      port          = 8443
      username      = data.doublecloud_clickhouse.dwh.connection_info.user
      secure        = true
      raw_sql_level = "off"
    })
    secret = data.doublecloud_clickhouse.dwh.connection_info.password
  }
} As you can see, most of the code consists of variables. Therefore, executing this code to set up the different environments is easy by adding a file and running with it: stage_name.tfvars terraform apply See the for more information on .tvars files. Terraform documentation // This variable is to set the
// AWS region that everything will be
// created in
variable "aws_region" {
  default = "eu-west-2" // london
}

// This variable is to set the
// CIDR block for the VPC
variable "vpc_cidr_block" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

// This variable holds the
// number of public and private subnets
variable "subnet_count" {
  description = "Number of subnets"
  type        = map(number)
  default = {
    public  = 1,
    private = 2
  }
}

// This variable contains the configuration
// settings for the EC2 and RDS instances
variable "settings" {
  description = "Configuration settings"
  type        = map(any)
  default = {
    "database" = {
      allocated_storage   = 10            // storage in gigabytes
      engine              = "postgres"    // engine type
      engine_version      = "15.4"        // engine version
      instance_class      = "db.t3.micro" // rds instance type
      db_name             = "chinook"     // database name
      identifier           = "chinook"     // database identifier
      skip_final_snapshot = true
    },
    "web_app" = {
      count         = 1          // the number of EC2 instances
      instance_type = "t3.micro" // the EC2 instance
    }
  }
}

// This variable contains the CIDR blocks for
// the public subnet. I have only included 4
// for this tutorial, but if you need more you
// would add them here
variable "public_subnet_cidr_blocks" {
  description = "Available CIDR blocks for public subnets"
  type        = list(string)
  default = [
    "10.0.1.0/24",
    "10.0.2.0/24",
    "10.0.3.0/24",
    "10.0.4.0/24"
  ]
}

// This variable contains the CIDR blocks for
// the public subnet. I have only included 4
// for this tutorial, but if you need more you
// would add them here
variable "private_subnet_cidr_blocks" {
  description = "Available CIDR blocks for private subnets"
  type        = list(string)
  default = [
    "10.0.101.0/24",
    "10.0.102.0/24",
    "10.0.103.0/24",
    "10.0.104.0/24",
  ]
}

// This variable contains your IP address. This
// is used when setting up the SSH rule on the
// web security group
variable "my_ip" {
  description = "Your IP address"
  type        = string
  sensitive   = true
}
// This variable contains your IP address. This
// is used when setting up the SSH rule on the
// web security group
variable "my_ipv6" {
  description = "Your IPv6 address"
  type        = string
  sensitive   = true
}

// This variable contains the database master user
// We will be storing this in a secrets file
variable "db_username" {
  description = "Database master user"
  type        = string
  sensitive   = true
}

// This variable contains the database master password
// We will be storing this in a secrets file
variable "db_password" {
  description = "Database master user password"
  type        = string
  sensitive   = true
}

// Stage 2 Variables

variable "dwh_ipv4_cidr" {
  type        = string
  description = "CIDR of a used vpc"
  default     = "172.16.0.0/16"
}
variable "dc_project_id" {
  type        = string
  description = "ID of the DoubleCloud project in which to create resources"
} In Conclusion… There you have it. A small modern data stack that ETLs data from a PostgreSQL database into a ClickHouse database. The best news is that you can set up multiple data pipelines—as many as you need—using the code in this example. There is also a complete example that you can find , with additional features like adding a peer connection to your existing VPC and creating a sample ClickHouse database with a replication transfer between the two databases. here Have fun playing with your small modern data stack! This article was also . published on DZone

SOURCEWIZ

sourcewiz.co

-- employees

Since n.d.

COMPANY RANKING

SOURCEWIZ

EVERGREEN INDEX #9216

Representing this company? Click here to claim and customize this page!

Sourcewiz WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SOURCEWIZ

sourcewiz.co

-- employees

Since n.d.

COMPANY RANKING

SOURCEWIZ

EVERGREEN INDEX #9216

RELATED COMPANIESRANK

STORIES ABOUT SOURCEWIZ

Representing this company? Click here to claim and customize this page!

Sourcewiz WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!