Since launching out of TechStars Chicago in 2016, ConvertFlow has grown into a no-code platform used by 30,000+ marketing teams and agencies around the world, all while being powered by a small team

ConvertFlow

convertflow

Chart structure Splitting grid and traces Grid Color Detection Trace Pixel Classification Chart template detection Text Extraction Text recognition Template matching Axis Matching Pen Trajectory Evaluation Shear compensation Improving envelope Outlier removal Pen trajectory evaluation Conclusion Flow charts are widely used in gas industry. They record parameters of a gas flow (like pressure) on a paper disc. This data is then used to calculate gas volume passed through a pipe. The chart recorder contains two or three pens driven by sensors. These pens plot traces on a rotating paper disk. The disk performs one revolution per day or week, rarely per month. The disk is replaced after one revolution. Currently disks are processed manually. The operator uses the vector graphics editor to convert the scanned bitmap to contours and then to tabular data. This approach relies heavily on the operator’s attention and experience. There is a high probability of mistakes. After all, such job is really boring! We have developed an algorithm that performs this job automatically. Given a bitmap with a scanned chart, it produces tabular representation of chart data. Chart structure The template on the disc (usually green) is pre-drawn. It contains: Radial arcs for time; Concentric circles for quantity levels; Names of quantities (axes); Quantity values for concentric lines (ranges); Time marks: hours and week day names. Note that the same concentric line has different values for different axes. Splitting grid and traces Grid Color Detection The first task to solve is to separate trace pixels and pixels belonging to the blank area. This blank area contains two dominant colors: white and green. In theory, knowing this we could use simple thresholding to detect these two colors. But in practice this approach drops pixels on the edge between the grid and the blank paper. This happens because boundary pixels aren’t strictly white or green — they are somewhere between. To deal with it, a linear model is used. To build such model, we need an area where no trace appears. Such area is the outer ring of the chart, named “Time area” on the image. We take histograms for red-green, green-blue and blue-red correspondence and then build linear models for each. After that we evaluate the residual threshold for every pair using triple standard deviation. All pixels passing this threshold are labeled as non-trace pixels. Trace Pixel Classification The next step is to determine what trace every trace pixel belongs to. This is done using histograms of hue-saturation space. We perform clustering of h-s blobs and then the back-project result to pixels. To improve the result, we build the histogram using pixels whose bottom-right neighbours have similar color. Since we need the hue difference between pixels, all pixels with similar color should have a similar hue. This is not true for red pixels because they have hue values near both 0 and 180. To tackle this issue, we roll hue values to shift the significant part of the hue histogram to the center. The shift value is found as a minima of the convolution between the hue histogram and the kernel shown below. The idea behind is to “penalize” cases when large histogram values are near 0 or 180. After the histogram is shifted, we find trace pixels whose neighbours have similar hue. This step makes blobs’ edges on the h-s histogram sharper, because pixels near the trace edges are excluded. Similar pixels are found in the next way: Get all trace pixels whose bottom-right pixels are in trace too; Find the 95% percentile of the hue difference; Keep pairs with the hue difference below this level. Then, the mean hue and saturation values are taken for every neighbour pair. The obtained set of the hue and saturation values is used to build ‘filtered’ hue-saturation histogram. This histogram contains less noise produced by the ‘edge’ pixels. After that, the histogram is thresholded by the median value and connected components are found. We build Voronoi diagram and assign its labels to the source histogram. To make the result more robust, we mark the histogram bins below the 10% percentile as unlabeled. Now we get labeled trace pixels like this: There are some mislabeled pixels. To fix them, we perform the following steps: For every possible label we create a binary mask of pixels marked with this label; Gaussian blur is applied to every mask; For every trace pixel we take the index of the mask with the largest value. These steps change labels for the outlier pixels replacing them with the label of surrounding pixels: Chart template detection To determine the chart template we perform the following steps: Recognize all the text presented on the image; Detect axes names, level marks and time marks; Choose the template matching the obtained set of labels. Text Extraction First, we extract all the text char by char, group character images to labels and recognize them. In depth, the next steps are performed: The image is binarized using Otsu method. All trace pixels are made black. So, only the grid and text remain. Connected components are found. Characters are detected using the area and density (the ratio between the area and bounding box area) thresholds and then grouped to text instances. Here is a plot of all text instances found on a typical chart: Text recognition Every text instance is rotated to the horizontal position and passed to Tesseract. First, apply the mask based on the text instance geometry. Then we rotate the image to the horizontal position. To remove the redundant blobs near the text, we draw a line through the middle and use the flood fill to remove the redundant parts. Then we restore the middle row, invert the image and pass it to Tesseract. Template matching Having all text recognized, we find a chart template with the best matching. First, we define a list of all known chart templates. Every template is described by: The number of hours it contains (24 for day, 168 for week, etc.); Text marks appearing on the outer ring; A list of axes; every axis contains: a. Text label; b. Values presented on a chart; c. Possible pen colors. To find the template used on the given image, we check matching of the axis names and range labels in the next way: Group all range labels found on the image by the nearest (by angle) axis name label; Cyclically shift groups to handle combinations with a different starting group. This allows us to handle non-periodical axis sets like ABCABCAB. For every shift: a. Form strings “<axis_name><range_label_1><range_label_2>…” for every group; b. Form a similar string for every axis in the candidate template; c. Calculate string similarity for every pair group-axis; d. Get mean similarity through all pairs; 3. Resulting matching quality is a maximum among all shift values; 4. Take the template with the best quality. After the template is detected, its rotation angle on the image is found by the following steps: Take all labels found in the outer ring of the image; Take the same labels from the detected template; For every found label, expected label and rotation angle, find label-to-label matching quality as a product of string matching ratio and negated angular distance between the expected label and found label, rotated on a given angle. Negation makes the second part of the product increase when the angular distance tends to zero; For every found label and rotation angle, get the best matching expected label; Find a rotation angle with the maximum sum of matching qualities. Axis Matching After template matching we need to find the trace-to-axis correspondence. This is done by color matching. For every trace found a mean color is taken. Then for every possible pair (trace, axis) we find Cartesian distance in RGB space between the mean trace color and the known axis color. Some axes might have two possible pen colors. In this case, the minimal value is taken. Having such distances for all trace-to-axis pairs, we find the best matching by simply iterating all possible correspondences. Pen Trajectory Evaluation So, we have trace points classified by trace. We also know what template was used. This allows us to convert the image coordinates (in pixels) of every point to time and value (in hours and appropriate units: kPa, ℃, etc.). As a result, we get a set of points. If we arrange them in order by time, we have our task solved, right? No… Because now we have tons of points that tell us nothing about the pen’s movement. Let’s look at pixels in time-value plane and a corresponding chart fragment: It is easy to tell how the pen moved on the left and right ends of this excerpt, because we clearly see the line. But what about the middle part? How did the pen travel here? Intuitively, we can say that the pen moved up and down, shifting right a little every time. But how to restore this movement? To solve this task, the next steps are performed. Shear compensation First, we remove inner-points. Only border points are kept. This dramatically decreases the number of points for further processing. After that, we compensate the trace shear. This shear happens when the pen is placed incorrectly or is not fixed in the plotting head. To tackle it, we iterate over possible shear coefficients. For each one, we find an area of min-max envelope. The coefficient with the smallest result is chosen and applied to trace points. The image below shows the bounding area for shearing coefficients -0.01 and 0.01. Here is the result of shearing with the optimal coefficient: Improving envelope Next, we improve the obtained min-max envelope. Initially, all bins are uniformly distributed. Sometimes thin point “columns” appear on the edge between the bins, producing two bins instead of one. Simple movement of all bins won’t help us, because having one “column” aligned, we might get another one on an edge. Below is an example of two peaks and the envelope with step 2. To solve this issue, we improve the position of the bins by means of the next steps: Generate 10 envelopes for different bin offsets; Get max-to-min differences for each one; Apply three-item sliding mean; Having 10xN matrix and treating it as cost, we use forward-backward algorithm to obtain the sequence of indices (“path”) that minimizes the total area of the envelope. After these steps, we get the envelope with a smaller area than the initial one: Outlier removal Next, we remove the outliers. Sometimes grid pixels, hand marks or other things are mislabeled as trace points. For example, here we have added three outlier blobs: Applying the min-max envelope gives us an inadequate result: To fix this, we split high blocks into vertical “chunks” with a limited height. Now for every bin more than one chunk can exist. For example, here are three such cases: After that we find optimal indices of chunks for every vertical block. We use forward-backward algorithm with a vertical distance between the neighbouring chunks as cost function. This gives us a sequence of chunks covering inlier points. Next, we get an optimal envelope for inliers: Pen trajectory evaluation Having done this, we are ready to evaluate the pen trajectory. We assume the linear pen motion inside every bin, changing the movement direction between the bins. The goal is to maximize the area of the bin covered by the pen motion. For example, here is a possible envelope, pen path and coverage ratio for every bin: To maximize the coverage area, we use forward-backward algorithm with ten points per bin edge as possible states and one minus bin coverage as the cost of step. Here is the result applied to the example above: And to the real-world example: The red line on the image above shows the evaluated pen trajectory. It follows the thin line where needed and covers dense point areas using the up-down movement. Conclusion The algorithm described above successfully handles a wide range of input data from thin lines to dense “hatching”. The produced data allows us to calculate the gas flow characteristics without any human interaction, increasing the robustness of the whole chart handling process. By Ilya Sivkov

ChartRecorder: Computer Vision for Gas Flow Charts

Thank you to Tom and Tony for interviewing , , , , , , , , , and on . I’ve included all below — enjoy! Bright Brightwork CompleteSet ConvertFlow Fitbot Jiobit LogicGate Pak’d PartySlate Stationfy Bootstrapping in America ten interviews is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Techstars Chicago 2016 on Bootstrapping in America

In this article, I’ll answer how to format a string as Base64 in CSharp. We’ll be exploring C# string format and base64 encoding and how they can be used together to produce meaningful results. I’ll cover the benefits and drawbacks of this approach and best practices for using it in your code! Understanding C# string format and base64 encoding is important because they are commonly used in software engineering, and knowledge of these concepts will allow you to write more efficient and effective code. By using examples, I’ll demonstrate how these concepts can be applied in real-life scenarios, making it easier for you to understand and apply them to your own work. Let’s dive into C# string format and base64 encoding! Overview of C# String Format C# string format is a powerful tool for manipulating strings in code. The format method allows you to insert variable values into a string in a way that is more flexible and readable than simple concatenation. Not only is this beneficial for displaying information to a user, but this can be helpful when formatting strings properly for programmatic consumption as well. Simple C# String Format Examples A basic example of string format looks like this: string name = "John";
int age = 30;
string message = string.Format("My name is {0} and I am {1} years old.", name, age); In this example, we pass the values of the ‘name’ and ‘age’ variables into the string using placeholders, indicated by the ‘{0}’ and ‘{1}’ symbols. The values are then “formatted” into the string when it is printed out. We are also able to use , which are generally significantly more performant. They’re arguably much more readable as well. interpolated strings to do formatting Here’s the same example but done with string interpolation: string name = "John";
int age = 30;
string message = $"My name is {name} and I am {age} years old."; As we discuss string formatting options, what is applicable for will be applicable for interpolated strings as well. So the format string that gets provided for each argument can be used in either scenario! string.Format() Example String Format Options String format provides a variety of options for formatting the values you pass into a string. For example, you can specify the number of decimal places, include leading zeros, or even format numbers as currency or percentage values. The most common formatting options include: , , …, – placeholders for arguments. {0} {1} {N} and – escape curly brackets. {{ }} – optional format specifier. {:formatString} When using the format specifier, you can modify the appearance of the final string. For instance, you might want to format a decimal value with a dollar sign and two decimal places like so: decimal price = 12.3m;
string message = string.Format("The price is {0:C2}.", price); In this example, the format specifier applies a currency format and rounds the value to two decimal places. The result would be “The price is $12.30.” :C2 By leveraging string format, you can create dynamic and highly readable strings in your C# code. With placeholders and format strings, you’ll be equipped to handle virtually any string output scenario. Overview of Base64 Encoding . This allows arbitrary data to be transmitted in channels that might otherwise not accept non-printable characters, such as email or HTML documents. The encoding process involves dividing the input into blocks of three bytes, then converting each 24-bit block into four 6-bit values. These values are then represented as characters from the Base64 alphabet. Base64 encoding is a method of converting binary data into a printable ASCII string format The benefits of using Base64 encoding are numerous. For one, Base64 allows binary data to be transferred as an ASCII string, making it easier to transfer data over text-based channels. Additionally, Base64 is a fairly efficient encoding method, using only 33% more space than the original binary format. Base64 encoding is also commonly used in encoding passwords, sending image data, and encrypting data. In these instances, the use of Base64 makes it possible to pass data in a way that is secure, readable, and easily processed by computers. Base64 encoding has become a standard, accepted method of representing binary data on the web. When you use Base64 encoding, the encoded output is a string. This string consists of ASCII characters that represent the binary data in a way that can be transmitted safely. While the encoded string can sometimes be difficult to read for humans, your computer can easily decode the string back into its original binary format when it receives it. Using C# String Format with Base64 Encoding Combining C# string format with Base64 encoding allows us to build strings with encoded data that can be decoded later when needed. Using this technique can be exceptionally useful when passing data between different parts of an application, or when exchanging data between different applications that need to share information. To encode a string with Base64, we use the Convert.ToBase64String method. This method accepts a byte array and returns a Base64 encoded string. When we want to combine Base64 encoding with C# string format, we substitute our original byte array with the encoded string. Base64 Encoding with String Format Example Here’s an example: string original = "This is my original string.";
byte[] bytes = Encoding.UTF8.GetBytes(original);
string encoded = Convert.ToBase64String(bytes);

string formatted = $"The original string was '{original}', and it's Base64 encoded representation is '{encoded}'.";

Console.WriteLine(formatted); In this example, we first create our original string. We then convert it to a byte array using the UTF8 Encoding class. With the byte array in hand, we encode it using the method to create our Base64 encoded string. Convert.ToBase64String Finally, we into a formatted string. When we call with this formatted string, we get the following output: use C# string interpolation to embed both the original and encoded strings Console.WriteLine The original string was 'This is my original string.', and it's Base64 encoded representation is 'VGhpcyBpcyBteSBvcmlnaW5hbCBzdHJpbmcu'. In this way, combining string interpolation or string format format with Base64 encoding can be a powerful tool for passing and sharing data in your application or across different applications. Decoding Base64 Encoded Strings Once you have a Base64 encoded string, you may need to decode it back into its original binary format. Fortunately, C# provides a variety of ways to perform this task. Using The Convert Static Class One way is to use the method, which accepts a Base64 encoded string as input and returns a byte array. Here’s an example: Convert.FromBase64String string encodedString = "VGhpcyBpcyBhIHRlc3Q=";
byte[] bytes = Convert.FromBase64String(encodedString);
string decodedString = Encoding.UTF8.GetString(bytes);
Console.WriteLine(decodedString); In this example, we start with our Base64 encoded string. We then use the Convert.FromBase64String method to convert the encoded string into a byte array. Finally, we use the Encoding.UTF8.GetString method to convert the byte array back into a string. Using CryptoStream To Decode Base64 Another way to decode Base64 encoded strings is to use the CryptoStream class in the System.Security.Cryptography namespace. This method provides additional security through encryption, and is useful in cases where secure transmission of data is important. Here’s an example: DESCryptoServiceProvider crypto = new DESCryptoServiceProvider();
string encodedString = "VGhpcyBpcyBhIHRlc3Q=";
byte[] bytes = Convert.FromBase64String(encodedString);

using MemoryStream memory = new MemoryStream(bytes);
using CryptoStream decryptor = new CryptoStream(
    memory,
    crypto.CreateDecryptor(),
    CryptoStreamMode.Read);
using StreamReader reader = new StreamReader(decryptor);

string decryptedString = reader.ReadToEnd();
Console.WriteLine(decryptedString); In this example, we first perform the same Base64 decoding with . Then we create a new MemoryStream, where the encrypted data will be written to. After that, we create a new instance of the DESCryptoServiceProvider class, which provides encryption and decryption services using a symmetric algorithm. Convert.FromBase64String With our CryptoStream set up, we decrypt our previously-Base64-encoded string and create a StreamReader object. We can now easily read and display the decrypted string using ! Console.WriteLine Benefits of C# String Format with Base64 Using C# string format with Base64 encoding provides a flexible solution for working with data in your application. One of the key benefits of this approach is that you can embed encoded data in your strings while maintaining the readability of those strings. This can be especially useful when building formatted log messages or generating emails with encoded attachments. Another advantage to using C# string format with Base64 encoding is that it allows you to embed binary data directly into your formatted strings. This can be a major improvement over alternate approaches which require breaking the binary data into smaller pieces or compressing the data before including it in the string. Drawbacks of C# String Format with Base64 As with all things, this approach does also carry some drawbacks. One of the main disadvantages is that it can be more time-consuming to encode and decode strings compared to simpler concatenation approaches. This can lead to longer execution times and may impact the performance of your application if used excessively or in the wrong hardware environment. Additionally, it’s important to keep in mind that the data can increase in size when we base64 encode. Because a byte can be 256 values, Base64 only accounts for 64 values. That means we need additional space so that we don’t lose resolution. Because this will be variable depending on the string, you can assume on average it will be roughly 33% more size used. . Check out this StackOverflow answer for more Now You Know How to Format a String as Base64 in CSharp! Understanding C# string formatting with Base64 can bring many benefits to your . By learning how to use these concepts together, you can create more efficient and effective solutions to many common programming challenges. programming and software development skills Throughout this article, we explored the basics of C# string format and Base64 encoding, how to use C# string format & string interpolation with Base64, and the benefits and drawbacks of this approach. Combining C# string format with Base64 encoding can allow you to more easily encode and decode strings, resulting in more efficient and secure software solutions. I hope that you found something useful in this article! If you’re interested in more learning opportunities, and ! subscribe to my free weekly newsletter check out my YouTube channel Also appears . here

A Beginner's Guide Formatting Strings as Base64 in CSharp

What is the problem? Async await syntax gives us the most straightforward way to write asynchronous code. However, such construction cannot be used in certain scenarios. For instance, using async await keywords is prohibited in constructors. There are many reasons why it is not possible in the existing language version. Probably, one of the obvious ones is that async methods would return or to be able to properly handle exceptions without crashing the process, and trivially to be able to wait until the async operation is completed. Task Task<T> And if we simply take the following class, build the library containing it, and then look at disassembled code. public class MyObject
{
    public MyObject()
    {
        //diligently initializing
    }
} We will see the following line. instance void [System.Runtime]System.Object::.ctor() where the keyword indicates that the method is an instance method, meaning it is called on an instance of the class, not on the class itself (which would be indicated by the static keyword). instance So, technically, it calls some method (constructor) on the already existing object to initialize it. Could it be made asynchronous hypothetically? public class MyObject
{
    public async MyObject()
    {
        await InitializeAsync();
    }
} Of course, yes, everything is possible in our world; it only depends on the price and whether it will solve any real problem without introducing new complications and tradeoffs. I believe that developers have had discussions about it many times, and they clearly understand the feasibility and meaningfulness of incorporating this change. C# language However, even without language in-built such a feature, we can solve this problem. Ways to overcome There are plenty of approaches that could be used to achieve the asynchronous initialization. Async to sync If we are only allowed to have sync operation in constructors, we can intuitively apply the approach. async to sync public class MyBestObject
{
    public MyBestObject()
    {
        InitializeAsync().GetAwaiter().GetResult();
    }

    private async Task InitializeAsync()
    {
        //diligently initializing

        await Task.Delay(100);
    }
} This approach is not so bad if there is no synchronization context in place and the async operation is relatively fast, but in general, it is not a recommended practice since it is based on a lot of assumptions about the executing environment and details of the async operation. It leads to inefficient resource consumption, sudden deadlocks in UI applications, and a common violation of the async programming idea of “async all the way.“ Async factory A quite standard way for solving this problem is to use factories. public class MyBestService
{
    public async Task InitializeAsync() 
    {
        //diligently initializing

        await Task.Delay(100);
    }
}

public interface IMyBestServiceFactory
{
    Task<MyBestService> CreateAsync(CancellationToken cancellationToken);
}

public sealed class MyBestServiceFactory : IMyBestServiceFactory
{
    public MyBestServiceFactory()
    {
    }

    public async Task<MyBestService> CreateAsync(CancellationToken cancellationToken)
    {
        var service = new MyBestService();
        await service.InitializeAsync(cancellationToken);
        return service;
    }
} We could either use the static method in class or even specify a dedicated factory for that purpose. The second option is a little bit more compatible with the pattern since you can request in any class and then just call the method. MyBestService Dependency Injection IMyBestServiceFactory CreateAsync The main drawback of this approach is additional coupling since you (any class uses ) need to control the lifetime of a newly created object. IMyBestServiceFactory Additionally, it requires adapting solutions that use reflection ( ) or expressions ( ) to create and initialize instances. Activate.CreateInstace Expression.New Async initialization We could do the following trick to avoid problems with the pattern. Async Factory public class MyBestService
{
    private readonly Task _initializeTask;

    public MyBestService()
    {
        _initializeTask = InitializeAsync();
    }

    public async Task DoSomethingAsync() 
    {
        await _initializeTask;

        // Do something async
    }

    private async Task InitializeAsync()
    {
        //diligently initializing

        await Task.Delay(100);
    }
} As you can see, we are beginning asynchronous initialization in the constructor and saving the reference to the started task. Then, before doing any meaningful operation, we check that is completed by simply awaiting it. _initializeTask The using of this approach is very usual both in cases with a self-instantiating way of objects and IoC container. var myBestService = new MyBestService();
await myBestService.DoSomethingAsync(); However, this approach has several drawbacks: Lack of control over the initialization process. For instance, we cannot simply pass . CancellationToken It gives the caller an incompletely instantiated object for a while. Although it does not seem a problem in general, it could lead to more complicated debugging if the initialization operation takes too long. Also, calling operation timeout might be less than the required amount of time for this particular object which apparently will lead to issues during the execution. Real example For applications to access Azure services like storage, key vault, or cognitive services, authenticating with Azure is mandatory, regardless of their deployment environment - on Azure, on-premises, or locally during development. There are several ways it could be achieved, and one of the most secure ones is utilising the approach with the class in .NET DefaultAzureCredential That’s what Microsoft says about it The class provided by the Azure SDK allows apps to use different authentication methods depending on the environment they're run in. This allows apps to be promoted from local development to test environments to production without code changes. You configure the appropriate authentication method for each environment and will automatically detect and use that authentication method. The use of should be preferred over manually coding conditional logic or feature flags to use different authentication methods in different environments. DefaultAzureCredential DefaultAzureCredential DefaultAzureCredential So, in order to securely store PII (Personally Identifiable Information) which can be either Full name, Phone Number, Credit card number, or Social security number, we would like to encrypt it before storing it somewhere and provide viewing original value with only limited access. One of the possible ways is to use AzureKeyVault API for encryption and decryption. Although there is no huge impact of creating every time, to reduce latency and improve efficiency, it is preferable to have a single instance of it for the whole application. DefaultAzureCredential So, to prepare for encryption and decryption API, we need to have the following lines CryptographyClient var tokenCredential = new DefaultAzureCredential();
var keyClient = new KeyClient(new Uri(_configuration.KeyVaultUri), tokenCredential);
KeyVaultKey key = await keyClient.GetKeyAsync(_configuration.KeyName);
_cryptographyClient = new CryptographyClient(key.Id, tokenCredential); To avoid running them on every request, we can utilise described above Async initialisation internal sealed class DefaultAzureVaultAdapter : IAzureVaultAdapter
{
    private readonly AzureVaultConfiguration _configuration;
    private EncryptionAlgorithm _encryptionAlgorithm = EncryptionAlgorithm.RsaOaep256;
    private CryptographyClient _cryptographyClient = null!;
    private Task _initializationTask = null!;

    public DefaultAzureVaultAdapter(AzureVaultConfiguration configuration)
    {
        _configuration = configuration;
        _initializationTask = InitializeAsync();
    }

    public async Task<string> EncryptAsync(string value, CancellationToken cancellationToken)
    {
        await _initializationTask;

        byte[] inputAsByteArray = Encoding.UTF8.GetBytes(value);

        EncryptResult encryptResult = await _cryptographyClient.EncryptAsync(_encryptionAlgorithm, inputAsByteArray, cancellationToken);

        return Convert.ToBase64String(encryptResult.Ciphertext);
    }

    public async Task<string> DecryptAsync(string value, CancellationToken cancellationToken)
    {
        await _initializationTask;

        byte[] inputAsByteArray = Convert.FromBase64String(value);

        DecryptResult decryptResult = await _cryptographyClient.DecryptAsync(_encryptionAlgorithm, inputAsByteArray, cancellationToken);

        return Encoding.Default.GetString(decryptResult.Plaintext);
    }

    private async Task InitializeAsync()
    { 
        if (_cryptographyClient is not null)
            return;

        var tokenCredential = new DefaultAzureCredential();
        var keyClient = new KeyClient(new Uri(_configuration.KeyVaultUri), tokenCredential);
        KeyVaultKey key = await keyClient.GetKeyAsync(_configuration.KeyName);
        _cryptographyClient = new CryptographyClient(key.Id, tokenCredential);
    }
} And finally, register our with a singleton in IoC container. AzureVaultAdapter public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddAzureVaultAdapter(this IServiceCollection services)
    {
        services.TryAddSingleton<IAzureVaultAdapter, DefaultAzureVaultAdapter>();
        return services;
    }
} Conclusion In this article, I covered several approaches for asynchronous object initialisation. Understanding requirements is the key to choosing the right approach. You can find the source code used to describe these patterns in this repository. https://github.com/alex-popov-stenn/CSharpAsyncInitPatterns/tree/main Thank you for reading! See you next time!

Asynchronous Initialization in C#: Overcoming Constructor Limitations

A couple of years ago, was all the rage, but it had one big issue: reactive stopped as soon as you accessed a SQL database. You had a nice reactive chain up to the database, defeating the whole purpose. Given the prevalence of SQL databases in existing and new apps, one couldn't enjoy the full benefits of Reactive Programming but still pay the full price of complexity. Reactive Programming Since then, the landscape has changed tremendously. Most importantly, it offers many reactive drivers over popular databases: PostgreSQL, MariaDB and MySQL, Microsoft SQL Server, Oracle, you name it! Even better, some frameworks provide a reactive API over them. Even though I'm not providing consulting services regularly, I wanted to keep up-to-date on accessing data reactively. In this post, I'll describe Hibernate Reactive, Spring Data R2DBC, and jOOQ in no particular order. The base application uses Project Reactor and its types - and . For an added twist, I use Kotlin (without coroutines). Most code snippets have unnecessary type hints for better understanding. Flux Mono The demo model I don't want a complicated demo model, but I don't want it to be too simple. I'll use a single many-to-many relationship and a field with : LocalDate Spring Data R2DBC As far as I remember, the Spring ecosystem was the first to offer a reactive database access API. At first, it was limited to H2 - not very useful in production. However, new reactive drivers were easy to integrate. Spring Data RDBC builds upon the widespread Spring Data JPA. The biggest difference is that there's a single required annotation for entities, . @Id Here's the code for the table: person data class Person(
    @Id val id: Long,
    val firstName: String,
    val lastName: String,
    val birthdate: LocalDate?,
    @Transient
    val addresses: MutableSet<Address> = mutableSetOf()
)

interface PersonRepository : ReactiveCrudRepository<Person, Long> R2DBC repositories look similar to regular Spring Data repositories with one big difference. They integrate Project Reactor's reactive types, and . Note that it's easy to use Kotlin's coroutines with an additional bridge dependency. Mono Flux Now comes the hard problem: mapping the many-to-many relationship with the . Address First, we must tell Spring Data R2DBC to use a specific constructor with an empty set of addresses. data class Person(
    @Id val id: Long,
    val firstName: String,
    val lastName: String,
    val birthdate: LocalDate?,
    @Transient
    val addresses: MutableSet<Address> = mutableSetOf()
) {
    @PersistenceCreator
    constructor(
        id: Long,
        firstName: String,
        lastName: String,
        birthdate: LocalDate? = null
    ) : this(id, firstName, lastName, birthdate, mutableSetOf())
} We also need to define the repository, as well as a query to list all addresses of a person: Address interface AddressRepository : ReactiveCrudRepository<Address, Long> {

    @Query("SELECT * FROM ADDRESS WHERE ID IN (SELECT ADDRESS_ID FROM PERSON_ADDRESS WHERE PERSON_ID = :id)")
    fun findAddressForPersonById(id: Long): Flux<Address>
} Now comes the least tasteful part: Spring Data R2DBC doesn't support many-to-many relationships at the moment. We need a hook that queries the addresses after loading a person. class PersonLoadOfficeListener(@Lazy private val repo: AddressRepository)   //1
  : AfterConvertCallback<Person> {

  override fun onAfterConvert(person: Person, table: SqlIdentifier) =
    repo.findAddressForPersonById(person.id)                                //2
      .mapNotNull {
          person.addresses.add(it)                                          //3
          person
      }.takeLast(1)                                                         //4
      .single(person)                                                       //5
} Annotate with to avoid running into circular dependencies exception during injection @Lazy Use the above query Add each address Reactive trick to wait for the last bit of data Turn into a single Person As far as I can understand, Spring Data R2DBC still needs to execute additional queries, thus leading to the (in)famous . N+1 query problem One configures database access via all available Spring alternatives: properties, YAML, Spring profiles, environment variables, etc. Here's a YAML example: spring.r2dbc:
  url: r2dbc:postgresql://localhost:5432/postgres?currentSchema=people
  username: postgres
  password: root Hibernate Reactive If you're familiar with regular Hibernate, you'll feel right at home with Hibernate Reactive. The mapping is the same in both cases: @Entity
@Table(name = "person", schema = "people")                           //1
class Person(
    @Id var id: Long?,
    @Column(name = "first_name")                                     //2
    var firstName: String?,
    @Column(name = "last_name")                                      //2
    var lastName: String?,
    var birthdate: LocalDate?,
    @ManyToMany
    @JoinTable(                                                      //3
        name = "person_address",
        schema = "people",
        joinColumns = [ JoinColumn(name = "person_id") ],
        inverseJoinColumns = [ JoinColumn(name = "address_id") ]
    )
    val addresses: MutableSet<Address> = mutableSetOf()
) {
    internal constructor() : this(null, null, null, null)            //4
} Define the table and the schema if necessary Define column names, if necessary Define the join column JPA requires a no-argument constructor We also need to configure the database. Hibernate Reactive uses the traditional XML-based JPA approach: <persistence xmlns="https://jakarta.ee/xml/ns/persistence"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="https://jakarta.ee/xml/ns/persistence https://jakarta.ee/xml/ns/persistence/persistence_3_0.xsd"
           version="3.0">
  <persistence-unit name="postgresql">
    <provider>org.hibernate.reactive.provider.ReactivePersistenceProvider</provider>   
    <properties>
      <property name="jakarta.persistence.jdbc.url" value="jdbc:postgresql://localhost:5432/postgres?currentSchema=people" />
      <property name="jakarta.persistence.jdbc.user" value="postgres" />
      <property name="jakarta.persistence.jdbc.password" value="root" />
      <property name="jakarta.persistence.schema-generation.database.action" value="validate" />
    </properties>
  </persistence-unit>
</persistence> The only difference so far from the regular Hibernate configuration Here's the source for the query itself: val emf = Persistence.createEntityManagerFactory("postgresql")                            //1
val sessionFactory: Mutiny.SessionFactory = emf.unwrap(Mutiny.SessionFactory::class.java) //2
val people: Mono<MutableList<Person>> = sessionFactory
        .withSession {
            it.createQuery<Person>("SELECT p FROM Person p LEFT JOIN FETCH p.addresses a").resultList
        }.convert().with(UniReactorConverters.toMono())                                   //3 Regular EntityManagerFactory Unwrap the underlying session factory implementation. Because we configured a in the , it's a ReactivePersistenceProvider persistence.xml Mutiny.SessionFactory Hibernate Reactive integrates with , but an extension allows to bridge to Project Reactor if wanted Vert.x Note that Hibernate Reactive is the only library among the three to return a instead of a . In layman's terms, it means you get the whole list at once instead of getting the elements one by one and being able to do something on each one individually. Mono<List<T>> Flux<T> jOOQ Reactive As for the two above frameworks, jOOQ Reactive is similar to its non-reactive version. You first generate the code from the database schema, then use it. <plugin>
    <groupId>org.jooq</groupId>
    <artifactId>jooq-codegen-maven</artifactId>                               
    <executions>
        <execution>
            <id>jooq-codegen</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>generate</goal>
            </goals>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupId>org.postgresql</groupId>                                 
            <artifactId>postgresql</artifactId>
            <version>42.6.0</version>
        </dependency>
    </dependencies>
    <configuration>
        <generator>
            <name>org.jooq.codegen.KotlinGenerator</name>                     
            <database>
                <inputSchema>people</inputSchema>                             
            </database>
            <target>
                <packageName>ch.frankel.blog.reactivedata.jooq</packageName>
            </target>
        </generator>
        <jdbc>                                                                
            <driver>org.postgresql.Driver</driver>
            <url>jdbc:postgresql://localhost:5432/postgres</url>
            <user>postgres</user>
            <password>root</password>
        </jdbc>
    </configuration>
</plugin> The version is defined in the parent Spring Boot Starter parent POM Set the necessary database driver(s). Note that one should use the driver non-reactive There's a Kotlin generator! Configure database configuration parameters Once you've generated the code, you can create your data class and design the query. jOOQ class hierarchy integrates with Java's collections, Java's Reactive Streams, and Project Reactor. The code may look complex if you're neither a SQL nor a jOOQ expert. Remember that variable types are unnecessary, but added for documentation purposes: fun findAll(): Flux<PersonWithAddresses> {                       //1
  val people: SelectJoinStep<Record5<Long?, String?, String?, LocalDate?, MutableList<Address>>> = //2
    ctx.select(
      PERSON.ID,
      PERSON.FIRST_NAME,
      PERSON.LAST_NAME,
      PERSON.BIRTHDATE,
      DSL.multiset(                                              //2
        DSL.select(
          PERSON_ADDRESS.ADDRESS_ID,
          PERSON_ADDRESS.address.FIRST_LINE,
          PERSON_ADDRESS.address.SECOND_LINE,
          PERSON_ADDRESS.address.ZIP,
          PERSON_ADDRESS.address.CITY,
          PERSON_ADDRESS.address.STATE,
          PERSON_ADDRESS.address.COUNTRY,
        ).from(PERSON_ADDRESS)
           .where(PERSON_ADDRESS.PERSON_ID.eq(PERSON.ID))
      ).convertFrom { it.map(addressMapper) }                   //3
  ).from(PERSON)
  return Flux.from(people)                                      //4
             .map(personWithAddressesMapper)                    //3
} Return a regular Project Reactor's Flux Use , see below. multiset Convert the row to an ordinary Java object via a function The magic happens here: wrap the regular query in a for is a Project Reactor Flux people Publisher Let's dive a bit into from the point of view of a non-jOOQ expert - me. Initially, I tried to execute a regular SQL query with results I tried to flatten with Project Reactor's API. I failed miserably because of my lack of knowledge of the API, but even if I had succeeded, it would have been the wrong approach. multiset After hours of research, I found via : multiset a post from Vlad Mihalcea The value constructor is one of jOOQ's and standard SQL's most powerful features. It allows for collecting the results of a into a single nested collection value with semantics. MULTISET non scalar subquery MULTISET -- MULTISET value constructor In the above query, we first select all addresses of a person, map each row to an object, and flatten them in a list on the same result row as the person. The second mapper maps the row, including the address list, to a dedicated person with an addresses list. I'm not a SQL master, so is hard at first glance. However, I confirm that it's a powerful feature indeed. multiset Note that . nested collections are fetched eagerly on a per-record basis, whereas top-level records are streamed reactively Conclusion We have browsed the surface of the main three reactive database access: Spring Data R2DBC, Hibernate, and jOOQ. So, which one should one choose? The main deciding factor is whether you already use one of their non-reactive flavors. Use the framework you're familiar with since both reactive and non-reactive usages are similar. I think that jOOQ is extremely powerful but requires a familiarity I currently lack. If you have complex queries that don't map easily to other approaches, it's the way to go. Besides that, I've no strong opinion, though I find Hibernate Reactive's configuration too limited by JPA and its return type puzzling. Mono<List<T>> Thanks for and for their reviews on their respective sections of expertise. Lukas Eder Mark Paluch The complete source code for this post can be found on . GitHub To go further: Hibernate Reactive Reference Documentation Integrating Hibernate Reactive with Spring jOOQ Reactive Fetching How to fetch multiple to-many relationships with jOOQ MULTISET How to Turn a List of Flat Elements into a Hierarchy in Java, SQL, or jOOQ Spring Data R2DBC Comment bien s'entendre avec avec Spring Data R2DBC... ou pas Originally at on July 9th, 2023 published A Java Geek

Reactive Programming - The New Way to Write Apps?

In this article, we’re going to discuss an alternative approach to handling the data access layer in TypeScript with and without ORM. MongoDB Traditionally, developers leverage Object-Relational Mapping (ORM) tools to map between data types in databases and object-oriented programming languages. However, ORM may sometimes lead to performance issues, complex configurations, or inflexibility to read/write data. To circumvent these challenges, we will utilize a method that employs for data validation and type inference, pattern for converting Entities to DTO (Data Transfer Objects), and encapsulating data logic within a Service class. zod companion object Why Schema Validation? Schema validation is a programming process that ensures data conforms to a predefined schema or structure. The schema serves as a blueprint or model that defines permitted data and its organization, often specified in terms of data types, constraints, and relationships. By enforcing schema validation, programs can catch data errors early, enhance data consistency, and prevent potential issues related to incorrect or malformed data. There are two common scenarios in which we need to apply schema validation: When handling input data, such as with API request payloads. In this context, schema validation can help ensure that the submitted data conforms to the expected format and structure. This is especially important for complex data sets that may contain a variety of different fields and data types. When performing read and write operations with a NoSQL database, such as MongoDB. In this context, schema validation can help ensure that the written data conforms to the expected schema, and can also validate retrieved data. This is especially important when data consistency is a concern, or when there are strict requirements for data quality and accuracy. This article is focusing primarily on the second scenario. Why Zod? is a TypeScript-first library designed for schema declaration and validation. It enables developers to construct schemas that validate runtime data and generate TypeScript types, ensuring type safety without additional manual type annotations. By integrating with TypeScript, enhances the language's static typing capabilities with runtime validation, offering features like custom error handling, nested schemas, and transformations. Zod zod Please note that although MongoDB has a “JSON Schema Validation” feature, some MongoDB API-compatible services like AWS DocumentDB do not support it. Furthermore, I believe that the application code should be conscious of and enforce all the data constraints it is operating with. Therefore, I would still prefer to have schema validation as part of the application code, and this is where can be helpful. zod Defining Entity with Zod Let’s consider an example where we’re working with users in a MongoDB database. We’ll start by defining an entity representing how a user is stored in the database. import { z } from "zod";
import { ObjectId } from "mongodb";

export const userEntitySchema = z.object({
  _id: z.instanceof(ObjectId),
  name: z.string(),
  email: z.string().email(),
});

export type UserEntity = z.infer<typeof userEntitySchema>; In the above code, we’re defining a and a type. The schema describes that a user entity has three fields: , , and . The type is inferred directly from the schema. userEntitySchema UserEntity _id name email UserEntity Defining Data Transfer Object (DTO) with Zod Next, we’ll define a DTO that represents how we’re going to use the user data in our application. export const userDTOSchema = z.object({
  id: z.string(),
  name: userEntitySchema.shape.name,
  email: userEntitySchema.shape.email,
});

export type UserDTO = z.infer<typeof userDTOSchema>; Here, we’ve defined a similar schema and type for , but notice that the field has been replaced with , while and are both using the shape from . UserDTO _id id name email userEntitySchema Applying the Companion Object Pattern To convert an entity to a DTO, we’ll apply the companion object pattern. This pattern involves creating a companion object to hold static functions related to the class or type. export const UserDTO = {
  convertFromEntity(entity: UserEntity): UserDTO {
    const candidate: UserDTO = {
      id: entity._id.toHexString(),
      name: entity.name,
      email: entity.email,
    };
    return userDTOSchema.parse(candidate);
  },
}; In the above code, we’ve defined an object called with one function: — this function can convert a object to a object. UserDTO convertFromEntity UserEntity UserDTO To explain it a bit more, TypeScript manages “types” and “concrete object definitions” in different namespaces, that’s why we could define a type as well as a concrete object called the same name. An object defined in this way usually has some util or helper functions that apply to the same-name type, thus it is so-called a “companion object”. UserDTO Please also note that we do not only guarantee the type safety on compile time but also ensure the runtime type safety via — which will make sure that the object meets the schema validation requirement. schema.parse() Encapsulating Data Logic in a Service Class Finally, we will encapsulate our data logic within a service class, which will handle all CRUD operations for the users. This service class will interact directly with MongoDB using the MongoDB Node.js driver. import { MongoClient, Db } from "mongodb";

export class UserService {
  private readonly db: Db;

  constructor(mongoClient: MongoClient) {
    this.db = mongoClient.db();
  }

  private getUsersCollection() {
    return this.db.collection<UserEntity>("users");
  }

  async findUser(id: string): Promise<UserDTO | null> {
    const entity = await this.getUsersCollection().findOne({ _id: new ObjectId(id) });
    return entity ? UserDTO.convertFromEntity(entity) : null;
  }

  async createUser(dto: Omit<UserDTO, "id">): Promise<UserDTO> {
    const candidate = userEntitySchema.parse({
      ...dto,
      _id: new ObjectId(),
    });
    const { insertedId } = await this.getUsersCollection().insertOne(candidate);
    return UserDTO.convertFromEntity({ ...dto, _id: insertedId });
  }

  async updateUser(id: string, dto: Omit<Partial<UserDTO>, "id">): Promise<UserDTO | null> {
    const candidate = userEntitySchema.partial().parse(dto);

    const { value } = await this.getUsersCollection().findOneAndUpdate(
      { _id: new ObjectId(id) },
      { $set: candidate },
      { returnDocument: "after" }
    );
    return value ? UserDTO.convertFromEntity(value) : null;
  }

  async deleteUser(id: string): Promise<void> {
    await this.getUsersCollection().deleteOne({ _id: new ObjectId(id) });
  }
} In this class, we implemented all CRUD operations for users: , , , and . All these methods interact with MongoDB directly through the MongoDB Node.js driver and handle conversions between and . UserService findUser createUser updateUser deleteUser UserEntity UserDTO Please note that this example may not represent real-world business data constraints, but rather an idea to encapsulate the data logic around . User Some people may prefer to enforce the Repository pattern and call it , making the class solely focused on CRUD operations without any business logic, then use inside the . For small to medium-sized projects, I would recommend to start with which should already efficiently fulfill the needs without too much overhead. UserRepository UserRepository UserService Service Bonus: out-of-box IDE support with IntelliSense By define types in the way we described in this blog, IDE will get us covered without need for extra library or plugin. For example, when we try to build query using , IDE or code editor knows what are the fields we can filer on: .findOne() Code snippet for idea verification Here we could have a simple code snippet to verify our idea: import _ from "lodash";

const main = async () => {
  const mongoClient = new MongoClient("mongodb://localhost:27017/data-access-example");

  try {
    await mongoClient.db().dropCollection("users");
  } catch (e) {
    if (_.get(e, "codeName") !== "NamespaceNotFound") {
      throw e;
    }
    console.log(`Collection "users" does not exist, no need to drop`);
  }

  await mongoClient.db().createCollection("users");

  const userService = new UserService(mongoClient);

  const createdUser = await userService.createUser({ name: "example", email: "example@example.com" });
  console.log({ createdUser });

  const updatedUser = await userService.updateUser(createdUser.id, { name: "exampleX" });
  console.log({ updatedUser });

  const foundUser = await userService.findUser(createdUser.id);
  console.log({ foundUser });

  await userService.deleteUser(createdUser.id);
};

main()
  .then(() => {
    process.exit();
  })
  .catch((e) => {
    console.error(e);
    process.exit(1);
  }); Conclusion In this blog post, we’ve discussed an alternative approach to handle the data access layer without ORM when using TypeScript and MongoDB. This method offers flexibility and eliminates potential bottlenecks associated with ORMs, while maintaining the consistency and clarity of our data layer. With the library for data validation and type inference, companion object pattern for conversion operations, and encapsulation within service classes, we're able to keep our code clear, concise, and highly functional. zod Also published here.

Handling ORM-Free Data Access Layer in TypeScript With MongoDB

While full homomorphic encryption (FHE) was originally achieved in 2009, it's starting to receive more attention now that computing power has caught up to its computational demands; however, the goal of homomorphic encryption goes back a decade. A cursory search will find multiple definitions for it. This is the most direct and succinct: A form of encryption that allows computation on ciphertexts generating an encrypted result that, when decrypted, matches the result of the operations as if they had been performed on plaintext. — Massimo Bertaccini ( ) Cryptography Algorithms Consider that statement for a moment. Homomorphic encryption enables operations to be performed on encrypted values such that the decrypted results produce the result of the operations as though they were performed on the original unencrypted values. The earliest mention I found is (1978) in which the authors are attempting to prevent sensitive financial data stored in an on-premise system from being accessed by system programmers using a remote time-sharing service. On Data Banks and Privacy Homomorphisms Sensitive financial data is stored in the data bank. Time-sharing services that compute the data do not implement adequate security. System programmers have access to financial data. The paper states: Use a special privacy homomorphism to encrypt its data so that the time-share computer can operate on the data without the necessity of decrypting it first. To put it another way, I want to delete the of my data without giving away to it. processing access To put it another yet another way, function(A, B) = function (A ) where A are encrypted values of A and B, respectively. This enables third parties to compute encrypted data and provide the results without having any knowledge of the original value. With RSA, two values encrypted with the same key can be multiplied, and the product can be decrypted yielding the product of the encrypted values. , B &nbsp;and B Obtain 5_6 using RSA(5)_ RSA(6): RSA(5) = (5^17) mod 3233 = 3096 RSA(6) = (6^17) mod 3233 = 824 3096*824 = 254864 RSA^-1(254864) = 254864^2753 (mod 3233) = 30 5*6 = 30 However, this does not work for addition. RSA is a Partially Homomorphic Encryption (PHE) scheme in that it supports only multiplication. The timeline below shows the progression from (PHE) schemes through Somewhat Homomorphic Encryption (SHWE) which permits limited combinations of addition and multiplication. These operations could be repeated up to five times before reaching an invalid result. Then along came the first FHE scheme from which enabled repeated addition and multiplication without fidelity loss. Craig Gentry Homomorphic Encryption Libraries Since 2009, FHE schemes and libraries have proliferated. Library Schemes Language License Microsoft SEAL BFV, BGV, CKKS C++/C# Wrapper MIT Open FHE BFV, BGV, CKKS, DM, CGGI C++ BSD 2-Clause License HEAAN CKKS C++ Creative Commons 3.0 TFHE TFHE (Torus) Rust BSD-3-Clause-Clear Concrete TFHE Rust BSD-3-Clause-Clear Lattigo BFV, BGV, CKKS Go Apache 2.0 Intel Homomorphic Encryption Toolkit N/A – Uses SEAL, PALISADE, HLib Python,Docker Apache 2.0 Microsoft SEAL (Simple Encrypted Arithmetic Library) is developed in C++ 17 along with a C# wrapper. Originally released in 2015, it is still under active development. The includes that should be explored if you are interested in diving further. github repo in-depth C# examples These schemes are supported by SEAL and include homomorphic operations for addition, division, subtractions, and exponentiation, but not division. Brakerski-Gentry-Vaikuntanathan (BGV)/Fan-Vercauteren (BFV) Whole numbers (long) Use if precision is a requirement Computation is slower than CKKS Values encrypted with BGV can be used with values encrypted with BFV Values encrypted with BGV/BVF cannot be used with values encrypted with CKKS Revisiting Homomorphic Encryption Schemes for Finite Fields Cheon-Kim-Kim-Song (CKKS) Floating points (double) Use if precision is not required Values encrypted with CKKS cannot be used with values encrypted with BGV/BFV Computation is faster than BFV/BGF Encryption and Computation with Microsoft SEAL These examples use the Nuget package and the CKKS scheme. Health trackers, like FitBit and AppleHeath, require giving up some privacy. They collect personal information like the number of steps and geocoordinates of the wearer's movements, heartbeat frequency, and other vital statistics. With homomorphic encryption, these health metric aggregators can still operate on our data without having access to the raw data. Microsoft.Research.SEALNet In this simple example, the client sends run metrics including distance and time. The server aggregates the encrypted data and computes the average speed. To get started, a public key is created by the client and shared with the server. It is used to encrypt data and also to perform arithmetic operations on encrypted data. Both client and server use an The client generates and retains a private key used for decryption used by the aptly named . Encryptor. Decryptor A malicious party that intercepts the data could use the public key on the encrypted data and return it to the caller by spoofing the server. The server should sign the result and the client should verify the signature to ensure it's being returned from a trusted party. using Microsoft.Research.SEAL;
protected SEALContext _context;
private KeyGenerator _keyGenerator;
private Encryptor _encryptor;
private Decryptor _decryptor;
protected IFitnessTrackerApiClient _apiClient;
. . .
_context = SEALUtils.GetContext(_config.Value.PolyModulusDegree, this.SchemeType);

_keyGenerator = new KeyGenerator(_context);

_keyGenerator.CreatePublicKey(out PublicKey publicKey);
_keyGenerator.CreateRelinKeys(out RelinKeys relinKeys);

_encryptor = new Encryptor(_context, _publicKey);
_decryptor = new Decryptor(_context, _keyGenerator.SecretKey);
. . .
PublicKeyModelCKKS keyModelCKKS = new(
  SEALUtils.KeyToBase64String(_publicKey), 
  SEALUtils.KeyToBase64String(relinKeys));
    await _apiClient.SendPublicKeyCKKSAsync(keyModelCKKS); Relinearization keys are passed from client to server. Per Microsoft SEAL, they have no semantic meaning other than to reduce the size of following a multiplication operation. Ciphertext At the command line, the user enters time and distance. Each value is encoded to then encrypted to , and finally encoded to base64 prior to sending it to the server. PlainText, Ciphertext using Microsoft.Research.SEAL;
var plaintext = new Plaintext();
_encoder.Encode(value, _scale, plaintext);

var ciphertext = new Ciphertext();
_encryptor.Encrypt(value, ciphertext);

using (var ms = new MemoryStream())
{
  ciphertext.Save(ms);
  return Convert.ToBase64String(ms.ToArray());
} Since homomorphic encryption schemes don't support division (CKKS is no exception), the client needs to give the server an assist. The formula for speed is: distance/time Another way to write it is, distance * (1/time) So, the client sends: RunItemCKKS metricsRequest = new(
  EncryptBase64(runItem.Distance), 
  EncryptBase64(runItem.Time), 
  EncryptBase64(1 / runItem.Time));

await _apiClient.AddNewRunningDistanceCKKSAsync(metricsRequest); The server performs a similar bootstrapping function creating a using only the public key and re-linearization keys, then processes the request: SEALContext var distance = SEALUtils.BuildCiphertextFromBase64String(request.Distance, _sealContext);
var time = SEALUtils.BuildCiphertextFromBase64String(request.Time, _sealContext);
var timeReciprocal = SEALUtils.BuildCiphertextFromBase64String(request.TimeReciprocal, _sealContext);
Ciphertext speed = new();

_evaluator.Multiply(distance, timeReciprocal, speed);
_evaluator.RelinearizeInplace(speed, _relinKeys);

_runListCKKS.Add(new EncryptedRunInfoCKKS(distance, time, speed)); A reciprocal of the time is used to calculate the speed for the submitted run and the method reduces the size of the resulting Distance, time, and speed are saved to an in-memory RelinearizeInplace Ciphertext. List. The method aggregates the list and returns: GetMetrics total number of runs total distance total time average speed public SummaryItemCKKS GetMetrics()
{
int count = _runListCKKS.Count;

var totalDistanceCKKS = SumEncryptedValues(_runListCKKS.Select(m => m.Distance));
var totalTimeCKKS = SumEncryptedValues(_runListCKKS.Select(m => m.Time));
var totalSpeed = SumEncryptedValues(_runListCKKS.Select(m => m.Speed));
. . .
protected Ciphertext SumEncryptedValues(IEnumerable<Ciphertext> encryptedData)
{
. . .
Ciphertext encTotal = new();
_evaluator.AddMany(encryptedData, encTotal);
return encTotal;
. . .
} To get the average speed, the reciprocal of the count is used, so rather than using: (sum of speed)/number or runs we use: (sum of speed) * (1/number of runs) Since the server keeps track of the number of run submissions, the total number of runs is not an encrypted value. That unencrypted value can be converted to and used in an operation performed on While this is a simple example, the implications are worth noting. The server can provide additional values used on encrypted data enabling third-party providers to apply proprietary data to computations against customer-provided encrypted data. Plaintext Ciphertext. In the example below, the total count is converted to a reciprocal and added to a CKKS operations expect encrypted list values, even if the list only contains a single value. So, the is encoded to a . The multiplies totalSpeed by (1/number of runs) resulting in the average speed. To conserve space, the result is applied to the totalSpeed and the re-linearization keys reduce the size of the output. List<double>. List<double> PlainText MultiplyPlainInplace Ciphertext Plaintext encodedCountReciprocal = new();
List<double> averagePaceList = new();
double runCountReciprocal = 1 / (double)count;
averagePaceList.Add(runCountReciprocal);
_encoder.Encode(averagePaceList, _scale, encodedCountReciprocal);
        
_evaluator.MultiplyPlainInplace(totalSpeed, encodedCountReciprocal);
_evaluator.RelinearizeInplace(totalSpeed, _relinKeys); The values are base64 encoded, returned, and decrypted on the client. The client takes the additional step of converting decrypted to . Plaintext List<double> var payload = Convert.FromBase64String(encryptedDistanceText);

using var ms = new MemoryStream(payload);
var ciphertext = new Ciphertext();
ciphertext.Load(_context, ms);

var decryptedText = new Plaintext();
_decryptor.Decrypt(cypherText, decryptedText);

List<double> distanceList = new();
_encoder.Decode(decryptedText, distanceList); Sending the following values yields totals and the average pace calculated by a service that tabulated the total and average without decrypting data sent from the client. Miles Time 2.5 0:35:32.643 2.2 0:32:48.826 2.8 0:34:52.036 The server computes the data and returns the total runs, distance, time, and pace without decrypting any of the data sent. Summary Microsoft SEAL is a Microsoft Research project and is not ready for a production application. All classes implement and used unmanaged resources. Production code would need to protect against a memory leak. The class does not follow the single responsibility principle. It includes methods for both BGV/BVF and CKKS. Further, CKKS supports vector operations, but not vector addition. A single can be encrypted to a single but there is no means of summing the values in a single It can be done with , but that is a suboptimal solution. IDisposable SEALContext List<double> CipherText CipherText. vector rotation This Fitness Tracker code sample is a simple introduction to a complex topic.  There are a number of compelling use cases with more involved logic: Gene marker detection Biometric template protection Secure human monitoring for aging in place Credit card fraud detection Homomorphic encryption has largely been used in academia and in prototypes. Practical applications have been limited by the computing power required by homomorphic encryption schemes. GPU acceleration and ASIC processors will continue to erode this barrier. Intel is developing the , purpose-built for homomorphic encrypt schemes, in collaboration with Microsoft and DARPA. NASDAQ is also funding with machine learning for fraud detection. The FHE.org community is an excellent resource for the most current developments in this space. It has an active . FHE Hardware Accelerator an R&D project using homomorphic encryption Discord server All code samples in this article are available at . For more in-depth Microsoft SEAL example, see these . johniwasz/microsoft-seal-samples .NET examples Also published here.

Microsoft SEAL and the Dawn of Homomorphic Encryption 

Microsoft SEAL y el amanecer del cifrado homomórfico

Microsoft SEAL et l'aube du chiffrement homomorphe

Microsoft SEAL और होमोमोर्फिक एन्क्रिप्शन का डॉन

Microsoft SEAL と準同型暗号の夜明け

Microsoft SEAL e o surgimento da criptografia homomórfica

Microsoft SEAL và Bình minh của Mã hóa Đồng hình

Microsoft SEAL 和同态加密的黎明

At , I had a task to fetch some files from GitHub API. Initially, I thought it'd be a simple GET request but soon I realized the work is a headache. The file I needed was quite large and hence the response was telling me to use something different than I already knew. I ended up spending two days on this issue. Hackernoon Now it's time to write an article so that my future self and others don't have to spend so much time on a simple issue. In this article, I'll show you how to get files larger than 1MB from the GitHub API. You will learn about the following topics until the end: Fetching larger files from GitHub API Working with commit's SHA Manipulating blobs and base64 encoded data For the purpose of this article, I am using data from HackerNoon's contest. The data is publicly available on if you want to follow along. Startup of the Year GitHub Get Files Smaller than 1MB Fetching smaller files is simple and you just need to pass three parameters to the API. First is the owner's name, which could be your username. The second is repo, which is the name of the repository where your file is committed. And path is the path to the file from the root. I have created a simple codepen if you want to use working code snippets. For better understanding, I have created four variables to store data at each state. let fileSHA, fileBlob, fileContent, file stores the SHA of the file we want to fetch. fileSHA stores the blob fetched from the API. fileBlob is used to store the decoded string. fileContent has the data that we actually need. file You can find the reference code in the next section, just change the folder path with the actual file path (with file extension) and you're good to go! The Problem with Large Files However, if you fetch a file larger than 1MB, the API would throw an error. This is due to the fact that the above endpoint does not support large files. To retrieve the data, we must instead use the API. Git Database The Git Data API allows you to read and write raw Git objects to and from your Git repository and list, as well as update Git references. This API primarily works with blobs, which is the focus of this article. Git converts a large file into a base64 encoded blob rather than storing the entire file for better performance. As a result, when you request the same file, it expects you to use the Database endpoint, which returns the blob. To retrieve a blob, you need to pass file's SHA. You might ask, what is SHA and how to get it? Every time you commit a new file, git creates a unique ID called hash orSHA to keep a record of changes. This could cause an issue sometimes but we would discuss this at the end of this article. Get SHA of the file For now, we need to figure out a way to get `SHA` of the desired file. GitHub API has an endpoint to interact with the file content that we can use. GET /repos/{owner}/{repo}/contents/{path} In response, we get an array of objects having metadata about every file in the directory. The Metadata contains SHA that we can store and use later. ); }

  getFileBlob(fileSHA)
} const getFileSHA = async () => { try { const response = await fetch( "<https://api.github.com/repos/hackernoon/where-startups-trend/contents/2021/>" const data = await response.json(); console .log(data); fileSHA = data[ 3 ].sha console .log(fileSHA); } catch (error) { console .log(error); has the SHA of , so we store it in for future uses. The console output would look something like this: data[3].sha votes_by_region.json fileSHA Fetch Blob of the file We have the file SHA which is required to get the blob. We need to use a different endpoint to work with blobs. This endpoint is similar to the last one but we have to pass file SHA as the third parameter. GET /repos/{owner}/{repo}/git/blobs/{file_sha} The following function fetches the blob of our file when we provide its SHA as a parameter. ); fileBlob = data.content
  convertBlob(fileBlob) }
} const getFileBlob = async (fileSHA)=> { try { const response = await fetch( `https://api.github.com/repos/hackernoon/where-startups-trend/git/blobs/ ${fileSHA} ` const data = await response.json(); } catch (error) { console .log(error); This request returns a base64 encoded blob. Base64 is an encoding scheme that represents binary data in an ASCII string format. It is useful when we need to store or transfer data without any modification. The console output will look gibberish but it's just how base64 encoded data is. Convert Blob into Useful Data In the final step, we have to convert the base64 encoded blob into something that we can use in our program. This step is quite frustrating because the standard ways don't seem to work in some cases. After spending hours on this single problem, I have found three solutions. I don't guarantee that every solution will work out for you, but I am pretty sure one of them surely will. This is the standard way natively supported by JavaScript. is a Web API interface that decodes base64 encoded data into a plain string. Easiest Way: atob() atob is supposed to be read as ASCII to Binary since it converts ASCII encoded base64 data into binary form. The output will be a decoded string. To convert it into its original data type, we use JSON.parse(). fileContents = atob(blob) file = JSON .parse(fileContents) If the above method throws an error, Node.js provides a method that you can use. It takes the string to be decoded as the first parameter and the encoding technique as the second. Second Way: Buffer.from() } try { const fileContents = Buffer.from(fileBlob, "base64" ).toString() file = JSON .parse(blobToString) console .log(file) } catch (error) { console .log(error) There's a good chance the previous method won't work if you're working on the frontend. In such case, create a function that uses . Final Way: decodeURIComponent }); } // define the function const decodeBase64 = str => { utf8Bytes = decodeURIComponent (str).replace( /%([0-9A-F]{2})/g , function ( match, p1 ) { return String .fromCharCode( '0x' + p1); return atob(utf8Bytes); You can call this function by passing the as the parameter, and get your decoded string! fileBlob fileContents = decodeBase64(fileBlob) file = JSON .parse(fileContents) console .log(file) Nothing Worked? Well, there are two final options that you can try. Either you can use FileReader API or use decodeURIComponent in a different way. : You can find MDN docs FileReader API here : Refer to blog by Rajeev Singh that handles 16bit encoded strings. decodeURIComponent this Alternatively, you can try an NPM package called as well. I have not used it myself so don't have opinions regarding it. js-base64 Restrictions As mentioned earlier, SHA is the unique ID that changes every time you modify the file. Also, since we have a hardcoded object index in the function, API might respond with SHA of a different file if you add or delete files in the directory. getFileSHA() To tackle this, you can use the exact file name instead of the index so you can keep the function unmodified after committing new changes. Summary and TL;DR Git stores larger files in blob format and hence we use GitHub Database API to fetch files larger than 1MB. We need to provide a username, repo name and the file's SHA to get the blob. To get SHA, we provide a folder path to the endpoint and store it in a variable. Later we pass SHA as a parameter to another endpoint and get a base64 encoded blob. The blob must be decoded into a plain string and then converted into its original format using before you can use it. JSON.parse() More references GitHub repositories' contents Git Database Getting started with the Git Database API Do let me know which decoding method worked for you. I am most active on , and ! Twitter LinkedIn Showwcase Happy fetching!

How to Fetch Large Data Files Through GitHub API

Powershell continues to be an important part of the Microsoft ecosystem. Most of this learning is based on Microsoft documentation, a few blogs, and google and stack overflow searches. Please note that the focus of this article is on building complex packages and tools rather than single file scripts. Modules Modules are one of the fundamental blocks of packaging in PowerShell. Most of the teams would probably end up creating a module to collect a set of common utility functions or share the finished component with larger teams through . Alternatively, modules can also be imported locally (see bootstrap below) and used as part of scripts. PowerShell Gallery Powershell Modules typically should follow a standard structure and configuration to ensure consistent outcomes and reduce potential issues. The following directory structure ensures that you can have the right place for each component involved. The recommended module directory structure aligns with standard guidance for writing the module. The root directory should be named to uniquely identify module and match the module name. The file is pretty standard files generated and then updated to align with specific module. The “ ” and “ ” in PSD1 file points to “ ” file that contains basic module reference. available PSD1 RootModule ModuleList <module name>.psm1 The folder contains scripts files that contain functions, class that should be exported. The folder should be used to store helper functions. folder can be used to store dll or other components that is used by module. public private bin The file below contains some additions to reference available . psm1 here #Get public and private function definition files.
$Public  = @( Get-ChildItem -Path $PSScriptRoot\Public\*.ps1 -ErrorAction SilentlyContinue )
$Private = @( Get-ChildItem -Path $PSScriptRoot\Private\*.ps1 -ErrorAction SilentlyContinue )

#Dot source the files
Foreach($import in @($Public + $Private))
{
    Try
    {
        . $import.fullname
    }
    Catch
    {
        Write-Error -Message "Failed to import function $($import.fullname): $_"
    }
}

 Set-Variable -Name SCRIPT_PATH -Value (Split-Path (Resolve-Path $myInvocation.MyCommand.Path)) -Scope local
 Set-Variable -Name FULL_SCRIPT_PATH -Value (Resolve-Path $myInvocation.MyCommand.Path) -Scope local
 Set-Variable -Name CURRENT_PATH -Value ((Get-Location).Path) -Scope local
 Set-Variable -Name LOGGER -Value $null -Scope global
 
 # Export functions
 Export-ModuleMember -Function <Function name> Export function in the modules instead of class where possible to avoid reload issues associated with class. All the functions defined in scripts should contain basic documentation tags like , , (s), , . The following sample can be used as a guide for creating functions. .SYNOPSIS .DESCRIPTION .PARAMETER .OUTPUTS .EXAMPLE <#
  .SYNOPSIS
    This contain all the sample utility functions
  .DESCRIPTION
    Sample utility functions should be used across all sample applications
#>

<#
  .SYNOPSIS
    Quick function description
  .DESCRIPTION
    This is a very long description about the function below
  .PARAMETER  OneParameter
    Description of parameter including allowed values.
  .PARAMETER TwoParameter
    Another description
  .OUTPUTS
    Exit codes:
    0 : Success
    1 : Unkown error
    2 : More specific error code
  .Example
    Run-SampleFunction -OneParameter "some value"
 #>
Function Run-Samplefunction {
  [CmdletBinding()]
  [OutputType([int])]
  param (
    [Parameter(Mandatory = $True, Position = 0)]
    [PSCustomObject]$OneParameter,
    [Parameter(Mandatory = $True, Position = 1)]
    [PSCustomObject]$TwoParameter,
   )
  Process {
    $LASTEXITCODE = 1
    # Write-LogDebug "Entering function Run-Samplefunction"
    try {
       # ....do something.... 
    } catch [System.Exception] {
      $ErrorMessage = $_.Exception.Message
      # Write-LogError "Message: $ErrorMessage"
    }
    # Write-LogDebug "Leaving function Run-Samplefunction"
    return $LASTEXITCODE
  }
 } Note the use of that provides some advanced like binding predefined arguments to the function. In addition to that well-defined s and can reduce the boilerplate validation code. Using ( , , ) ensures that your functions are prepared for . Use of and return exit code is recommended approach but should be evaluated in line with internal coding guidelines. CmdletBinding() capabilities parameter output input processing methods Begin Process End pipelines try…catch Common Utility Most of the big developments should evaluate the use and need of the following components. Logging A simple level-based logging mechanism can be created using log4net and/or Write-Host function with function signatures like , to ensure that logging at different levels can be enabled and or disabled through a common mechanism (see bootstrap below). Write-LogDebug ($Text) Write-LogInfo ($Text) Configuration Develop a JSON or equivalent model of config that has the ability to load configuration from a file using equivalent to the following command $configData = New-Object -TypeName PSCustomObject -argumentlist (Get-Content -Raw -Path $ConfigFile | ConvertFrom-Json) It may be helpful to have a standard naming convention while designing the configuration file to ensure that you have a common configuration ( ) along with component-specific configuration ( ). In addition to that develop helper functions that can return config associated with the key name specified in dot-separated format can ensure that each component can have its own format for configuration and know how to handle it without bothering about the rest of the configuration structure. See the section on bootstrap below on how the configuration loading can be performed. config.common config.component Credential/Security It is important to develop a helper security function at the start of the project and use it consistently in the code to ensure that over time the code can use updates being made from a security perspective. Such helper functions should create for reading passwords using config (either by decrypting from config or retrieving from the vault with a simple configuration change) and return object to be used. Another important helper function can be a common authentication mechanism when multiple authentications can be performed (for example Azure can support multiple authentication mechanisms like id/password, service principal/password, certificate), it may be helpful to have a single authentication function that chooses actual authentication process based on configuration. PSCredential Scripts In a large development, multiple scripts across different use-cases would typically be developed. Such a development if it follows standard structure can significantly simplify the overall process. At the very basic it is recommended, that a standard directory structure is developed to package these development components. The root directory should contain with specific details about how the packaging has been done, various scripts available and specific examples to run the script. The directory contains the various scripts along with a common which provides a standard mechanism to handle configuration, load common modules, initialize logging, handle common parameters (see below for more details). The directory contains the configuration file with default values and the folder contains various templates and other files needed to run the script. Additional directory like , may be created as needed by script as part of the execution process. Readme.md bin bootStrap.ps1 config data logs tmp Bootstrap The can contain typical operations that can not be handled in a module due to dependency on specific parameters. It is recommended, that the bootstrap should have a function like the one below that can be called by any script. bootstrap.ps1 Function BootStrap { [CmdletBinding()] [OutputType([PSCustomObject])] Param([string[]] $ModulesToLoad, [string] $ConfigFile, [string] $ConfigToLoad, [string] $LogLevel, [string] $LogType) This function should perform the following functions Identify the root location of deployment (parent of the shown above) <Root Directory> Add the root location to (if not already set) to ensure that all the modules that are part of the package can be imported without dependency on external repository like . $env:PSModulePath PSGallery Import basic modules provided If (!(Get-Module Sample.Common-Utils)) {     Import-Module Sample.Common-Utils -Force -ErrorAction Stop } Initialize logger and any other common system needed. Load all the modules passed as after checking if the module is available. If not, then it tries to install using $ModulesToLoad Install-Module $ModuleName -AllowClobber -Force -ErrorAction Stop -Scope CurrentUser Tries to locate the config file by resolving it as it is and then trying to locate it in directory and then in directory using the variable (see above). bin config $SCRIPT_PATH Load the configuration data and return the same. All the scripts should be defined with well-defined Parameter list and run the bootstrap as follows $LASTEXITCODE = 1
$SCRIPT_PATH= Split-Path (Resolve-Path $myInvocation.MyCommand.Path)
./$SCRIPT_PATH\bootStrap.ps1
# This line allows passing all command line parameters to bootstrap function
$BootstrapParameters =$PSBoundParameters
$loadModuleList = “Sample.Common-Utils”,”Sample.Az-Utils”
$BootstrapParameters.Add(‘ModulesToLoad’, $loadModuleList)
$ConfigData = BootStrap @BootStrapParameters
# continued code for script Powershell continues to play an important part of automation within the Microsoft ecosystem. Leveraging the recommendations described above can significantly improve the developer experience and reduce the overall time of development for large projects. First published here

Getting Started with Microsoft's PowerShell

Hi, in this article, I am going to create a portable BSD operating system, nomadBSD. It’s quite useful and has a simple function. Let’s dive into the content. What’s BSD? BSD is a high-dimensional system. FreeBSD uses about 40GB, while nomadBSD needs about 5GB or more in the workspace. This system is also based on FreeBSD. However, as a result of improvements and changes, nomadBSD, a portable “persistent” and low-size BSD distribution, has emerged. #BSD and derivatives BSD, or Berkeley Software Distribution, is a plug-in chain created by the University of California for AT&T’s UNIX. Derivatives such as FreeBSD, OpenBSD, NetBSD, DragonFlyBSD refer to UNIX-like operating systems. These systems are licensed systems, and their license is referred to as a BSD license in the literature. These are complex operating systems such as GNU/Linux distributions. The most popular BSD distribution is FreeBSD, and many BSD distributions are based on this distribution. BSD operating systems are extremely powerful and are the operating systems that power many servers. It is stated that BSD operating systems are more performant and, in most cases, noticeably faster than GNU/Linux distributions. In addition, since it is much older than the Linux kernel, it offers a more stable user experience. Although the development speed is slower, the control of the code by a single architect at the top is the most important factor that ensures the stable operation of the code. In many GNU/Linux distributions, the code in the master branch is not controlled by one person, and many people can change the code. BSD systems can also use Linux packages or window managers familiar with GNU/Linux distributions. For this reason, there may be many more software packages available on BSDs. Installing & configuring nomadBSD nomadBSD is a minimal operating system that can be used as a live system, “persistent” and ready to use directly with installation, automatically detecting hardware. 1.2 GHz CPU speed and 1 GB of RAM are recommended as basic requirements. Although it is not beyond USB-2.0, it can be used much more easily with USB-3.0 and above technologies. It was a very pale expectation to expect performance from an operating system that is already persistent and carried on USB. You can boot by selecting your USB from the boot menu of any machine, and the operating system recognizes the hardware and initiates the system boot process. You can download the system using the link. Here, it is explained how to open the archive and how to write the image to the USB memory according to different operating systems. It can simply be written to a USB stick using the dd tool on any operating system. Then you can customize the system with the setup wizard and install it on the USB. The disk encryption feature can be activated optionally during installation. During boot, it waits for you to enter the password you set to connect the disk and continues the boot process after matching. The system is persistent. At the first startup, the installation takes place with the installation wizard, and when the system is restarted, many software that may be needed for daily use such as browser, text-editor, FTP client, a simple code editor, the mail client will appear pre-installed. Since the system is based on FreeBSD, the package manager is pkg. With this command, the system can be updated; the desired package can be installed and removed. The system uses Openbox as the window manager. The lightness and speed of this system are a chosen details. In addition, as seen here, window managers or desktop environments used in Linux systems can also work with BSD. When you want to install FreeBSD, you can install without a window manager. Then you can install desktop environments such as xfce4, KDE, gnome in the pkg repository. Installation and review on Oracle Virtualbox If you want to use it by creating a virtual machine on Oracle VirtualBox, there is a dedicated section in the NomadBSD manual for this. “.img” extension archive is converted into a virtual disk with VDI extension with . VBoxManager $ VBoxManage convertfromraw nomadbsd-130R-20210508.amd64.img \ 
nomadbsd.vdi --format VDI Then the disk size can be adjusted again with the same tool. $ VBoxManage modifyhd nomadbsd.vdi --resize 10000 Then, after creating a new machine on VirtualBox and making the necessary system settings, this VDI virtual hard disk can be connected to the machine. At the first startup, the installation wizard welcomes us. Nothing is visible except the wizard. Do not close the window. Otherwise, you may have to restart the system. Language and keyboard layout selection is made. It prompts again to add a second keyboard layout. You can skip adding. After the timezone selection, the root password is determined. Username is preset as “nomad”. You can encrypt the disk with the block encryption option with Geli. You need to set a password for this. This password is used before connecting the disk during boot. Then the applications and shell to be used by default are selected. In summary, the actions to be taken are examined, and the system setup is started by pressing the “Commit” button. The installation process is completed quickly. When we restart, the boot is started by selecting multiple users. Geli password is entered for disk encryption, and disk binding is performed. If you are using a USB memory stick after the system is turned on, you can make the network connection from the icon in the upper right. Many packages that may be needed for daily use come preloaded. Many tool kits such as visual editor, office suite, mail client, a package manager with visual interface, firewall come pre-installed. You can install many packages with the pkg package manager. For example, package was installed from package repositories below, and from themes was used in the fish shell. neofetch pkg dangerous ohmyfish Like all BSD systems, this is a very similar system to the GNU/Linux distributions that the average Linux user can easily get used to. The system is specially configured so that even beginner users can use it easily. For example, you can perform package update, deletion, and installation operations with OctoPkg graphical package manager. Here you can uninstall apps that come pre-installed but you think you won’t be using. To install Linux packages on NomadBSD, $ sudo -i 
$ sysrc linux_enable=YES 
$ service linux start You can use these commands on shell. Then you can install the Linux packages in the pkg repository with the following command. Tools and databases such as Java, docker, vagrant, MongoDB, ElasticSearch are also available in FreeBSD repositories. It can be easily installed using the pkg package manager. It is also possible to install this system on a disk. But for me, a FreeBSD installation would be much more convenient for a workstation or a personal desktop computer. This system is a stable and FreeBSD-powered operating system designed as a lightweight alternative. For this reason, if heavy work needs to be done, a FreeBSD desktop would be a more appropriate and less tiring choice. For those who want to switch to the BSD side with this distribution, I recommend that they start with GhostBSD first. They can then switch to FreeBSD and OpenBSD, which is a famous security-oriented operating system. This sorting will greatly reduce the feeling of unfamiliarity in the transition process. Also published . here

A Complete Guide to Creating a Portable BSD 

On the road to better healthcare So, how do I see the future of healthcare using AI? Well, let’s just face it. AI is heading to transform medicine and somewhere even replace real-medicine workers. Every year we observe the appearance of new and more advanced solutions. This, by the way, provides a whole slew of advantages, one of the most important of them is reducing the time needed to reach a diagnosis that allows medical workers to better prioritize patient case. Right now there is already a sufficient number of successful developments. AI and deep learning can analyze a lot more factors and cases than live-health workers do. To be more precise, we are able to use AI for genome research, drug development, medical imaging. AI-based devices can learn, analyze large amounts of information, and make decisions much faster than humans. Wow.. isn’t it just amazing? But what does it mean and is it really so? I suggest reviewing some of the recent advancements and startups to figure it out! and also for a healthy life check out these 12 amazing Healthy lifestyle facts . AI & Deep Learning: Current Status and Perspectives in Medicine In medical practice, there is such an important thing as medical intuition. When a patient visits a doctor, the doctor just looks and literally asks a couple of questions and he already knows what is going on and what disease a patient may have. Let’s look at this from a data analysis perspective. So, what is medical intuition? It is like a doctor using his built-in neural network. Having analyzed a sufficient number of cases, a doctor is able, consciously or not, to identify some additional factors that help him narrow down the solution space. And further, he comes to hypotheses, which he considers a possible diagnosis. Is AI better than medical intuition? Whether we like it or not, medical intuition is an automated thing that can be performed not less well by a machine. What is more, some diagnoses can be formulated better by machines. Let’s take a look at the most prominent cases. Top Recent AI Startups in The Medical Field This is a service that allows you to directly communicate with a person and give him your recommendations. The medical application communicates with the patient, asks about symptoms and complaints, and in response gives recommendations, including which doctor to visit, offers to contact a specialist for a remote consultation. Ada. About 20% of lung cancer and breast cancer go unnoticed in the diagnosis. Lunit is developing deep learning and 3D visualization software to help detect difficult-to-diagnose diseases such as airway cancer. Its use increases the probability of detecting cancer to 83–86%. Over time, the technology will improve and accuracy will become even higher. Lunit. This program monitors the condition of people who have recently undergone long-term treatment or suffer from chronic diseases. It was developed by a startup from San Francisco and raised 8 million venture capital investments initially. The application was created in order to structure data on the state of a person, it sends them to a specialist and gives recommendations. Also, the system is able to remind about the time of taking medications and when there is a need to visit a doctor. Sense.ly. The creation of new drugs is one of the most important tasks of modern medicine. The development of a single medicine in a research laboratory costs $ 2.6 billion, in total about $ 150 billion is spent on this process. Therefore, in 2014, only 46 new drugs appeared. Insilico Medicine creates artificial intelligence that will help in the development of drugs, biomarkers and the study of aging mechanisms. The company wants to improve the quality of life of every person. Insilico Medicine. (Boston) was founded by Andy Back, a graduate of Harvard Medical School. Back spent more than five years studying at Harvard and decided to help other doctors with the timely diagnosis of complex diseases, such as cancer, using machine learning to quickly and accurately analyze cell images. PathAI Startup opportunities are used not so much by doctors as by researchers at pharmaceutical companies. Among the startup client’s are such giants of the pharmaceutical industry like Novartis, Gilead Sciences and Bristol-Myers Squibb. (San Diego) helps blind and visually impaired people “see” the world by combining the capabilities of the human body and the capabilities of artificial intelligence in an application or smart glasses. The company claims that their AI development, Chloe, is still at the initial level of its capabilities: it can perform simple tasks like reading instructions on a medicine package, but the developers have big ambitions for its further development. Aira One of the company’s goals is to make navigation possible with computer vision. The product can be used free of charge for sessions shorter than five minutes and for all sessions in more than 25 thousand places where Aira has partnership agreements. For example, JFK Airport or the Wegmans chain of stores. Diagnostics Systems Using Julia Language Such programming languages like Python and R are popular for applications of machine learning. But in recent years, Julia is acquiring its place and has become the new de-facto for machine learning. Julia offers best-in-class support for modern ML-frameworks like TensorFlow and MXNet, making it easy to adapt to existing workflows. Here are some of the powerful projects built on this language. IBM and Julia Computing for Diagnosing Diabetic Retinopathy Diabetic retinopathy is a complication of diabetes, caused by high blood sugar levels damaging the back of the eye (retina). This disease affects more than 126 million diabetics. Timely screening and diagnosis can help prevent vision loss for millions of diabetics worldwide, but many of them lack access to health care. There are over 126.2 million people in the world (as of 2010) with diabetic retinopathy, and this is expected to rise to over 191.2 million by 2030. According to the WHO in 2006, it accounted for 5% of world blindness. Deep Learning in Julia comes to the rescue and helps to significantly reduce this number. By combining Julia’s superior speed and performance with IBM’s Power8 server and NVIDIA Tesla K80 GPU accelerators, researchers increased image processing speed 57x — a dramatic improvement. They used MXNet.jl, a Julia package for deep learning. As a first step, they load a pre-trained model — ImageNet model called Inception with weights in their 39th epoch. On top of that, they specify a simple classifier. contextflow and Julia Help Radiologists Diagnose Difficult Cases Faster The first and foremost problem with the current state of radiologists’ working process is increasing workloads and lack of time. For diagnosing and treating disease, radiologists need to spend more time overall searching for information and a smaller proportion of their time actually interpreting examinations. contextflow’s image search engine in collaboration with Julia made efforts to resolve this dilemma. contextflow is a 3D image-based search engine that uses deep learning to put the knowledge encoded in millions of medical images and reports at fingertips. Simply mark a region of interest in an image, contextflow returns reference cases, and associated knowledge. While radiologists spend up to 20 minutes searching for information, contextflow using Julia promises to cut down search time to ~2 seconds. Taking into account a global shortage of radiologists, this could be a real solution. For doing so, contextflow uses Knet.jl — a great low-level Julia deep learning package, to build the models to identify reference cases for physicians. More Ways AI Is Making an Impact on Medical Care Google Deepmind Predicts an Exacerbation of Kidney Disease Two Days Before It Occurs This deep learning method can continuously predict the risk of future patient deterioration, drawing on recent work that models adverse events based on electronic medical records and using acute kidney damage — a common and potentially life-threatening condition — as an example. The model was developed on a large data set of electronic medical records covering various clinical environments, which included 703,782 adult patients in 172 inpatients and 1,062 outpatient settings. The model predicts 55.8% of all stationary episodes of acute kidney damage and 90.2% of all acute kidney damage that required subsequent dialysis, with a lead time of up to 48 hours and a ratio of 2 false alerts for each true alert. In addition to predicting future acute kidney damage, the model provides confidence estimates and a list of clinical features that are most common for each prognosis, as well as predicted future trajectories for clinically relevant blood tests. Despite the fact that recognition and quick treatment of acute kidney damage is known to be difficult, the developed approach may offer opportunities for identifying patients at risk within a time interval that allows for early treatment. Augmented Reality Microscope with Real-Time Integration of Artificial Intelligence for Cancer Diagnosis Microscopic evaluation of tissue samples plays an important role in the diagnosis and formulation of cancer and, therefore, directs therapy. However, these estimates demonstrate significant variability, and many regions of the world do not have access to trained pathologists. Although artificial intelligence promises to improve access and quality of care, the cost of digitizing images for pathology and the difficulty in deploying AI solutions remain barriers to real use. Augmented Reality Microscope (ARM) superimposes AI-based information on the current sample presentation in real-time, providing seamless AI integration into routine workflows. Researchers demonstrate the usefulness of ARM for detecting metastatic breast cancer and identifying prostate cancer, with latency compatible with real-time use. It is expected that ARM will be able to remove barriers to the use of AI designed to improve the accuracy and efficiency of a cancer diagnosis. Neural Network to Instantly Predict the Tertiary Structure of the Protein Another powerful breakthrough in healthcare made by AI. ProteinNet neural network is able to predict the structure of a protein in milliseconds. Well, predicting the structure of a protein from its sequence is a central biochemistry problem. Convolutional neural networks show promising results. Advances in deep learning that replace complex, human-developed conveyors with differentiable models optimized from start to finish suggest potential benefits. In their study, scientists presented an end-to-end differentiable model to study protein structure. The model links the local and global structure of the protein through geometric units that optimize global geometry without breaking local covalent chemistry. The tested model uses two complex tasks: forecasting new folds without co-evolutionary data and forecasting known folds without structural patterns. In the first problem, the model achieves modern accuracy, and in the second, it is within 1–2 Å; competing methods that use co-evolution and experimental patterns have been refined over the years, and it is likely that the differentiated approach has significant potential for further improvement, ranging from drug discovery to protein construction. Certainly, there are lots of other AI applications in medical care. This means only one thing for me, we have great potential for further development. Final word: What if AI do a mistake? For me now everything looks fine with AI plus medicine except for one thing. While cutting-edge technologies save time, money and serve patients more efficiently, the majority are curious — is it safe enough? What if AI software will give the wrong diagnosis and so on. Who will be responsible for the false results? It’s hard to answer. From the legal point of view, we do not understand who to shift the responsibility to. What’s the solution to this? Well, I think, medical workers should supervise the work of AI. In my opinion, it is highly important for the medicine to change not towards the complete replacement of doctors, but towards symbiosis and joint collaboration with AI. All existing AI-powered systems should act like supporting systems for making medical decisions. Also read - The Healthy Humans blogs Thanks for reading!

Artificial Intelligence & Deep Learning for Medical Diagnosis

This tutorial describes how to create a simple currency exchange app in just about 100 lines of code using the Create React App boilerplate. To see it in action check out this . live demo 🛠️ Preparation First, create the react app. I call it "rates". $ yarn create react-app rates Run the command in your root $ yarn start Your web app will be available on http://localhost:3000 ⚓ Fetching the currencies rate For currency exchange data we will use free API on exchangeratesapi.io We will load the latest data of the currencies rates. Install the and libraries. It helps to easily fetch API. swr unfetch $ yarn add swr unfetch Create the fetcher function outside the App component React ; ; fetch ; API_URL = ; fetcher = path => { res = fetch(API_URL + path); json = res.json(); json;
}; // src/App.js import from "react" import "./App.css" import from "unfetch" const "https://api.exchangeratesapi.io" const async const await const await return // function App... Add in App component to fetch data useSWR React ; ; fetch ; useSWR ; { { : currencies } = useSWR( , fetcher); ;
 } import from "react" import "./App.css" import from "unfetch" import from "swr" // API_URL = ... // const fetcher = ... ( ) function App const data "/latest?base=EUR" return Welcome to your Currency exchange app! < > div </ > div Now we have the currencies rates in variable. currencies.rates To see the JSON data of currencies rates you can open the API url directly into your browser: https://api.exchangeratesapi.io/latest?base=EUR The query is used to get all rates relative to EUR currency. ?base=EUR 💄 Add the UI For ui, we will use the design system. Install it: material-ui $ yarn add @material-ui/core Create the UI for our currency exchange app in the component. App // ...
import {
  Container,
  Paper,
  Grid,
  TextField,
  Select,
  MenuItem
} from "@material-ui/core";

// ...

function App() {
  const { data: currencies } = useSWR("/latest?base=EUR", fetcher);

  if (!currencies) {
    return null;
  }

  return ( Currency exchange EUR {Object.keys(currencies.rates).map((rate, key) => ( {rate} ))} EUR {Object.keys(currencies.rates).map((rate, key) => ( {rate} ))} );
} < = > Container className "currency-exchange-container" fixed < > h1 </ > h1 < = = = > Paper className "currency-exchange-paper" variant "outlined" elavation {1} < = > Grid container spacing {3} < = > Grid item xs {6} < = /> TextField type "number" </ > Grid < = > Grid item xs {6} < = /> TextField type "number" </ > Grid < = > Grid item xs {6} < > Select < = " "}> MenuItem value { EUR </ > MenuItem < = = > MenuItem key {key} value {rate} </ > MenuItem </ > Select </ > Grid < = > Grid item xs {6} < > Select < = " "}> MenuItem value { EUR </ > MenuItem < = = > MenuItem key {key} value {rate} </ > MenuItem </ > Select </ > Grid </ > Grid </ > Paper </ > Container Now, look at the following code snippet. We create the currency selection box by iterating over all possible rates from our API data. Since we fetch rates relative to EUR, so we should manually add the EUR item, because it doesn't exist in rates object. <Select> { .keys(currencies.rates).map( ( ))}
  < EUR < = " "}> MenuItem value { EUR </ > MenuItem Object ( ) => rate, key {rate} < = = > MenuItem key {key} value {rate} </ > MenuItem /Select> When the currencies are not loaded, we just return null, because of currencies rates is undefined at that moment. (!currencies) { ;
  } if return null Add some styles to App.css { : ; : ; : ;
} { : flex; : column; : center; : center; : ;
} { : ; : ;
} { : ;
} h1 font-weight 300 color #636363 margin-bottom 3rem .currency-exchange-container display flex-direction align-items justify-content height 100vh .currency-exchange-paper max-width 350px padding 30px 30px 40px 30px .MuiInput-root width 100% ⚙️ Add logic Now we add the inputs state to our component App { { : currencies } = useSWR( , fetcher); [fromValue, setFromValue] = useState( ); [toValue, setToValue] = useState( ); [fromCurrency, setFromCurrency] = useState( ); [toCurrency, setToCurrency] = useState( ); handleFromCurrencyChange = {
    setFromCurrency(e.target.value);
  }; handleToCurrencyChange = {
    setToCurrency(e.target.value);
  }; handleFromValueChange = {
    setFromValue( (e.target.value));
  }; handleToValueChange = {
    setToValue( (e.target.value));
  }; (!currencies) { ;
  } ( <h1>Currency exchange</h1> <Paper
        className="currency-exchange-paper"
        variant="outlined"
        elavation={1}
      >
        <Grid container spacing={3}>
          <Grid item xs={6}>
            <TextField
              type="number"
              value={fromValue}
              onChange={handleFromValueChange}
            />
          </Grid>
          <Grid item xs={6}>
            <TextField
              type="number"
              value={toValue}
              onChange={handleToValueChange}
            />
          </Grid>
          <Grid item xs={6}>
            <Select value={fromCurrency} onChange={handleFromCurrencyChange}>
              <MenuItem value={"EUR"}>EUR</MenuItem>
              {Object.keys(currencies.rates).map((rate, key) => (
                <MenuItem key={key} value={rate}>
                  {rate}
                </MenuItem>
              ))}
            </Select>
          </Grid>
          <Grid item xs={6}>
            <Select value={toCurrency} onChange={handleToCurrencyChange}>
              <MenuItem value={"EUR"}>EUR</MenuItem>
              {Object.keys(currencies.rates).map((rate, key) => (
                <MenuItem key={key} value={rate}>
                  {rate}
                </MenuItem>
              ))}
            </Select>
          </Grid>
        </Grid>
      </Paper>
    </Container>
  );
} ( ) function App const data "/latest?base=EUR" const 1 const 1 const "EUR" const "EUR" const => e const => e const => e parseFloat const => e parseFloat if return null return < = > Container className "currency-exchange-container" fixed Also, add the two following functions for currency exchange convertFromTo = { fromRate =
      fromCurrency === ? : currencies.rates[fromCurrency]; valueInEur = fromValue / fromRate; toRate = toCurrency === ? : currencies.rates[toCurrency];
    setToValue(valueInEur * toRate);
  }; convertToFrom = { toRate = toCurrency === ? : currencies.rates[toCurrency]; valueInEur = toValue / toRate; fromRate =
      fromCurrency === ? : currencies.rates[fromCurrency];
    setFromValue(valueInEur * fromRate);
  }; const => () const "EUR" 1 const const "EUR" 1 const => () const "EUR" 1 const const "EUR" 1 One function converts currencies forward, and other - backward. In both functions, firstly, we convert currencies to EUR, because all rates we fetched from API are relative to euros. The last thing is to add the React hooks which run the rates exchange after the input change. useEffect( {
    convertFromTo();
  }, [fromValue, toCurrency]);

  useEffect( {
    convertToFrom();
  }, [toValue, fromCurrency]); => () => () Here is a full file App.js React, { useState, useEffect } ; ; fetch ; useSWR ; {
  Container,
  Paper,
  Grid,
  TextField,
  Select,
  MenuItem
} ; API_URL = ; fetcher = path => { res = fetch(API_URL + path); json = res.json(); json;
}; { { : currencies } = useSWR( , fetcher); [fromValue, setFromValue] = useState( ); [toValue, setToValue] = useState( ); [fromCurrency, setFromCurrency] = useState( ); [toCurrency, setToCurrency] = useState( ); handleFromCurrencyChange = {
    setFromCurrency(e.target.value);
  }; handleToCurrencyChange = {
    setToCurrency(e.target.value);
  }; handleFromValueChange = {
    setFromValue( (e.target.value));
  }; handleToValueChange = {
    setToValue( (e.target.value));
  }; convertFromTo = { fromRate =
      fromCurrency === ? : currencies.rates[fromCurrency]; valueInEur = fromValue / fromRate; toRate = toCurrency === ? : currencies.rates[toCurrency];
    setToValue(valueInEur * toRate);
  }; convertToFrom = { toRate = toCurrency === ? : currencies.rates[toCurrency]; valueInEur = toValue / toRate; fromRate =
      fromCurrency === ? : currencies.rates[fromCurrency];
    setFromValue(valueInEur * fromRate);
  };

  useEffect( {
    convertFromTo();
  }, [fromValue, toCurrency]);

  useEffect( {
    convertToFrom();
  }, [toValue, fromCurrency]); (!currencies) { ;
  } ( <h1>Currency exchange</h1> <Paper
        className="currency-exchange-paper"
        variant="outlined"
        elavation={1}
      >
        <Grid container spacing={3}>
          <Grid item xs={6}>
            <TextField
              type="number"
              value={fromValue}
              onChange={handleFromValueChange}
            />
          </Grid>
          <Grid item xs={6}>
            <TextField
              type="number"
              value={toValue}
              onChange={handleToValueChange}
            />
          </Grid>
          <Grid item xs={6}>
            <Select value={fromCurrency} onChange={handleFromCurrencyChange}>
              <MenuItem value={"EUR"}>EUR</MenuItem>
              {Object.keys(currencies.rates).map((rate, key) => (
                <MenuItem key={key} value={rate}>
                  {rate}
                </MenuItem>
              ))}
            </Select>
          </Grid>
          <Grid item xs={6}>
            <Select value={toCurrency} onChange={handleToCurrencyChange}>
              <MenuItem value={"EUR"}>EUR</MenuItem>
              {Object.keys(currencies.rates).map((rate, key) => (
                <MenuItem key={key} value={rate}>
                  {rate}
                </MenuItem>
              ))}
            </Select>
          </Grid>
        </Grid>
      </Paper>
    </Container>
  );
}

export default App; import from "react" import "./App.css" import from "unfetch" import from "swr" import from "@material-ui/core" const "https://api.exchangeratesapi.io" const async const await const await return ( ) function App const data "/latest?base=EUR" const 1 const 1 const "EUR" const "EUR" const => e const => e const => e parseFloat const => e parseFloat const => () const "EUR" 1 const const "EUR" 1 const => () const "EUR" 1 const const "EUR" 1 => () => () if return null return < = > Container className "currency-exchange-container" fixed ✨ Finished! Congratulations! You have done the currency exchange app using the CRA (Create React App). The full source code you can find in my repository . epranka/rates . Live demo Thanks for reading this. I hope it was helpful to you. Feedback and questions are appreciated. Follow on , , and let’s connect on Twitter GitHub LinkedIn (Originally published ) here

Build Your Own Currency Exchange Web App in ~100 lines using CRA [A Step By Step Guide]

A CLI tool that converts exported Medium posts (HTML) to Jekyll/Hugo compatible markdown with front matter medium-2-md is a simple CLI tool which takes a directory containing Medium posts’ HTML files (or a single post URL) and converts them into markdown. Not only that, but it also adds a rich front matter to these converted markdown files so that they can be directly used in Jekyll or Hugo. The CLI tool is written with node.js. It expects the input HTML files to have the same tags and attributes as the files contained in the posts directory shown in the picture above. That way it is able to extract all the information needed for the front matter. It has not been tested for general purpose HTML to markdown conversion; it is only suitable for HTML files exported from Medium stories/posts. Usage medium-2-md is available as an so that it can be easily downloaded and used without the need of going through the code. All you need to do is download and install it (globally) and then use it directly from the command line. Follow the steps listed below to use medium-2-md. npm package Step 1: Export and extract your Medium posts from your Medium account. Go to and scroll to . Click the download button. This will give you a archive containing all your Medium content. https://medium.com/me/settings Download your information medium-export.zip Extract the .zip archive downloaded in the previous step. It will have a sub-directory called . posts Copy the path of this directory. posts Step 2: Install node.js and medium-2-md on your system. Download and Install node.js — [https://nodejs.org/en/download/](https://nodejs.org/en/download/.) . Install medium-2-md package — . npm i -g medium-2-md Note: The flag means that you are installing the package globally. It is essential in order to use the package directly from the command prompt/terminal. -g Step 3: Run the following command to convert all your Medium posts (HTML) to markdown files. medium-2-md convertLocal '<path of the posts directory>' -df That’s it. The output markdown files will be stored in a sub-directory called in the posts directory itself. (By the way, that big number is coming from the Date.now() JavaScript function, added to differentiate the output folders in case we go crazy with it.) md_<a big number> Convert from post URL The CLI tool can also be used to convert a single Medium post from it’s url. It requires an additional parameter for output directory where the converted markdown file can be written. To convert a Medium post to markdown from it’s url, run the following command after installing the npm package, -o medium-2-md convertUrl '<url of the Medium post>' -o '<path of output directory>' -f The converted markdown files will also have front matter which will have the title, description, published date and canonical URL of the original Medium post/story. More details about the package are available in the readme file at the . medium-2-md GitHub repository This tool helped save a lot of time when I was converting all my stories/posts to markdown. I tested the converted markdown in a test Hugo site and it worked like a charm. A couple of things that we need to take care manually are code fences and tags. There is no tag related information in the exported Medium files. The code fences are also exported as plain text and the markdown conversion ignores them too. Give it a try and feel free to provide your feedback using the issue tracker on the GitHub repository — . https://github.com/gautamdhameja/medium-2-md Originally published at www.gautamdhameja.com on December 30, 2018.

medium-2-md: Convert Medium posts to markdown with front matter

by Parth Shrivastava Good design, impeccable content and wonderful CTAs were once some of the most essential aspects of putting a website on the internet. Lately, Live Chat has been added to this list. Nowadays, every website you see is equipped with a Live Chat widget. Reasons? A research by says that 77% of customers will not prefer to make a purchase on a website if there’s no live chat option available. Also, a staggering 79% of people chose live chat as their preferred medium of customer support. FurstPerson The boom is here and everyone knows it. Hence, tons of companies are coming up with their live chat solutions. In this post, I will: list down some of the most widely used live chat platforms discuss their primary use case share some insights on their pricing(I have mentioned monthly pricing for most of them until otherwise stated specifically) Note that, this is just a comparative guide. Each and every business has a different use case, hence I leave the decision to choose the best one for you. Intercom Intercom has seen the live chat evolution and is one of the early players. It is widely loved and has a strong fan base across geographies. It has 3 primary products; Messages — for targeted email marketing, Inbox — live chat and Articles — knowledge base management. Use Case Intercom is widely used for targeted marketing using chat and email. It is primarily used for marketing and sales. They also have recently opened their App Store for developers to build apps on top of Intercom. Pricing All the three products of Intercom have different pricing. For Messages, the pricing is based on the number of users/contacts you have. Inbox has per seat pricing. The pricing for both has feature tiers as well. Articles have a fixed price of $49/month. It comes with a free 14 days trial and if you buy all three of the products, you can start with $136/month. Most of the reviews I have read have quoted Intercom as a bit pricey. But as they say quality comes with a cost. Drift Drift pitches itself as the world’s first conversational marketing platform. They have a powerful live chat with chatbot which does a lot of tasks such as qualifying leads, scheduling meetings and rendering FAQs in chats etc. Use Case Drift is used for sales and marketing. Everything about Drift is very conversational and it can be seen from their products as well. They specialize in highly targeted Account-Based Marketing. Pricing Drift comes with a free plan which gives you one seat and 100 contacts along with a few basic features. The pricing is based on feature tiers and ranges starts from $50/month for standard plan and $500/month for pro plans. Crisp Crisp is a simple but powerful live chat software. It also has knowledge base management, video calls and marketing campaigns as additional important features. Use Case Though it has the option to send marketing emails to your users, Crisp is primarily used for customer support chat. Also, the features align with customer support use case heavily. Pricing Crisp has a free forever plan, come with basic live chat, 2 seats and 30 days chat history. Next tier, the Pro plan, has features such as triggers, unlimited chat history, and team inbox. It starts at $25/month for 4 seats. The Unlimited plan has all the features such as campaigns and knowledge base and starts at $95/month for 4 seats. Kommunicate Kommunicate is a human+bot hybrid live chat software. Kommunicate boasts of custom chatbots, email+chat support, FAQs, chatbot-human handoff and provides dedicated web plugin and mobile SDKs for native and hybrid apps. Use Case Kommunicate is made for customer support. Though you can use Kommunicate for sales and marketing as well. But most of the features resonate with customer support and service use case. The major focus is automating support using a collaboration of humans and bots. Pricing It comes with a free forever plan with 2 seats, basic reporting, and 90 days chat history. The paid plans start from $10/active agents with additional features such as chatbots, third-party integrations, agent-wise reporting etc. They have dedicated data hosting plans as well with custom Enterprise plans as well. Olark Olark is your traditional straightforward live chat tool which is made to help you close more sales and support more customers. Use Case Olark has features tailor-made for customer support and success. To name a few — automated messages, agent reports, and team management. Pricing Olark being no bullshit straightforward live chat software has fairly simple pricing. The pricing is based on the number of agents and starts from $17/month per seat. HelpCrunch Helpcrunch is customer communication platform. It has features to help you serve your users from lead to customer stages such as live chat, in-app messenger, auto messages, ticketing, and email automation. Use Case Helpcrunch can be used at any stage of customer service. They have live chat for sales, ticketing for customer support and email automation campaigns for marketing. Pricing Their pricing has feature tiers and is also based upon the number of agents. The Basic plan starts with $12/agent/month but you can only add one agent and use it only for 1 app or website. For multiple agents and apps, you have to move to their Premium ($21/agent/month)or Enterprise (custom)plans. Acquire Acquire is a sales and support automation platform. They have live chat, customer profile management, sales and support chatbot, co-browsing capabilities and voice/video calls. Use Case Acquire has features which resonate with sales and customer support use case. They have chatbots that can bring in automation in your sales and support efforts. Pricing Aquire’s pricing is agents based and is also tiered by features. They also have a Free plan with 250 customer profiles/contacts and one agent. The Premium plan starts from $35/agent/month with sales and support features. Chatbots and automation are part of Team plan, starting for $200/team(5 agents)/month. Freshchat Freshchat is a customer messaging software from Freshworks’ suite of products. It has multiple features such as live chat, marketing campaigns, chatbots, and knowledge base management etcetera. Use Case Freshchat have features which primarily cater to customer support and sales. In-app campaigns and triggers can be used for marketing as well. They also have features for customer self-serve. Pricing Their pricing is based upon the number of active agents you have. Freshchat comes with a Free plan as well where you can add up to ten agents and 10K monthly unique visitors. Paid plans have feature tiers as well, starting from $20/agent/month. The full feature suite costs you $50/agent/month. Zendesk Chat Zendesk is one of the most widely used ticketing software for customer support. It was just the matter of time when they ventured into live chat. Zendesk Chat is a simple and straightforward chat solution for leaders in the customer support industry. Use Case Zendesk Chat is made for customer support. They also have lead collection forms for sales enablement and chatbots for automation. Pricing Zendesk Chat have agent based pricing along with feature tiers. They have a Free plan, where you can add one agent and retain 14 days’ history. The paid plans start from $20/agent/month to full feature tier at $70/agent/month. Tawk.to Tawk.to is a free live chat software. It is equipped with basic live chat features without any fancy frenzy. It has canned responses, real-time monitoring, localization, dashboard reporting etc. Use Case You can use Tawk.to for customer support and sales. Pricing It is Free to use. Really! They just charge you if you want to remove Tawk.to branding, which costs you $15/month. Customerly Customerly is feedback, support and automation platform. It has multiple products — Customer support live chat, Email marketing and automation, and Survey and satisfaction. Use Case Customerly’s features are made to assist you in customer support and marketing. The live chat product also has video calling capabilities. Pricing Each of the three aforementioned products has separate pricing. The Customer support live chat product is priced at $29/month for 10k contacts. They also have a free plan where they give the whole suit free for up to 500 contacts. Pure Chat Pure Chat is similar to Tawk.to — no fancy frenzy but good old live chat software. Use Case Pure Chat is made for customer support. It can also be used for lead collections and sales enablement. Pricing Pure Chat is Free to use for three agents and one website. The Pro plan starts at $99/month which includes unlimited agents and contacts, and advanced features such as contact enrichment, visitor tracking, unlimited websites, and real-time analytics. ConvertFox ConvertFox is a live chat, email marketing, and user analytics tool. They also have knowledge base management, lead qualification bot, ticketing, and marketing automation. Use Case ConvertFox can be used for marketing, sales, and customer success. Pricing ConvertFox has agent based pricing with feature tiers. They have a free plan with one agent and 500 contacts with basic live chat. The paid plans with advance marketing and customer success features start at $29/month for one seat and 2500 contacts. Their Pro plan, which has all the features, costs $79/month for one seat and 2500 contacts. Additional seats come at an extra cost. Kayako Kayako is a live chat software for customer service and helpdesk support. They are primarily a ticketing software which also released live chat product recently. Use Case Given Kayako is a ticketing software, their features are mostly suited for customer support. Pricing Kayako’s pricing is similar to most ticketing software. It is based on the number of agents and has feature tiers. The pricing starts from $15/agent/month. Their Enterprise plan, which has full feature suite, starts from $100/agent/month. Tidio Tidio provides a common place for all your customer conversation through live chat, email and Facebook messenger. Use Case Tidio can be primarily used for customer success and support. The live chat and proactive triggers make it useful for customer success. Pricing Tidio is free to use for up to three agents. Note that typing indicator, auto-reply, initiating a chat with customers is not available in the free plan. The live visitor attributes are available in the next plan starting from $15/month for three agents. The customer success automation part is priced at $15/month for 5000 contacts. Chatra Chatra is simple chat software for live and offline messaging. Chatra has simple and powerful features such as group chats, localization, auto messages, and lead collection forms. Use Case Chatra has features that can be used for sales and marketing (such as cart abandonment and email list building). Pricing Chatra has a free forever plan where you can have one online agent and basic live and offline chat features. The premium sales and marketing features can be availed from their paid plan. It is agent-based and costs $19/agent/month. Chaport Similar to Tidio and Tawk.to, Chaport is a simple and to-the-point live chat software. It provides live chat, saved replies, reports, localization and contact enrichment, among other live chat features. Use Case Primarily used for customer support live chat, Chaport can be used for sales as well. Pricing Again, Chaport’s pricing is similar to Tidio. They have a Free plan with unlimited agents with up to 5 online at a time. Premium features such as unlimited agents online at a time, saved replies, contact enrichment, typing insights are the part of Pro plan which starts from $14/agent/month. ProProfs Chat ProProfts has a large suite of products for marketing, customer service, sales, entertainment etc. ProProfs Chat is one of those products. They have features like co-browsing, canned responses, lead capture forms, and feedback gathering forms. Use Case ProProfs Chat can be used for customer support and sales Pricing ProProfs has feature ties with a cap on the number of agents in each tier. They have a free plan with one agent, one-month chat history and 100 chats/month. The premium plans start at $9/month. The full feature chat comes for $199/month. Collect.chat Collect.chat is a chatbot-powered live chat solution. They have a chatbot builder interface as well. Hence primarily they are into live chat automation. Use Case Collect.chat can be used for customer support, sales, and surveys. Their drag and drop chatbot builder makes it easy to make conversational interfaces. Pricing Collect.chat ahs a free plan with 30 responses a month and 30 days of chat history. The paid plans are tiered on the basis of the number of responses and features. They start at $19/month and full feature suite cost you $60/month for 5000 responses. Zoho SalesIQ Zoho SalesIQ is a live chat software from Zoho’s suite of products. They have features such as email campaigns, user activity tracking, lead scoring and reporting, among others. Use Case Zoho SalesIQ is primarily used for sales, customer support, and marketing. Pricing As most of the software listed here, SalesIQ also has a free plan for up to 5000 monthly contacts with 2 agents. The paid plans are based on the number of agents and total monthly contacts. They start at $19/month and the Enterprise plans cost you $129/month. And now the decision rests on your and your business use case. There are plenty of fishes in the see, you just need to catch the right one. Also, as this list is non-exhaustive, I will keep appending it. If you have any suggestions, please share with me in comments. Also, which one is your preferred live-chat software? I would love to know about it. PS: The information shared here is best of my knowledge and collected through various sources. If you belong to the aforementioned organizations and wish to update any information OR want your product to be added to this list, DM me on . Twitter This article was originally posted here .I talk about all thing B2B marketing and writing(non-fiction and fiction), currently working as an Inbound Marketing Lead at Applozic, Inc .

Stories, News & Mentions about CONVERTFLOW

Convertflow’s Stories on HackerNoon