The Yukon government is on track to post a $3.6-million surplus this fiscal year, according to the territory's fall economic update. That's down from more than $48 million this past spring, when ...

Fires, mine closure, public sector spending carve chunk out of Yukon surplus

Two U.S. defense officials have confirmed to Fox News that the U.S. intercepted two one-way attack drones targeting Iraq's al-Asad air base where American troops are located. The incident happened ...

US military intercepts 2 attack drones targeting Iraq air base where American troops are located

The new foodservice collaborations are a direct result of Chunk Foods’ partnership with the Talk of the Town Restaurant Group, enabling the company to reach a broader consumer base with its cholesterol-free Chunk steak, made from fermented soy and wheat.

Chunk Foods’ Plant-Based Steak Expands Menu Presence in Florida Steakhouses and Seafood Restaurants

A drive against ticketless travel revealed that college-goers formed a sizeable chunk of those taking a free ride, particularly in the Tambaram-Chengalpattu route of Chennai Division. Officials ...

Students form sizeable chunk of ticketless travellers in Chennai suburban trains

Parts of Western Hills Viaduct will be closed Wednesday and Thursday for emergency repairs after chunks of concrete fell from the top to the bottom deck of the 91-year-old structure. The middle ...

Falling 'chunks of concrete' prompt closing of parts of Western Hills Viaduct

It was much larger than the basketball-sized chunk that fell near the court's front entrance in 2005. The weight of the marble that fell is unknown, but the Georgia marble used in the court's four ...

Supreme Court avoided disaster when a chunk of marble fell in a courtyard used by justices

It was much larger than the basketball-sized chunk that fell near the court’s front entrance in 2005. The weight of the marble that fell is unknown, but the Georgia marble used in the court’s ...

We may earn revenue from the products available on this page and participate in affiliate programs. Learn More › A great base layer is essential to operating at peak performance in the outdoors. Sure, we all know that wearing cotton is bad: it holds onto ...

The Best Base Layers of 2023

Keith McNally may want to delete his Instagram account. The famed restaurateur went on an epic rant Thursday about losing a “big chunk” of business — and being threatened with having his ...

Balthazar owner Keith McNally fumes over losing ‘big chunk’ of business after inflammatory Israel post

Get the most recent info and news about Chunk Base on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

A collection of apps to help you find the best Minecraft seeds, and locate biomes and structures in your world on an interactive map.

Chunk Base

We often work on building applications where a single application might scale to multiple instances with couple of changes. A good example for this case will be, when you are building an application for multiple countries. Different countries might have different requirements i.e different sections or different form fields, but the base application i.e common part should remain same. TLDR; This becomes even challenging when the application is already build and in later stages of application lifecycle, it needs to scale to multiple countries. What are the approaches people follow in the market ? - Fork & maintain multiple repositories (Hard to believe, but happens) - Multiple replica copies country-wise in same repo (Doesn’t make sense) - IF-ELSE-IF-ELSE-IF ladder, hell! (Sprint ending this week, let’s wind it up) - A/B testing Segmentation (We’re Frontend Developers not Devops) - and on… The solution should prevent us from maintaining multiple repositories, prevent DRY (Don’t Repeat Yourself), encourage separation of concerns, doesn’t ship not required code, ease development & deployments which should be done from Frontend. Mixins & Dynamic Imports to the rescue !! (Presenting this concept in Vue Conference, Bangalore, India) Let's continue: Quick Refresher- Mixins (Vue): It is a flexible way to distribute reusable functionalities for Vue components, you can even define lifecycle hooks in it - It can be imported in multiple components - Mixins can be in your country specific component code Overridden - Only override the function which need changes over the function imported from Mixin, Vue will manage the merge of both Dynamic imports (Webpack): This allows you to split your code into various bundles which can be country specific javascript chunks, loaded on demand or in parallel How to implement this ? We will have one parent AppBody in each country folder, it will contain multiple reusable child components like <user-profile-form> required. < > template < > div < > user-profile-form </ > user-profile-form < > user-password-reset </ > user-password-reset </ > div </ > template The child component will import and override the base mixin app/ feature code. Let below be code of one of the child component of AppBody.vue parent file. userInfoMixin Utils ; { : [userInfoMixin], methods: { 
      setTheme(data){
        Utils.setTheme( );
      }
    },
    mounted() { .countryCode = ; .setTheme();
    }
  }; import from "../../mixins/UserInfoMixin.js" import from '../../common/Utils' export default mixins //Base code 'pt-br' this "+55" this Now our Directory Structure: Above we have country specific folders to maintain separation of concerns, respective country developer can’t mess with any other country code. We have to keep above discussed child component in the respective country folder, the base code will go in the common folder. How to switch countries ? userInfoMixin Utils ; { : [userInfoMixin], methods: { 
      setTheme(data){
        Utils.setTheme( );
      }
    },
    mounted() { .countryCode = ; .setTheme();
    }
  }; import from "../../mixins/UserInfoMixin.js" import from '../../common/Utils' export default mixins //Base code 'pt-br' this "+55" this The AppBody is the parent component, different for different countries holding the child components which are inheriting from JS & SCSS Mixins Apart from invoking using query params, you can use webpack variables which will help you in build & deployment process using: REGION=INDIA npm run build Running the above command will give us region/country chunks (0,1.. below) which are loaded dynamically (only specific country chunk)after the base app code, which is just app.js 53.3 kb. So in the end, only your country code is loaded. apart from base app.js. Extending CSS Using Mixins of SCSS: Now adding SCSS code to our AppBody.vue file. @ ; { { : ; : ; : solid ; : ; : ; : ;
 	} {
 		appearance: none; : transparent;
 		-webkit-appearance: none;
 		&: : -webkit-inner-spin-button, &: : -webkit-calendar-picker-indicator, &: : -webkit-clear-button {
 			display: none;
 			-webkit-appearance: none;
 		}
 	} import "../../assets/scss/component-styles/userBasicInfo.scss" //base code .basic-info /*overriding with country specific requirement */ input width 100% padding 10px 0 border-bottom 1px #e6e6e6 font-size 16px color #000000 border-radius 0 input [type='date'] background Now, above methodology ensures the parent code which is same, is not re-written and only the different requirement needs to be written by overriding the parent function in the inheriting JS Function/CSS Class. The separate country code chunk will be loaded dynamically during rendering after the base app code, Also the build process can be made country wise by passing webpack variables during the deployment, which means one country release will be independent of others. The above method is one of the possible solutions. If you came across the same requirement, I would love to hear the solution from you.

Scaling Vue App for Multiple Instances

Intro Secure or private file storage? What is the difference? If your application provides secure file handling, it means that only you and the application you are relying on own your data. e.g., Dropbox and Google Drive. But, if you need true privacy and don’t want the company you rely on to share or sell your files with a 3rd party, you need to think about moving to private solutions like Tresorit or Sharekey. In this article, I will  try to explain how to speed up file uploading for private applications. Here is the image illustrating the process of how app private file uploading should work. Intro about the file uploading process File uploading is a trivial and pretty common task for web development. There are plenty of articles about how to upload files using JavaScript from the web. And here, I will explain a simplified mechanism of how to do it. If we are talking about the general approach of file uploading, it consists of several steps: Reading files from devices Preparing files for uploading Uploading a file. If we are talking about the 1st and 3rd points, the approach is quite clear. Let’s add some notes about the 2nd point: Here, to exclude problems, if somewhere in the process of uploading a file, the error is thrown, the process of uploading should be restarted from scratch. Also, if you are going to transmit huge files through the network, you can face performance issues and even get a 413 error (Large payload). The solution to walk around such issues is quite simple: we need to just split the file into chunks. After this, we can add encryption to encrypt each chunk, and we are good to go. Encryption Encryption is the heart of the private application. In short, encryption is a way of scrambling data so that only authorized parties can understand the information. It helps to hide your data from anyone who has no access to it. JS, like many other programming languages, brings you the possibility to add encryption to your application. But a bit differently. Commonly the process of encryption is based on manipulating binary data. Traditionally the JavaScript language didn’t really support interaction with binary data. Sure, there were strings, but they really abstracted the underlying data storage mechanism. There were also arrays, but those can contain values of any type and aren’t appropriate for representing binary buffers. Eventually, the ArrayBuffer object was created and now is the core part of the language. An instance of ArrayBuffer represents a buffer of binary data that is of fixed length and cannot be resized. Instead, a “view” into the buffer must first be created. Due to the ambiguity of what binary data can mean, we need to use a view to read and write to the underlying buffer. There are several of these views available in JavaScript e.g. , Uint8Array Int8Array… Why did we mention it here? The first thing is that to encrypt our file, we need to first represent it as and then manipulate its bytes in order to make it impossible to understand by a third party. And the second thing is SharedArrayBuffer(but this will be sooner). TypedArray Okay, we finished with the representation of the file. The one thing left is the process of encryption. I don’t think this is the right article to dive deep into the nuances of the encryption process. Let’s just use some open-source solution for encryption. In this, I will use the package, which is as close as possible to the original C implementation and which is quite easy to use. tweetNaCL.js So let’s start implementation and see what we can optimize. First steps and the first problems. Let’s start with the file reading process, which is quite straightforward. Let’s say we have input with type which has id=”file_handler”. file let chunksQueue = [];

const fileInput = document.getElementById('file_handler');

fileInput.addEventListener('change', function () {
  const file = fileInput.files[0];

  readFile(file).then((res) => {
    const chunksAmount = getChunksAmount(res.length);
    chunksQueue = getChunksQueue(chunksAmount);
    sendChunk(res).catch((e) => console.log(e));
  });
}); The method for reading files is Uint8Array, which is the preferred format for encryption. function readFile(file) {
  return new Promise((resolve) => {
    const fileReader = new FileReader();

    fileReader.onload = function () {
      const data = fileReader.result;
      const fileUint8Array = new Uint8Array(data);

      resolve(fileUint8Array);
    };

    fileReader.readAsArrayBuffer(file);
  });
} Also, let’s define - the size of one file chunk. This size will be calculated like . Based on chunksAmount, we will organize chunksQueue, which will contain file chunks that should be transferred to the server. CHUNK_SIZE Math.ceil(file.size / CHUNK_SIZE) function getChunksAmount(fileSize) { return Math.ceil(fileSize / CHUNK_SIZE) } 

const getChunksQueue = (chunksQuantity) => {
  return new Array(chunksQuantity).fill().map((_, index) => index);
}; function, which is responsible for encrypting, uploading, and retrying uploading in case of error: sendChunk async function sendChunk(file, fileName) {
  if (!chunksQueue.length)
    return;

  const chunkId = chunksQueue.shift();
  const begin = chunkId * CHUNK_SIZE;
  const chunk = file.slice(begin, begin + CHUNK_SIZE);

  const encryptedChunk = encrypt(chunk, secretKey);

  upload({ fileChunk: encryptedChunk, fileName, chunkId })
    .then(() => {
      sendChunk(file, fileName);
    })
    .catch(() => {
      chunksQueue.unshift(chunkId);
    });
} is the function for encrypting chunks: encrypt function encrypt(chunk, secretKey) {
  // creating nonce which should also be transferred to server, but not in this article
  const nonce = nacl.randomBytes(24);

  // chunk encrypted with nonce and secret key
  const encryptedChunk = nacl.secretbox(chunk, nonce, secretKey);

  // base64 encoded typed array to be sent to the server
  return nacl.util.encodeBase64(encryptedChunk);
} Don’t forget to create the user’s secretKey, which will be used for encrypting and decrypting files: const secretKey = nacl.randomBytes(32); And the last client function - the function for uploading file chunks to the server: async function upload({ fileChunk, fileName, chunkId }) {
  return fetch(`${BACKEND_URL}/upload`, {
    method: 'POST',
    body: JSON.stringify({ fileChunk, fileName, chunkId }),
    headers: {
      'Content-Type': 'application/json',
    },
  })
    .then(response => response.json())
    .then(res => {
      if (res.status === 200)
        return 'ok'

      console.error('Server Error');
  });
} The last step will be handling file chunks on the server. Firstly let’s create a simple server.js file that will be responsible for running the express server.  There would be two endpoints: one for serving the client static index.html file and another one for the API route for uploading file chunks. All chunks will be stored on the filesystem related folder(uploaded). // server.js
const express = require('express');
const path = require('path');
const fs = require('fs');
const bodyParser = require('body-parser');

const app = express();

const hostname = '127.0.0.1';
const port = 5000;

// increase memory for request to exclude 413 error
app.use(express.json({ limit: '500mb' }));
// for parsing body of request
app.use(bodyParser.json());
// for accessing js files from index.html
app.use(express.static('public'));

app.get('/', (req, res) => {
  res.sendFile(path.join(__dirname, './index.html'));
});

app.post('/upload', (req, res) => {
  const { fileChunk, fileName, chunkId } = req.body;
  const directoryPath = path.join(__dirname, `./uploaded/${fileName}`);

  // create folder for fileChunks in the file system
  fs.mkdir(directoryPath, (e) => {
    // if folder already exists, do not throw error
    if (e && e.code !== 'EEXIST') {
      console.error(e);

      res.status(500).send({ status: 500, error: e.message });
    }

    // writing of file chunk into created file folder
    fs.writeFile(path.join(directoryPath, `./${chunkId}`), fileChunk, (e, bytes) => {
      if (e)
        res.status(500).send({ status: 500, message: e.message });
      else
        res.status(200).send({ status: 200, message: 'Success' });
      });
  });
});

app.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
}); Looking into the proposed approach, we can notice that encryption of the chunk is happening for each chunk which can affect UI in case of performance. As said in the previous encryption point, encryption in JS is a manipulation of TypedArrays. In programming, we can only manipulate arrays data using a loop. Looping has a negative effect on the event loop (creepy-sounding) which finally may reduce I/O responsiveness. synchronously, So in some cases, when the file is too huge, encryption of its chunks may stop the web page from responding for some time. To avoid such behavior, we need to transmit encryption tasks to another place. We can try to make it asynchronous, but the best way will be to give this job to another thread. Parallelism Browser JavaScript has no single, bespoke implementation like most other programming languages do. There are several engines that are used in different web browsers, like v8 in Chrome and SpiderMonkey in Firefox. But most of them have Web Workers, which are mainly used for archiving parallelism. There is more than one type of Web Workers (e.g., SharedWorker, ServiceWorker). The simplest one is the Dedicated Worker, which will be used in the current article. The main difference between Dedicated Worker and others is that dedicated worker is only accessible by the script that has called it. Firstly, let’s add headers to our express static response on the server, so the client can access file: worker.js app.use(express.static(
  'public', {
    setHeaders: function(res) {
      // heade necessary for accessing worker file from the client
      res.set('Cross-Origin-Embedder-Policy', 'require-corp');
    }
  }
)); On the client, we need to create a worker instance and change the logic of to use workers for encryption instead of the main process. sendChunk // instanciating worker
const worker = new Worker("worker.js");

async function sendChunk(file, fileName) {
  if (!chunksQueue.length)
    return;

  const chunkId = chunksQueue.shift();
  const begin = chunkId * CHUNK_SIZE;
  const chunk = file.slice(begin, begin + CHUNK_SIZE);

  worker.onmessage = (msg) => {
    const { encryptedChunk: fileChunk } = msg.data;

    upload({
      fileChunk, 
      fileName, 
      chunkId,
     }).then(() => {
        sendChunk(file, fileName);
      }).catch(() => {
        chunksQueue.unshift(chunkId);
      });
   }

   worker.postMessage({ chunk });
} And finally, dedicated worker code: // worker.js
console.log('Running worker...');

if( 'function' === typeof importScripts) {
  importScripts('nacl.min.js');
  importScripts('nacl-util.min.js');
  importScripts('encrypt.js');

  const secretKey = nacl.randomBytes(32);

  self.onmessage = (msg) => {
    if (msg.data.chunk) {
      const { chunk } = msg.data;
      const encryptedChunk = encrypt(chunk, secretKey)

      postMessage({ encryptedChunk });
    }
  }
} We had a very long journey, so let’s try to analyze what we had at that moment and try to find some clips. Here is this example where we moved heavy synchronous calculations inside the worker so it would not block our main process. But, if we look through the current code, we can find out that we are using message-passing API to pass chunks of the file to the worker. This can slow down the performance of the uploading process because each time, file chunks are cloned to the worker. What if we could use something like shared memory, to which our worker will have access same as the main process? Shared Memory class allows you to share memory between two threads without depending on the message passing. It is also the same object as ArrayBuffer mentioned in the Encryption point, which has only one difference - it is shared between all workers. Data gets sent to and from the worker by using the method, and certain types are so-called transferable objects that are transferred from one context to another with a zero-copy operation, resulting in high performance. SharedArrayBuffer postMessage() To better understand how we can use this, let’s edit our previous example. As you can see, even with the usage of shared memory, we still are transferring data between the worker and the main process . This can be solved by using another shared memory for an encrypted object or just moving a request to API inside the worker. (illustrated on 3-rd image, red arrow) In this article, we will use 2nd approach. So our workflow will be transformed into: But when you try to use SharedArrayBuffer, you will find that you need to copy the file Uint8Array into SharedArrayBuffer, as SharedArrayBuffer is just a point to the memory, and it just accepts byteLength when it is initialized. You need to create its view and store data in it using a copy of the existing one. So copying of file Uint8Array would slow down the process of uploading. Suddenly the final solution appeared. Why not just pass the file object from the input? After refreshing a bit about what is File instance in JS, I found that the is the interface that provides information about files and allows JavaScript in a web page to access their content. So this is just a pointer to the file. So it has no file content but only refers to it. So I decided to just pass it as a message from the main process and inject file content into the worker. File Final solution According to image 5, we need to move all the logic from the main process to the worker. Only the input handler will stay: // main.js
document.addEventListener("DOMContentLoaded", () => {
  const worker = new Worker('/worker.js');

  const fileInput = document.getElementById('file_handler');

  fileInput.addEventListener('change', function () {
    const file = fileInput.files[0];

    worker.postMessage({ file });
  });
}); And worker.js: console.log('Running worker...');

if( 'function' === typeof importScripts) {
  importScripts('nacl.min.js');
  importScripts('nacl-util.min.js');
  importScripts('encrypt.js');
  importScripts('upload.js');
  importScripts('sendChunk.js');
  importScripts('readFile.js');

  let chunksQueue = [];
  const CHUNK_SIZE = 1000000;

  function getChunksAmount(fileSize) { return Math.ceil(fileSize / CHUNK_SIZE) }

  const getChunksQueue = (chunksQuantity) => {
    return new Array(chunksQuantity).fill().map((_, index) => index);
  };

  self.onmessage = (msg) => {
    if (msg.data.file) {
      const { file } = msg.data;

      readFile(file).then((res) => {
        const chunksAmount = getChunksAmount(res.length);
        chunksQueue = getChunksQueue(chunksAmount);
          sendChunk(res, file.name, chunksQueue).catch((e) => console.log(e));
        });
      }
   }
} Full source code you can find on GitHub: https://github.com/arty-ms/ParallelUpload Conclusion We can see that usage of parallelism can bring lots of effort for JS developers and can make your UI more responsible and effective, as all computation logic may be executed on the thread. When I was starting this article, according to my plan, I thought that the usage of SharedArrayBuffer would be the final solution. But in the process of writing code, I realized that it was not true. But It's worse to mention that SharedArrayBuffer is one of the core features of multithreading JS and everyone who uses workers API should be familiar with it. However, it is not useful for every case. The next article will cover the usage of wasm code in JS, one of the most underrated client tools. Sources used Multithread JavaScript. Concurrency Beyond the event loop. Thomas Hunter & Bryan English 2nd Edition. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/API/File

JavaScript: Enable Parallelism for Private File Uploading

Founders of open source analytics software - and - joined Ben Rometsch, CEO of ( ), on Podcast to discuss the company from its inception as part of an opens source project to its present state and what lies ahead. Piwik Maciej Piotr Flagsmith open-source feature flagging product The Craft of Open Source The Genesis of Piwik Piwik began as an open source statistics system back in 2007. Maciej began contributing to the project and using it for the advertising business he owned at the time. Over time, he became a core team member and was the pioneer of the idea to provide professional services and support as a component alongside the open source project, which by this point had changed name to Matomo. The team created a company called Piwik PRO and they made revenue offering customised services. By 2014, they realised that the platform was nearly ten years old and relied on legacy systems. So, they decided to rebuild it and create an enterprise version. Unfortunately, the vision of one of the founders was not aligned as he wanted to keep things entirely open-source and free. Therefore, in 2016 the company split. With Maciej as the CEO, the team bought out the founder that disagreed with the future vision of the company. This type of split is not unusual in open-source projects, with similar events taking place with MySQL and Drupal, for instance. The team began to build a product known as Piwik PRO Analytics Suite, alongside supporting clients who were still using the open-source Piwik. Developing Piwik PRO Analytics The commercial analytics suite and the open-source project were developed by two separate teams from 2016 onwards. The first team worked on the core platform and open-source product and was made up of the original contributors that had coded the original commits for GitHub. The second team of developers worked on the commercial Piwik PRO platform, concentrating on creating a cloud version and expanding its capabilities. This team worked entirely within the private repository on the cloud platform. Taking this approach of two separate teams meant that from the perspective of outside users, developers, and the open-source community not much changed when the commercial business was created. In fact, the changes were positive for the open-source community as now there was more investment into the platform and paid developers working on it. The decision was made to start from scratch rather than adapting and updating the existing Piwik system. One reason for this was that the developers wanted to refactor the whole thing on a new directory to keep customers with a high-volume of traffic happy. Maciej and Piotr also saw traction on the business side of things and the potential for growth. They also secured funding to create a proprietary product and offer a migration service from one platform to another. From a technical point of view, they didn’t want to build a modern microservices-based platform chunk-by-chunk on old technology and legacy systems. They wanted to use newer backend technology with a fresh database using ClickHouse which is specially designed for analytics services and platforms. ClickHouse is ideal for platforms that work with lots of data, aggregation, and filtering. The team then built a new UI on React JS, which allowed them to create a common components library, replacing and adding functions and features to the platform. The primary services are written in Python and the trackers use Rust for its speed. What Is A “Tracker” In Analytics Software? The tracker is the part of the platform that receives all data requests and carries out some transformations. It’s the endpoint that receives behavioral data captured by Piwik PRO when it is tracking a website or mobile app. Therefore, the tracker is the endpoint with the highest volume of traffic. To decide which coding language to use for the tracker, the company held an internal competition. Developers submitted proof of concepts which were then benchmarked against the open-source version. Rust won the first prize, outperforming others in terms of speed, which is why it is used for the tracker. On-premise Deployment Piwik PRO Analytics, despite being a cloud-based platform, is deployed on-premise. This seems like a slightly unusual decision at first glance but has its merits. First of all, although it is technically an on-premise deployment, the installation isn’t carried out by our team visiting the customer’s physical premises. Instead, it is done remotely. We verify that the customer’s infrastructure is capable of running our scripts. Then, a time and place for the installation is agreed upon and a nominated team member deploys the platform during that dedicated time slot. The main reason for the on-premise approach is that Piwik’s customers tend to be enterprises that need high-availability of analytics. The complexity of deploying a large-scale analytics platform means it cannot simply run on a virtual machine, for instance. It needs to run on physical servers to achieve high performance. Keeping Up With Google Analytics There’s no hiding the fact that all analytics platforms are in a race to keep up with Google Analytics and trying to offer extra value. Piwik PRO has done a good job of keeping up with the functionality offered by the paid version of Google Analytics. Not everyone knows that Google Analytics charges when you cross a certain threshold and high-traffic users end up paying a lot of money annually. Google’s new Analytics version for 2021 is designed to compete with Segment or Amplitude. This means that the platform has become much more complex. Piwik PRO, by contrast, seeks to offer a more user-friendly analytics platform that has some of its own unique features and flavour, rather than competing directly with Google which is hard to do. Also, Piwik PRO has kept up-to-date with changes in privacy laws. They quickly adapted the platform to comply with GDPR and respect individual’s data privacy. This gives them a competitive edge in today’s world of increasing data privacy awareness. The Future of Piwik? Both Maciej and Piotr think that focusing on data privacy and customisation is probably a wise move in today’s global climate. They discussed the idea that Piwik may be customised to collect the least amount of data, which means that site visitors won’t be irritated by a consent window. There will be different solutions available depending on the business use case, which creates a situation where companies can pick their own version of the platform and still remain compliant. When pressed on whether they have any open-source components in their business at the moment, they replied by saying no, but there’s a good chance there will be in the future, hinting that there may be some announcements regarding this subject soon. Another area of development for Piwik is integrations with other software and frameworks. This applies to both the open-source side of things and the commercial side of the business. This article just scratches the surface of what was covered on the podcast. To listen to the whole thing check out the . podcast here today

How Piwik built a Google Analytics alternative out of an open-source project

A few companies I've worked for have an IT policy on their secure computers designed to stop movement of sensitive data outside the enterprise. This policy encrypts all file data being written to removable media (USB drives, external hard drives, etc.) such that a computer within the same enterprise can decrypt and read the data. only I recently noticed ... the name of the file is not encrypted If the filename can be written to the removable disk in the clear, then of course we might see cleartext information like . This is a leak of a just few characters of cleartext, but could we leak more than just a few characters? What about data more complex than text? secret_file_for_customer.zip Enter : a tool that converts arbitrary data into metadata! metamorpher Metamorpher is very straightforward; it takes an input file and turns it into a directory structure of empty files whose contain a Base64 encoded representation of the input file. is just a way of encoding binary data into text which is important here because only a certain set of characters are allowed in filenames. filenames Base64 Example First, let's generate some data to work with: dd =/dev/urandom of=./random_data bs=1M count=100 if Encode the data with the flag: -e metamorpher -e ./random_data This creates a new subdirectory ( ) that contains the encoded data. There are thousands of empty files in there, each with a chunk of the file data. Here's a snippet of just the last three: ./random_data_metamorphed $ find ./random_data_metamorphed -mindepth 2 | tail -n 3
./random_data_metamorphed/8913d.HWdVeDtcf5Q8RDSy9iAK7BaOSIMi2dG5W9IZfhkLUX_qlwSjIWe1tJ_uFfpI-R7LRsGqurrviiPruE-sxP-s3gFQGJHbhuV5z1L2iTZOXsdOiVk_Q_lzkO0jx7boZ5Dm1NZmJtUAT8F7YVyherNkru7d2CRaxULF7Uo9UnbMvZfR16MGV4SzKX-SScIIRUARL6q8tubr2N-BIwRp6nfr57xajTkPlIv4to-x9CXCF765Zs3OyGY_eMijH
./random_data_metamorphed/89145.cY1wcToYWMW8oYj_C9ttE22iGbt6z3FDcgVLvfOMFgFG2rBtsN0rpJb5ZEO3sb_72jWbeWJ8CuFBomoiBrSu2tB4_ITa9rWZhcIB62sERQQwARqdFE2PbwMh0Uuuf1-y2wkazgJdYE0o5jO4NkxS7jj1C-b5--OY538ibwZizbG3IS-c0-1nFyhNRMsU1mgcuWp8BpavmYZoDJ_KmZDOJjUV0ZAPxrtuuGmfRdg5ZIzx5mmB-mztIoqCM
./random_data_metamorphed/655fe9206ffff5d7fbc2bbe426f0b2330b4eb567.sha1 The first two filenames above contain two things separated by a : . A hex number for ordering A chunk of the Base64 representation of the input file The third filename above holds a SHA1 checksum of the input file ( ). This helps guarantee data integrity when decoding. <sha1sum>.sha1 For this example, on an ext4 filesystem, the encoded directory takes up about 209 MB on disk. Notably, every file in the directory is zero-length. All of the disk space utilized is from filesystem overhead. This has an added advantage of obscuring how much data you're actually moving to the disk. Of course, encoding binary data in Base64 reduces the data density which accounts for the rough doubling in data size. Ext4 allows a maximum filename length of 255 characters, and the hexadecimal ordering characters at the beginning of each filename (plus a delimiter) subtract 6 characters from that number. This leaves 249 characters per filename for the Base64 encoded data. 100 MB of data encoded as a Base64 string is 139,810,136 characters long. 139,810,136 characters / 249 characters per filename = 561,487 files (rounded up to nearest integer). Sure enough, we see 561,487 files (plus the one checksum file): $ find ./random_data_metamorphed - f | wc -l
561488 type On my laptop, the encoding process for this example took about 18 seconds. We can copy this directory to a removable disk and let the encryption policy do its thing! It will "encrypt" the thousands of empty files, but if it doesn't touch the filenames, the data is stored in the clear. Once we copy the contents to an unmanaged PC, the flag can decode the directory structure back to a file: -d metamorpher -d ./random_data_metamorphed -o ./copy_of_random_data The decoding of this example took about 3 seconds on my laptop. Metamorpher will automatically check the output data against the checksum. That's it!  We've copied 100 MB of data with perfect integrity despite an encryption policy attempting to stop us. Takeaway As with all data security techniques, there's a balance between user convenience and security. It's obvious that sensitive character strings (like ) could be leaked through a filename, but leaking a few dozen characters per filename isn't that big a deal. However, this method shows that an insider could quickly encode a large archive of sensitive data into filenames on disk and send the contents to a competing company or rival nation. secret_file_for_customer.zip I've reached out to the two software vendors I know of who implement this type of encryption policy. The first vendor noted their tool supports a mode where files written to the disk are dropped in a container directory and archived prior to encryption which would protect against this method. They also mentioned they may review how the security implications of allowing cleartext filenames are presented in their documentation. The other vendor simply stated that they do not claim to prevent intentional data leakage from authorized users as it's an unachievable goal. While that may be true, I'd imagine their enterprise customers might appreciate an effort to make data leaks more difficult for insiders. In doing research for this post, I came across a couple of projects that do similar things and may be of interest to readers who've made it this far: includes an method for data exfiltration. PyExfil FTP mkdir abuses Google Docs by storing Base64-encoded files as text documents to get free unlimited storage space. UDS

Bypassing Enterprise Data Encryption Policy with Metadata [A How-To Guide]

We’re fresh on the heels of the Google and Reddit version of “Let’s Make a Deal.” To the tune of $60 million per year, tech behemoth Google has been granted real-time access to its catalog of user-generated content for the purpose of training Google’s . AI models As Reddit moves toward going public, the $60 million presents itself as a viable new revenue stream, perhaps setting a precedent for other tech companies to follow in their footsteps by turning user data into a new revenue stream in a legal manner. Hey, if Reddit can do it, you can do it too! Let’s first take a gander at how Reddit and Google are pulling this off, then explore an option that can make a business cash while also fostering loyalty from their users: As it pertains to Reddit users, the legality comes from the Reddit Terms of Service agreement. When creating or submitting content to Reddit, users agree to grant Reddit a “worldwide, royalty-free, perpetual, irrevocable, non-exclusive, transferable, and sublicensable license to use, copy, modify, adapt, prepare derivative works of, distribute, store, perform, and display [the user’s] Content and any name, username, voice or likeness provided in connection with [the user’s] content in all media formats and channels now known or later developed anywhere in the world…and you irrevocably waive any claims and assertions of moral rights or attribution with respect to [the user’s] content.” This is, of course, legalese for “We, Reddit, can pretty much do whatever we want with your data.” So, this is a legal monetization of data via licensing, however is it an ethical means of monetization? That’s for you to decide. Reddit is a big enough website with few viable competitors that it can, for the most part, monetize user data without any form of compensation given to users and not have to worry about losing a meaningful chunk of its user base. One Reddit user, u/Morrissey_99, asked in the r/technology thread on the deal, “[So] do I get a part of that $60M Reddit? It’s my data you were using, after all”. The answer, of course, is no Morrissey_99 you will not. You agreed to royalty-free, perpetual, etc., etc., licensing of your data prior to using the Reddit product. \User data is a hot commodity. Whether it’s AI training, quantitative trading, marketing measurement, or improving internal products and systems, it’s clear that data has tremendous value. Reddit’s data is worth $60 million annually to Google. That’s a lot. If you’re reading this, you’re probably not Reddit- or Google-sized, but you probably are sitting on a lot of data that can be productized and, in turn, monetized. But how do you do that in a way that works for everyone? How do you do it in a way that is ?  It’s pretty basic: be transparent and give users something they actually want in return. very cool 😎 Welcome to the revolutionary world of generally-speaking-do-things-in-the-opposite-way-of-Reddit-and-Google method monetization, where legalese takes a long walk off a short pier,  the Morrissey_99s of the world get compensated for their clearly valuable asset, and businesses open up a brand recurring new revenue stream from an asset they already have or can acquire with (relative) ease. I realize monetizing data is technically not doing the opposite of Reddit, but it is perfectly alright to borrow goals (turn data into revenue) from Reddit, but probably not advisable to borrow their systems (make it so users can’t use your product without forking over their data). Anyway… Explicit Consent is Key, Rewards Entice Whether it is the content generated on a website, app usage details, transaction history, or demographic information, the impetus is on the user to decide how that data can be used. Data privacy regulations like GDPR and CCPA make it abundantly clear users need to know what the heck is going on with their data and give them the options to decide whether the data can be used at all. The Reddit solution, essentially creating a gigantic blanket statement that allows them to do essentially whatever they want with the data while giving the user nothing in return other than the ability to use their product, is one way of going about things. I don’t know if there’s any actual term for this type of “consent,” but for the sake of the blog, let’s call it “consent by acquiescence.” The vibe is, “This seems kinda messed up, but whatever, dude, if that’s what it takes.” Leaves a different taste in the mouth than the proposed alternative: “Hey, we’d like to use data a, b, & c for use cases x, y, & z. We’d like to use this data indefinitely, but each month, we can pay you $10, or we can upgrade your subscription to the paid version at no other cost. Oh, and you can opt-out whenever you want.” This is an informal encapsulation of the mytiki (here’s my biz plug) ethos: data is valuable. Businesses should be able to use data, but only if they get consent from the user. That is covered by most laws. But the kicker is, hey, instead of alienating your user base by monetizing an asset they created without giving them something in return, why not, say, for legally licensing their data to you? This turns the data into zero-party data--data willingly given to a business with the expectation of something of value in return. give them something in return We built technology to make this very thing possible. You can check it out at if you’re so inclined, but let’s focus our attention more on the logic behind why this model is the most intuitive. mytiki.com It’s Legal, but It’s About True Ownership, Not Fine Print The concept of zero-party data licensing is straightforward but game-changing in its execution. Unlike the all-too-common, lengthy terms of service agreements that users scroll through and accept without a second thought, zero-party data licensing is about empowering the user. It recognizes that the data created by individuals is their property — and that any use of this data by businesses must be both legal and fair. What distinguishes data licensing from the “we-own-your-content” clauses hidden in the small print is the principle of ownership. Here, the user is not a passive participant but an active owner of their data. This means the data is not just observed or collected without explicit consent (as with first-party data), but it is owned and controlled by the individual. The mechanics of a zero-party data license also significantly bolster its legitimacy. Participation is entirely optional; users retain full access to the app or website regardless of their choice to license their data. They are not sneakily enrolled but must make an active decision to participate. Moreover, the compensation is not vague or promised in the form of intangible rewards; it is explicit and tangible. This could be in the form of a cash payout, discounts, loyalty points, or other concrete rewards. Zero-party data licensing agreements are immutable and digitally signed by both the user and the business, ensuring that the terms are enforced and protected. The user's ownership is not only recognized but respected and rewarded, making the process of data monetization transparent and equitable. When users are dealt with in the data economy, businesses can foster trust and loyalty. Why hide behind complex legal jargon when you can forge a partnership with users based on respect for their data rights and clear, mutual benefits? There are all sorts of legal questions that arise when it comes to monetizing data. Many are solved with user consent, and user consent is much more likely when they’re getting something of real value in return. If you’re sitting on data you can’t sell or are looking to source monetizable data in a way that genuinely interests users, zero-party data licensing is the best way to do it. In summary: It’s legal. But not in a Reddit-hired-some-lawyers-to-write-something-to-ensure-they-can-use-everyone’s-data-for-whatever-they-want type of way. It deals with users to monetization. It’s data, after all. their It allows businesses to create trust and loyalty by rewarding users and sticking to the actual terms of the agreement. It’s specific. It allows data to be utilized to its full potential with value-adds for everyone involved. Conclusion The Reddit-Google $60 million handshake shines a light on data monetization as a viable means of opening new, recurring revenue streams. The user data market is growing quickly. The rise of privacy initiatives coupled with growing user awareness of the value of their data brings about an obvious fork in the road when it comes to the way we as businesses use and monetize data: we can make slight modifications to the ways of old by baking a data ultimatum into terms of service agreements, or we can move toward strategies that directly involve and compensate the users generating the data. User data will continue to generate interest from buyers in many fields, including the obvious cash cow of training AI. Product-led businesses are missing out on opportunities to add to their revenue by neglecting the potential for data itself to become a viable product. Zero-party data licensing is a novel but logical strategy for turning user data into revenue while building a loyal user base. Take the story of Morrissey_99s and folks like them; it really hits home that users want to be recognized for their contributions. This isn't just about finding some middle ground; it's about coming up with totally new ways that respect what users bring to the table while also making business sense.

How to Monetize User Data Like Reddit (Without Being Sketchy) 

Power users drive some of the most successful companies — people who love their product, are highly engaged, and contribute a ton of value to the network. In ecommerce marketplaces it’s power sellers, in ridesharing platforms it’s power riders, and in social networks it’s influencers. The importance of power users All companies want more power users, but you need to measure them before you can find (and retain) them. While DAU/MAU — dividing daily active users (DAUs) by monthly active users (MAUs or monthly actives) — is a common metric for measuring engagement, . it has its shortcomings Since companies need a richer and more nuanced way to understand user engagement, we’re going to introduce what we’ll call the “Power User Curve” — also commonly called the activity histogram or the “L30” (coined by the Facebook growth team). It’s a histogram of users’ engagement by the total number of days they were active in a month, from 1 day out of the month to all 30 (or 28, or 31) days. While typically reflecting top-level activity like app opens or logins, it can be customized for whatever action you decide is important to measure for your product. The Power User Curve has a number of advantages over DAU/MAU: It shows if you have a hardcore, engaged segment that’s coming back every day. It shows the variability among your users: some are slightly engaged, whereas others are power users. Contrast this with DAU/MAU: it’s a single number and so blurs this variance. When mapped to cohorts, Power User Curves let you see if your engagement is getting better over time — which in turn helps assess product launches and performance of oher feature changes. Power User Curves can be shown for different user actions, not just app opens. This matters if the core activity that matters for your product is deeper in the funnel. In other words, while the DAU/MAU gives you a single number, the Power User Curve gives entrepreneurs several avenues of analysis to assess their product’s engagement to the most addicted users — in a single snapshot, over time, and also in relation to monetization. This is useful. So how does it work? The Power User Curve will “smile” when things are good The shape of the Power User Curve can be left-leaning or smile-like, all of which means different things. Here’s a smile: The Power User Curve above is for a social product, and shows the characteristic smile shape that indicates there’s a group of highly engaged users using the app daily or nearly daily. Social products with frequent user engagement like this lend themselves well to monetization via ads—there’s enough users returning frequently that the impressions can support an ad business. Remember that Facebook would have a very right-leaning smile, with 60%+ of its MAUs coming back daily. What matters is that, over time, the platform is able to retain and grow its power users: successive Power User Curves should ideally show users shifting over more to the right side of the smile. As the density of the network grow, and with stronger network effects, it’s expected that there’s more reason for users to return on a daily basis. Let’s look a different example, which doesn’t smile: The Power User Curve can show when strong monetization is needed This Power User Curve of a professional networking product looks quite different than that of a social product. It’s left-weighted with a mode of just 1 day of activity per month, and decays rapidly after those few days. There’s no power users. But this light engagement can be okay — not every company needs to have a smile-shaped Power User Curve, just as not every product category necessarily lends itself to an ultra-high DAU/MAU. When there’s low engagement, what matters is that the company has a way to extract enough value from users when they areengaged. Think about an investing product like Wealthfront or networks like LinkedIn — few users are likely to actively check it on a daily basis, but that’s ok, since they have business models that aren’t tied to daily usage. CEOs of such companies should therefore,think about: Is there a way to create revenue streams where the business can still monetize effectively despite users’ infrequent engagement? Or, who are the users using this product more frequently, and how can I get more of them? Is there something about the product — e.g. onboarding, the core experience, etc. — where a significant chunk of the user base isn’t experiencing the ‘aha moment’ that makes them “get” the product, and therefore not getting value from it right now (and if so how to get there)? Some products should be analyzed in a 7 day timeframe – like SaaS/productivity – and others on 30 days Another flavor of the Power User Curve is a histogram of users’ engagement for a 7-day period, also commonly called L7. The 7 day Power User Curve shows weekly actives, not monthly actives. Plotting this version can make sense if your product naturally follows a weekly cycle, for instance, if it’s a productivity/work-related product that users engage with Monday through Friday. B2B SaaS products will often find it useful to show this version, as they want to drive usage during the work week. Note that using DAU/MAU wouldn’t be the appropriate metric for this product as it’s not designed to be a daily use product. You can also see there’s actually a smile curve through 5 days, but fewer users are using it 6-7 days, which makes sense for the power users of a workweek product like this. CEOs of such product companies should therefore want to understand: Who are the users engaging just 1 or 2 days each week? Are there certain teams or functions within an organization that are getting more value, and how can I build out features to capture the teams with less engagement? Or, if the product is really driving a lot of value for specific departments — how can I understand their needs better and make sure we continue building in a direction that supports their daily workflow (and that we can upsell new features)? Plotting the Power User Curve for different WAU or MAU cohorts can also be very insightful. Over time, you can see if more of your user base are becoming power users, by seeing the shift towards higher-frequency engagement. The trend of over time can show if the product is getting more engaging over time Here’s an example: The Power User Curve for MAU cohorts from August through November shows a positive shift in user engagement, where a larger segment of the population is becoming active on a daily basis, and there’s more of a smile curve. You can see when the line starts to inflect in order to see when a critical product release or marketing effort might have started to bend the curve.  This might be a place to double down, to increase engagement. For a network effects product, you might expect to see newer cohorts gradually improve as you achieve network density/liquidity. On an ongoing basis, you can measure the success of product changes or new releases by looking at different cohorts’ Power User Curves. If a product unblocks a bunch of features for power users, you might see a gradual increase in power users. The frequency histogram can be keyed on actions beyond the visit — did someone show up or not — you can also go with deeper user actions. For instance, you may want to plot the core activity that maps closely to how your business is monetized… or that better represents whether users are getting value from your product. This is important because it forces you to think about what really matters to measure. The Power User Curve can be based on core activity, not just app opens or logins The above chart for a content publishing platform shows the total number of days in the month users posted content. A lot of products have smile-shaped core activity Power User Curves, because while most people tend to contribute lightly, there is a small contingent of users who are power users. Think of the distribution of Youtube creators, or Ebay sellers, or even how often you post on Facebook. As the CEO or product owner of a platform like this, it’s important to design the platform such that the everyone has a chance to succeed. On Facebook, the news feed algorithm makes sure that if you feel strong affinity to a person or organization, you’ll still see their posts even if the sheer volume of other content (for instance, from more prolific media companies) would otherwise drown it out. On OfferUp, even if I seldom sell items, when I do list something, their algorithm makes sure that it’s surfaced to the relevant potential buyers. Not everything is a daily use product, and that’s okay. Why does this all matter? Power user analysis allows you to get a better understanding of how users are engaging with your product, and make more informed decisions using that data. That might mean choosing an appropriate business model that works for your pattern of engagement, or designing better re-engagement loops for lower-engaged user segments, or doubling down on use cases that your high-engagement user base is already getting value out of. The beauty of the Power User Curve over DAU/MAU is that it shows heterogeneity among your user base, reflecting the nuances of different user segments (and therefore what drives each of those segments). Creating versions of Power User Curve by various user segments can also be particularly insightful. For instance, for a business with local network effects (like Uber or Thumbtack), showing Power User Curves by market can reveal which geographies are developing density and strong network effects. Power User Curves show if your product is hitting a nerve among a super engaged core group of users, even if perhaps the overall blended DAU/MAU is low. It also doesn’t have to just reflect app opens or logins — you can hone in on an action that maps closely to users getting specific value out of your specific product and plot the Power User Curve for that action. The key for founders is to know that there isn’t a single silver bullet to measure perfect engagement — rather, the goal is to find the set of metrics that are appropriate for their businesses. Comparing the Power User Curve of a social app vs. a work collaboration app doesn’t make sense, but looking at your own Power User Curve over time, or finding benchmarks for your product category, can tell you what’s working… and what’s not. Originally published as “ " The Power User Curve: The best way to understand your most engaged users

What to Learn from the Power User Curve

While running a self managed elasticsearch cluster like any other database, it's important to make provisions for data backups. Data backups on Elasticsearch can't be done by simply copying elasticsearch data files from one disk to another, this tutorial guides you through making the best use of the Elasticsearch snapshot module for creating cluster snapshots and leverages the Azure blob storage for securely storing your backed up data. Also besides backing up data, the snapshot api also comes in handy for migrating data from one cluster to another. As mentioned earlier, this tutorial uses the as a backup store, other storage services such as Amazons s3 can be used as well, but to follow this tutorial comprehensively you'd need to have an azure subscription, you can sign up for a free trial . You would also need to have access to the Elasticsearch cluster node terminal. Azure blob storage here Moving on... STEP 1 Setting Up An Azure Blob Storage Account follow the steps below to create an azure blob storage account, if you don't have one already for this purpose. 1. On the Azure portal, click on the storage accounts link on the sidebar, or you can use the search resource option to search for "storage account" if this link isn't present on your side bar 2. Click on add new at top left corner of the storage account panel 3. Fill in the required information on the first panel, your can accept the defaults for the next sections, and then create your storage account. 4. Access your just created storage account and create a new container 5. on your storage account page, on the side bar click on "access keys", copy the account name and they key1. That's it for the storage account setup, we then proceed to the next step STEP 2 Installing The Azure Repository Plugin For Elasticsearch To start taking snapshots we need to first register a snapshot repository within our elasticsearch cluster, this repository defines where Elasticsearch should store snapshots taken, learn more about it . Remember these repositories could be an HDFS or a cloud storage service and in this case we are using  the Azure blob storage service. here To register the repository for azure, ssh into your ES cluster node and enter the following commands sudo bin/elasticsearch-plugin install repository-azure If this doesn't work, then try the command below. The bin folder may differ depending on how your elasticsearch was setup sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install repository-azure Afterwards restart your elasticsearch cluster sudo systemctl restart elasticsearch Next, hit the following endpoint to confirm the plugin is installed, , you should get the following output, an object containing details of installed plugins on that cluster. http://eshost:port/_nodes?filter_path=nodes.*.plugins after confirming the plugin is installed, the next step is to setup the plugin to work with our storage account, we'd be configuring the plugin using the storage account name and key bin/elasticsearch-keystore add azure.client.default.account enter your azure storage account name here, next bin/elasticsearch-keystore add azure.client.default.key enter your account key here, if you have issues running these command you should confirm the location of your elasticsearch bin folder, you could try this path , it depends on your installation configuration. After setting the values in the keystore, restart your elasticsearch cluster. /usr/share/elasticsearch/bin/elasticsearch-keystore sudo systemctl restart elasticsearch Next, we set up a snapshot repository, you can do this by sending a post request to the following endpoint and pass the these json payload http://eshost:port/_snapshot/name-of-your-repo { : , : { : , : , : , : }
} "type" "azure" "settings" "container" "backup-container" "base_path" "backups" "chunk_size" "32MB" "compress" true you can leave the type as is, in the settings, the container name should be the name of the container created in your storage account, the defines a folder where snapshot data should be stored, this is useful if you are taking snapshot of different indices or even data from different clusters and storing them in one container. base_path The defines how small big files can be broken down to prior to being transferred . you can get more details about the settings by following this . Below is a screenshot of my settings chunk_size link You should get an "acknowledged: true" as a  response. You can view all your registered repositories by making a get request to the following endpoint http://eshost:port/_snapshot/ Step 3 Taking Actual Snapshots Now let's move on to taking actual snapshots, for this I've created a sample index "sample_records", we could back this up or better still let's back up all indices in the cluster along with the clusters settings.  To do this, make a post request to the following endpoint http://eshost:port/_snapshot/azureblob_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E with the following payload { : , : , : , : } "indices" "index_1,index_2" "ignore_unavailable" true "include_global_state" true "partial" true By default, when option is not set and an index is missing the snapshot request will fail. By setting to false it’s possible to prevent the cluster global state to be stored as part of the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don’t have all primary shards available. ignore_unavailable include_global_state This behaviour can be changed by setting to true. This is from elasticsearch's official docs . Below is a snapshot of my backup request, partial here Here, notice I didn't didn't add the indices parameter, when the indices parameter isn't included, all indices present in the cluster is going to be included in the snapshot. Take note of the snapshot name / this is the url encoded version of this this is translated to the current date the snapshot was taken, i.e also note that it isn't a prerequisite to name your backups this way, you can give it any name you want to, this just makes sense in case you are doing daily or weekly backups via cron for example, to be able to reference snapshots easily. %3Csnapshot-%7Bnow%2Fd%7D%3E, <snapshot-{now/d}>, /snapshot-2020.04.09, Next, we are going to monitor the status of an ongoing snapshot,  to do this, send a get request to the following endpoint, , http://eshost:port/_snapshot/azureblob_backup/<snapshot-name> note that using the url encoded format won't work if you used it to save your snapshot, instead use the literal string, e.g now http://eshost:port/_snapshot/azureblob_backup/snapshot-2020.04.09, you should get the following response. { : [
        { : , : , : , : , : [ ], : , : , : , : , : , : , : , : [], : { : , : , : }
        }
    ]
} "snapshots" "snapshot" "snapshot-2020.04.09" "uuid" "OjLZEfXDS-mKVqsSi7VteQ" "version_id" 6080399 "version" "6.8.3" "indices" "sample_records" "include_global_state" true "state" "SUCCESS" "start_time" "2020-04-08T09:53:47.926Z" "start_time_in_millis" 1586339627926 "end_time" "2020-04-08T10:03:15.361Z" "end_time_in_millis" 1586340195361 "duration_in_millis" 567435 "failures" "shards" "total" 15 "failed" 0 "successful" 15 Take note of the " array, this shows the names of backed up indices, the state shows the current status of the snapshot it can be either , IN_PROGRESS, FAILED, SUCCESS or PARTIAL, if the snapshot is in a PARTIAL state, it means some indices could not be backed up, the names of these indices are saved in the array. indices" "failures" Now, lets check our azure storage to see if the snapshot was saved in our container. There we go! a snapshot of our entire cluster! Restoring A Snapshot This assume you a new empty cluster you wish to copy your snapshot data to, first on your new cluster you have to configure the Azure repository plugin by following the same steps above, use the same Storage account as that which was used to take the snapshot, keys and all. Ensure also that the matches that which was used in creating the snapshot. Next, send a post request to the following endpoint to restore your snapshot to the new cluster, base_path http://eshost:port/_snapshot/<repo-name>/<snapshot-name>/_restore Note that there compatibility requirements for backups, below are the compatibility ranges If you plan to export data from one ES cluster to another, you need to be aware that not all versions may be compatible with your exported data. A snapshot of an index created in 6.x can be restored to 7.x. A snapshot of an index created in 5.x can be restored to 6.x. A snapshot of an index created in 2.x can be restored to 5.x. A snapshot of an index created in 1.x can be restored to 2.x. Conversely, snapshots of indices created in 1.x be restored to 5.x or 6.x, snapshots of indices created in 2.x be restored to 6.x or 7.x, and snapshots of indices created in 5.x be restored to 7.x or 8.x. This is from the official elasticsearch docs. cannot cannot cannot Bonus! Automating Things The most popular use case of the Elasticsearch snapshot module is for making backups of your cluster and more often than not backups are automated. So I've written a simple node script that helps take a snapshot of your cluster and sends a mail notification on the backup status, this script can be triggered by a cron job set to run daily or however frequently you'd like. axios = ( ) nodemailer = ( ); transporter = nodemailer.createTransport({ : , : { : , : }
}); SNAPSHOT_URL = CLUSTER_NAME = ; dateObj = (); month = dateObj.getUTCMonth() + ; day = dateObj.getUTCDate(); year = dateObj.getUTCFullYear(); hour = dateObj.getUTCHours(); minute = dateObj.getUTCMinutes(); seconds = dateObj.getUTCSeconds(); backuptime = ;



axios.post( ,{ : , : }).then( { .log(response.data.accepted) (response.data.accepted === ){ .log( )
                checker();
            } { .log( )
                notify( )
            }

        },(error)=>{ .log( , error)
            notify( )

        }); checker = { intervalId =  setInterval( { .log( )
           axios.get( )
                .then( { status = response.data.snapshots[ ].state; .log(status); (status === ){ clearInterval(intervalId);
                        notify( )

                    } (status === || status === ){ clearInterval(intervalId);
                        notify( )
                    } (status === ){ clearInterval(intervalId);
                        notify( )
                    } { }
                })
                .catch( { .log( , error)
                    clearInterval(intervalId);
                })
        }, );

  } notify = { mailOptions = { : , : , : , : message
        };
  
       
        transporter.sendMail(mailOptions, (error, info)=>{ (error) { .log(error);
            } { .log( + info.response);
            }
        });
} milisecConvert = { hours,minutes;
        hours = .floor(milisec/ / / );
        minutes = .floor((milisec/ / / - hours)* ); minutes > ? munites : ;
} const require "axios" const require 'nodemailer' /**
 * Configure email 
 */ let service 'emailservice' auth user 'somemail@domain.com' pass '*****************' const 'http://localhost:9200/_snapshot/azureblob_backup/' const 'tutorial_cluster' let new Date let 1 //months from 1-12 let let let let let let ` - - - - - ` ${year} ${month} ${day} ${hour} ${minute} ${seconds} ` snapshot- ` ${SNAPSHOT_URL} ${backuptime} "ignore_unavailable" true "include_global_state" true ( )=> response console if true console "start checking for status" else console "send failure notification" `Could not start backup for ` ${CLUSTER_NAME} console "Backup Not Started Error ===>" `Could not start backup for ` ${CLUSTER_NAME} let ( ) function let => () console "checking....." ` snapshot- ` ${SNAPSHOT_URL} ${backuptime} ( ) function response // handle success let 0 console if 'SUCCESS' //send a success mail & clear interval ` Has Been Backed Up Successfully \n  completed in minute(s)  \n please check  http:// snapshot- for details` ${CLUSTER_NAME} ${milisecConvert(response.data.snapshots[ ].duration_in_millis)} 0 ${SNAPSHOT_URL} ${backuptime} else if 'ABORTED' 'FAILED' //send failure message & clear interval ` Backup Failed  please check  http:// snapshot- for details ` ${CLUSTER_NAME} ${SNAPSHOT_URL} ${backuptime} else if 'PARTIAL' //send failure message & clear interval ` Backed up with a few issues please check  http:// snapshot- for details ` ${CLUSTER_NAME} ${SNAPSHOT_URL} ${backuptime} else //continue ( ) function error console "request status error >>>>>" 5000 let ( )=> message //set mail options let from 'somemaili@domain.com' to 'tech@yourcompany.com' subject ` Elasticsearch Backup Notification` ${CLUSTER_NAME} text if console else console 'Email sent: ' let ( )=> milisec let Math 1000 60 60 Math 1000 60 60 60 return 1 'less than 1' You need to install the dependencies, and for this script to work and also you should have had your snapshot repository set up already. You can and should set up a cron to run this script at specified intervals. axios nodemailer NOTES Elasticsearch snapshots are incremental, that is if a record already exists in the snapshot it wouldn't be part of the next snapshot. This makes the snapshot process run a lot faster especially after the first run. If you do setup a cron for your snapshots, you should take into consideration of how often your data changes or increases while setting intervals. Did this help? let me know. O dabọ ✌

Utilizing the Elasticsearch Snapshot Module for Databackups on Azure blob Storage

Onboarding is the process of making active customers out of users of your product or service. For a company to be able to win new customers, its onboarding process must be top-notch. And in a world where fail, being among the succeeding 10% would take extra effort. one can expect up to 90% of cryptocurrency startups to The same principles guide the various onboarding processes across the board. But an emphasis on cryptocurrency is necessary because of its uniqueness. Despite the now-growing popularity of digital currencies, many still don’t know how they work because of the complex nature of cryptocurrency and blockchain. It should be noted that when it comes to onboarding, the organization can only have guiding principles but not hard rules. That’s because each company is unique in its own way and has to adapt its strategies to its areas of strength. Here are 10 principles to help businesses easily convert traffic to converts and sales. Simplicity The typically complicated nature of cryptocurrency and blockchain is a turn-off for many and has slowed adoption. Therefore, cryptocurrency platforms, especially a crypto platform that offers revolutionary solutions to businesses, investors, and individuals, will have a great edge if it focuses on simplicity. This is not just about simplicity in design but in the overall process of using the platform. New users should not feel overwhelmed when they come across your product else they will leave. Customer is King In planning an onboarding process, use one that is well-tailored to the users’ needs. You may determine this by using different methods of analytics and also tracking customer actions on your platform. When it comes to cryptocurrency, people really just want to buy and sell the currency. Anything that doesn’t meaningfully contribute to that is mere garbage. Active Research and Analytics Before you begin planning onboarding at all and, perhaps, wasting your efforts, it is good to spend a great deal of time doing some market research. Determine the kind of environment and market that ensure the growth of your product. Learn about potential customers’ needs. Find out things your competitors are doing wrong and make those your selling points. Even after kick-off, perform analytics to know which actions lead to the conversions and if a change is necessary. Not knowing that is one of the . onboarding mistakes to avoid Show and Tell Onboarding is not just to get users to start utilizing your product; it is to show them how in a simple and non-intrusive manner. Instead of which are boring and arduous, short video tutorials and animations would be better to effectively explain all you want to. too many screen-by-screen walkthroughs You know how valuable your product is and what it does, but the real question is: do your customers? Educate For a cryptocurrency platform, it is important to build your onboarding around ensuring that new users are quickly acquainted with the services you offer. While you will have many pro new users, consider that a significant chunk of your user base would consist of cryptocurrency newbies. Therefore, build your onboarding around educating them about the functions of your platform. That is the more reason why you have to keep onboarding very simple. Engage with your Users Onboarding should not be a process that is just about the organization alone. The user too should be actively involved. While showing and telling, asking the users to repeat the processes explained would make for a better experience. Form a Forum Communities (including online ones) are very good. You can create a network of people who use your trading platform and give them an avenue to discuss, seek help, and banter. Online forums give people a sense of belonging and makes users more likely to stay on as lifetime customers. Personalize it If there were a perfect onboarding process, it still wouldn’t sit well with every single user of your product. As a result of that, it is important to make the experience unique and personal for each person. That doesn’t mean creating a different onboarding process (really?) It means ensuring that the process leaves some impact on the user. You surely know what they say about first impressions. Make sure you know your customers. Top executives like the importance of making onboarding personalized. Matt Sornson recognize Request Feedback And yes, it still works. But as mentioned earlier, keep things simple and short. Don’t make your customers start feeling they are sitting for an examination when you give them a lengthy list of questions. You may trigger them to rate your product and leave a review but be sure it is also in a non-intrusive manner. Selling Points Structure your selling point about how trading with you would help the customer. No one really wants to know what your software can do if its features won’t eventually be of help to them. This tweet by Jason Fried of Basecamp years ago sums it up: Conclusion Many make the mistake that onboarding ends when a user signs up for a paid subscription of your product. That is somewhat true, but ignoring everything after that is a disaster. Even after signing up, onboarding still continues and you owe them good treatment. Customer is king, right?

CHUNK BASE

chunkbase.com

-- employees

Since n.d.

COMPANY RANKING

CHUNK BASE

EVERGREEN INDEX #429

Representing this company? Click here to claim and customize this page!

Chunk Base WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

CHUNK BASE

chunkbase.com

-- employees

Since n.d.

COMPANY RANKING

CHUNK BASE

EVERGREEN INDEX #429

RELATED COMPANIESRANK

STORIES ABOUT CHUNK BASE

Representing this company? Click here to claim and customize this page!

Chunk Base WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!