AtScale is built for today’s data platforms and BI tools

AtScale

How to Structure Your K8s GitOps Repository at Scale [Part 1]

Learn More at EditingProtocol.com, where the HackerNoon CMS powers the most up-to-date technical documentation about The Editing Protocol , the independent technology publishing company, has released to the public, a set of rules and guidelines that can be used by humans and machines to determine whether a story is worth publishing, how to specifically improve the story’s content, and how to distribute the story with more reach and relevance. HackerNoon The Editing Protocol What is The Editing Protocol? One of the world leaders in online publishing, HackerNoon is home to over 45,000 published contributors. As a startup, the company was tasked with reviewing thousands of submissions a month with just a small team of editors. To do that, HackerNoon created a set of preliminary checks, rules, and quality guidelines that determine whether or not a story can be published, or should be rejected. This document was dubbed The Editing Protocol and can be read by humans to guide publishing at scale. “In publishing a hundred thousand stories, we’ve learned best practices for how to improve and distribute professional technology content on the internet,” said . HackerNoon Creator and CEO David Smooke “I’m excited to open up this technical documentation for feedback, and continue integrating the most cutting-edge technologies into the Editing Protocol.” To elevate the quality of published content, and optimize the user experience for contributors and readers alike, the editing protocol incorporates several inventive technologies, such as: An AI-powered (GPT-4) headline generator. AI image generator models like Stable Diffusion XL, Stable Diffusion v2.1, Stable Diffusion v2.0, Stable Diffusion v1.5, Midjourney Diffusion, Kandinsky 2.2, and Kandinsky 2.1. AI-generated content detection. AI text translator. AI TLDR generator. Plagiarism checker. Backlink checker bots. The protocol’s guidelines are rule-based and can be easily converted into conditional statements. Once programmed, the protocol can allow human editors to focus on the quality and improvement of publishable stories, while the system automatically informs writers of what rule they have broken, or guidelines they have missed. With The Editing Protocol, small teams can publish content at scale, using both human editors and rule-based flagging systems to provide an efficient publishing process. HackerNoon has already programmed a lot of the protocol into the custom CMS the company has built from the ground up. For instance, submissions that are below the minimum structural quality measure are automatically rejected, and writers are sent an email specifically stating how to improve the story they hope to publish before resubmitting. Furthermore, there is a section in the protocol that highlights the importance of . As one of the most important aspects of online publishing is the visual presentation of the story, HackerNoon built an into their CMS, allowing writers to create original images and help writers adhere to the protocol using the tech itself. originality score AI image generator HackerNoon believes the protocol will not only guide human editors to better vet stories, but also help developers automate the tedious workflows within the traditional editorial review process. How to Use The Editing Protocol The full protocol can be viewed via which is hosted on HackerNoon’s custom CMS builder. https://editingprotocol.com/ With The Editing Protocol, HackerNoon aims to help make the internet better by getting rid of the noise—low-quality self-published articles, spammy content riddled with even spammier links, and a horde of other bad SEO practices that prioritize clicks at the expense of quality. As the Internet changes with , it is likely that The Editing Protocol, too, will change. As such, whenever rules or processes are added, removed, or updated, users can find them on , and use that domain as the ground truth. the emergence of Web3 technologies editingprotocol.com

HackerNoon Releases “The Editing Protocol” - Technical Documentation for Digital Publishing at Scale

হ্যাকারনুন "দ্য এডিটিং প্রোটোকল" রিলিজ করেছে - স্কেলে ডিজিটাল পাবলিশিংয়ের জন্য প্রযুক্তিগত ডকুমেন্টেশন

HackerNoon veröffentlicht „The Editing Protocol“ – technische Dokumentation für digitales Publishing im großen Maßstab

HackerNoon, 대규모 디지털 출판을 위한 기술 문서인 "편집 프로토콜" 출시

HackerNoon выпускает «Протокол редактирования» — техническую документацию для масштабной цифровой публикации

HackerNoon "Düzenleme Protokolü"nü Yayınladı - Büyük Ölçekte Dijital Yayıncılık için Teknik Dokümantasyon

Big Data processing, BI analysis, and AI involve heavy usage of , including This requires tremendous computational power: hundreds of gigabytes of RAM, tens of CPU cores, as well as graphics cards and/or special chips to speed up calculations. ML neural networks. Problems When working in a big heterogeneous team (data analysts and researchers, development engineers, system engineers, etc.) on a large system, employees from different teams use various tools to solve problems. This applies both to development and runtime environments (Python, Matlab, Java, C/C++, etc.) and databases used (RDBMS, NoSQL, files, etc.). Retraining can be time-consuming, expensive, and even impossible in the short run. The introduction of new technologies can make it undesirable or impossible to stick to a universal toolkit. For example, most components of an AI system may be implemented in Java but you may need libraries in Python, and other languages to implement a new model. You will have to integrate them: reimplementing these features in the current toolkit may take a long time and be costly for the business. C/C++, Vertical scaling may be too expensive or impossible. Of course, it depends on the capacity of your provider’s data centers but, in practice, all of them have their limitations. This sets a limit on boosting computational resources in a single node. You will have to use several nodes grouped into a cluster to achieve the required capacity. In some cases, when specialized computing equipment (like , , and Tensor Cores) is required, it is reasonable to buy your own servers and install them in your provider’s data centers. Nvidia Tesla Solutions Containerisation To solve the problems of toolkit heterogeneity, OCI-compliant systems are used. A containerized application is an application and its runtime environment in a particular format that runs in an isolated environment in a container management system. Containers can contain almost any application and system-level applications (WEB, Data, ML, applications, DNS servers, NET balancers, etc.). All of them will run in an isolated environment to avoid conflicts between system libraries and to differentiate access and resources in a flexible way. Docker is the most popular container platform. It includes both tools to build containerized applications and a daemon that lets you run and manage containerized applications within a single node. Orchestration The core problem that Kubernetes is solving is the ability to manage containers at scale. Kubernetes is the most well-known and standardized cluster orchestration system. Kubernetes clusters are available from almost any major cloud provider (AWS, Azure, GCP, etc.). Of course, there are other containerization technologies and container cluster management systems, but Kubernetes on Docker is the most common and standard solution at the moment. Let’s start with the special considerations to be taken into account when choosing a solution for Kubernetes, and then proceed with the specifics of the AI systems architecture in a Kubernetes cluster. As described above, the main options are a managed cluster in the cloud and your own cluster on bare metal. Let's take a closer look at these cases. Cloud Cloud providers offer scalability, ease of use, and high availability through their robust infrastructure and redundancy measures. Cloud services provide flexibility, accessibility, and the ability to seamlessly integrate with other applications. Additionally, cloud providers often have robust security measures and disaster recovery solutions in place, providing reliability and data protection. Bare Metal Bare metal infrastructure provides dedicated hardware resources and offers more control and customization options compared to cloud computing. It is well-suited for workloads that require high performance, low latency, and specific hardware configurations. Furthermore, bare metal infrastructure allows for greater freedom in choosing and implementing specific platform-as-a-service (PaaS) and software-as-a-service (SaaS) solutions, without being limited to proprietary offerings often provided by cloud providers. Comparison Cluster in the cloud Pros Maintenance is performed by the cloud provider. You don’t need a separate cluster administrator, most of the time developers will be able to manage the cluster themselves. Databases and other infrastructure with automatic scaling (S3, DynamoDB, RDS, Redshift, etc.) are available. The number of cluster resources can be changed quickly if necessary. Malfunctions are less frequent. Cons Expensive (please see the comparison below). Higher probability of vendor lock-in when the provider’s technologies are used (e.g., DynamoDB). Malfunctions (when they occur) take longer time to fix. A limited list of appropriate equipment, and you can’t use custom hardware. In many cases, no transparent pricing plans. Comments Of course, it is easier to use a cloud cluster than to deploy your own, at first. But then, when you encounter problems with troubleshooting, finding the cause and possible solution can take much longer and be more painful. You will be limited in your access to the system level, which entails additional difficulties in diagnosing and solving these problems. Some you may not be able to solve by yourself without the access to the system level, so you will have to wait for help from tech support. Be careful when using a cloud provider's proprietary services. They lead to vendor lock-in. If at some point you become dissatisfied with your cloud provider, it will be harder to stop using their services because you already have solutions implemented on the basis of those products (e.g., proprietary NoSQL storages). Cluster on bare metal Pros Renting bare metal servers from hosting providers is several times cheaper than renting nodes from cloud providers (please see an example of price comparison below). Greater flexibility in selection, configuration and installation of the equipment. Colocation. You can build your own equipment and install it in a data center. It is relevant if you use specific equipment (graphics cards, tensor cores, hardware encryption, etc.). Troubleshooting often goes faster as there are no hard vendor lock-ins, which allows for rapid migration and provides a wide choice of solutions. Greater flexibility in the choice and configuration of systems and services. Cons You need a dedicated expert or even an entire administration team for cluster maintenance, including deployment, monitoring, and backup. Malfunctions are more frequent early on, depending on the skills of your expert. Scaling requires more time. It may include adding servers, changing the configuration, or migrating to other racks and/or data centers. Comments When you choose hardware, give preference to dedicated servers. When using virtual servers, you may notice periodic drops in IO or CPU performance, and this affects the speed and stability of the calculations. You should also pay attention to the organization of your server network. Keeping servers in the same data center works fine, but putting them on the same rack and switch is far better. High speed and stability of the connection between your servers fix a lot of problems. For bare metal providers, this service is called colocation. Architecture Deployment As you know, the Kubernetes cluster consists of control and data planes. Let’s focus on the data plane nodes: Data application nodes collect data for ML applications. Master Data is storage where data application pods upload prepared data. If the entire dataset is placed into local node storage, it is recommended to fully replicate the data to all the nodes. This significantly reduces the number of failures and network load when ML application pods read the data. ClickHouse and/or MinIO can be used as Master Data. You can use colocation and connect an entire rack to a single fast 10GE switch. However, even in this case, it is important to group the prepared data properly. Then you will be able to use one of the binary storage formats and avoid overhead expenses for scraping. Models and related functions run on ML application nodes. They perform inference and record the results. Result storage nodes must be able to take in data analysis results, evaluate them and provide access to them externally. You can use ClickHouse and/or MinIO as a Data Master. You can write your own scheduler using Kubernetes API or use built-in, Dagster, Airflow, and other schedulers. During the implementation, it is important to control the number of records in etcd for pods. Filled application pods need to be cleaned now and then, otherwise, etcd will become overfilled and the cluster will die. Load sharing across multiple nodes. When ML applications run, resources are used to their fullest, and this is a typical situation. It is important to always have free resources for system services. An exception is environments for dev testing: they can be located in a single node for cost saving. Backup and Monitoring subsystems should be in a separate space that does not depend on your Data and ML subsystems. It is important that it includes the creation of a set of backups at different points in time and their testing to manage sudden failures, as part of scheduled drills. Last but not least are your master nodes that contain the Kubernetes control plane. Conclusions If your AI system is in the development phase and does not require constant resource utilization, opting for a cloud provider is a sensible choice. By renting spot and on-demand instances for specific tasks, you can avoid paying for idle resources and potentially save money. However, for AI systems that require continuous cluster utilization and have specific hardware needs, employing a bare metal provider becomes a logical consideration, provided your team has the required expertise. In terms of disaster recovery and security, cloud providers typically offer robust built-in measures such as data replication, geographic redundancy, and automated backup and recovery solutions. This provides a level of protection against data loss and ensures business continuity in case of unforeseen events. On the other hand, bare metal infrastructure allows for more control over security measures, enabling organizations to implement their own customized security protocols and maintain direct control over data privacy and compliance. However, this also means that the responsibility for implementing and managing disaster recovery and security measures falls solely on the internal SRE.

AI at Scale: Cloud or Bare Metal

Large organizations have put effort into building an application and frown at developers extracting data from their websites. That is why they put a gateway in the form of user agents to let you know what is permitted. In most sites, you can find these details in the robots.txt file attached to a live URL just like using this link below: https://www.amazon.com/robots.txt In this article, you’ll learn about using Bright Data’s Web Scraper integrated development environment (IDE) to scrape datasets at scale using its ready-made functions and coding templates. Benefits of the Web Scraper IDE These are some of the benefits of using Bright Data’s Web Scraper IDE: The IDE is accessible from within the platform As the leader in proxy infrastructure, it offers scalability and accuracy in web scraping Its code templates help to speed up development It incorporates the Web Unlocker capability through the IDE to avoid captchas and blocking. What is Bright Data? Bright Data is a proxy network that helps you turn websites into structured data. To get started with the platform, create an . account Check out this resource to learn more about . Bright Data https://www.youtube.com/watch?v=YzoLTalL6Uo&embedable=true Working with Web Scraper IDE On your account dashboard, click the icon, and afterward, select the button to open the template window. Datasets and Web Scraper IDE Get started The new window pop-ups a dialog box where you can select pre-existing dataset options to work with or create one from scratch if you desire to do so. Select the options, and the page should look something like this with the collector code. eBay discovery and PDP Now scroll down the page, and under the tab, pass in the name of a product you want to analyze and extract its data. Once done, click the button to run the preview and start the extraction. input Preview : You must also note that you can enter your scripts within the section. PS Interaction code Looking at the result tab after running the preview, it formatted the result from the eBay website based on the following data classification as , , , of the product, and so on. output product_url title image price Saving the collector To save the collector, click on the button to open the configuration page as seen below: Finish editing Initiate the collector by API Under the tab, let’s initiate this project and work with the scripts provided by clicking the button. My Scrapers Initiate by API Creating authorization token Authorization in programming grants access to users and identifies you as the account's rightful owner. Click on the menu at the bottom left of the window to create an API token. Account settings Upon adding the API token, you will receive a token for verification; enter the secret code. Now that is done, copy your API token key, as you won’t be able to retrieve it unless you create a new one for use. Return to the page, and copy the scripts based on your operating system (OS) in your terminal. Make sure to replace with the key you copied in the previous section after the word . New collector API_TOKEN BEARER In your command line interface or terminal, the result of the API code should look something like this: curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"keyword":"ralph lauren polo shirt","count":10,"location":"","condition":"New unused"}]' "https://api.brightdata.com/dca/trigger?collector=c_liopmjh61f3o3lz7dz&queue_next=1" The request command makes the code active in the section of the dashboard page. Once again, please copy and paste the code into the CLI tool. Result API New collector : Remember to put your API token key in place of the value . PS API_TOKEN curl "https://api.brightdata.com/dca/dataset?id=j_liosdy1cdutdi7sod" -H "Authorization: Bearer API_TOKEN" Run the script in the CLI, and the datasets in an object with status should read and a . building message If the response continues to show, retry sending the request. When successful, you should see this result object. Using Postman Like the displayed object above, let’s use to get the response for the . Postman Result API If you do not have Postman, download it . Postman is an API platform for building, publishing, monitoring, testing, and documenting APIs. Check to learn more about Postman and its use. here this resource article Open the Postman app and input these values: In the section in Postman, pass in the URL in the GET method request Click the tab, and select the Bearer Token from the dropdown, pass in your token value Authorization Type Click on button to send the request Send If the is successful, you should see a status message of 200 in the section and an array of objects for the queried scraped data request response eBay Creating a Node Server is a JavaScript runtime environment that allows the execution of JavaScript code outside the web browser enabling developers to build server-side applications and command-line tools. Node Let’s create a web server. One of the requirements for initializing your project in the terminal is using the package manager, npm, which is automatically present after installing on your local machine. Check it using this command: Node.js node --version It displays the current version of Node. Create a new directory. For this project, it is named . datasets Change its directory and initialize the project with the command: cd datasets

npm init -y The flag accepts the defaults that look like this: -y package.json {
  "name": "datasets",
  "version": "1.0.0",
  "description": "",
  "main": "index.mjs",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
} Install the following packages: npm install -D nodemon Nodemon automatically updates and monitors for any changes in your source code and automatically restarts the server. npm install csv-parse The package is a parser for converting CSV text input into an array or objects. csv-parse Now, update the script section in the file to this: package.json {
  "name": "datasets",
  "version": "1.0.0",
  "description": "",
  "main": "index.mjs",
  "scripts": {
    "start": "node index.mjs",
    "start:dev": "nodemon index.mjs"
  },
  ...
} Next, create a new file in the root directory with the command: touch index.mjs To test this file, write a basic JavaScript script and run the server with the following command: npm run start:dev Social Media Data from Bright Data Scraping large datasets requires lots of effort and work using technologies like Node or Python. The way to get around this is to use a platform like Bright Data to obtain the information you need to get your results as soon as possible. Let’s get this dataset from Bright Data which will be the social media platform Instagram, with these steps: Sign up for a account. Bright Data Go to or select the on the . https://brightdata.com/cp/datasets/ Dataset Marketplace Datasets & Web Scraper IDE Open the , and under , select from the dropdown. Dataset Marketplace Categories Instagram.com Social media Click on and download the sample dataset in format. View dataset CSV Make sure to save the dataset in the root directory of the Node web server. Your folder structure should look something like this: .
└── datasets
    ├── node_modules
    ├── instagram.csv
    ├── package-lock.json
    ├── package.json
    └── index.mjs Reading CSV datasets in Node.js For this section, Node will read the comma-separated values (CSV), which dataset is from Bright Data. Update the file with the code: index.mjs import { parse } from "csv-parse";
import { createReadStream } from "node:fs";

const instagramAccount = [];

const isInstagramAccount = (info) => {
  return (
    info["posts_count"] > 300 &&
    info["followers"] > 6000 &&
    info["biography"] !== "" &&
    info["posts"] !== ""
  );
};

createReadStream("instagram.csv")
  .pipe(
    parse({
      columns: true,
    })
  )
  .on("data", (data) => {
    if (isInstagramAccount(data)) {
      instagramAccount.push(data);
    }
  })
  .on("error", (err) => {
    console.log("error", err);
  })
  .on("end", () => {
    console.log(`${instagramAccount.length} accounts are live`);
    console.log("done");
  }); The code above does the following: Using the method to open up a file or stream and read the data in it createReadStream() callback function used to filter the data from the actual CSV file isInstagramAccount : For connecting two streams, which means it connects to a readable stream source into the writeable destination, pipe parse() : Represents returning each row in our CSV file as a Javascript object with key-value pairs rather than just an array of values columns:true : Event handlers chaining pushing the newly created data into the empty array, and displays an error, shows the number of Instagram accounts present, and finally indicates when the script finishes .on instagramAccount done Running the scripts with the command should display the result like this in the terminal: npm run start:dev 643 accounts are live
done Conclusion Web scraping is an integral part of data extraction used in data science. The by Bright Data does all the heavy lifting in the background, presenting only the relevant data for your use. Web Scraper IDE This article walked you through understanding how to use the Web Scraper IDE and how you can build a custom datasets script to query large datasets of companies without fear of getting blocked by the company’s bots designed to help protect the company’s data. Resources CSV Parser for Node.js

How to Scrape Large Datasets at Scale

Recently, due to the various changes in macroeconomic factors, many businesses have faced difficulty coming up with forecasts that are reliable. According to , about 90% of the forecasting models do not go into production. This has been attributed to a complex pipeline that’s unscalable and a generally non-exploratory nature. Venturebeat Additionally, we also notice that a single model is generally not an answer to a range of forecasting needs as forecasting for an organization involves a lot of moving parts. In this article, we will cover general methods for putting time series forecasting models into practice while creating an ensemble approach that backtests a variety of models and chooses the best approach. Table of Contents Business Problem Approach Pipeline Overview Benchmarking Automation Conclusion Business Problem Predict the revenue a product would make over the financial year Revenue impact on the forecast, assuming a change in assumptions (like Take Rate, CPI) A scale to understand Contribution to Growth Anomaly Detection For a typical organization, forecasting is a process that goes hand in hand with budgeting, planning, and allocation. We typically look at a few metrics that we want to forecast and we run a time series model on historical data. During this process, we decompose the forecast into its corresponding trends, seasonality, and residual use. This result is then used to tune and come up with a forecast. Based on the nature and understanding of the business problem in hand we can also build assumptions in the form of regressors that can help instruct the forecast. These assumptions are based on features like spend actuals & intended spend, take rate, conversion efficiency, economic data like publicly available CPI forecasts, custom seasonality flags, etc. Approach Data preprocessing & preparation Forecast Decomposition into Trend, Seasonality, Holidays and Error Improve Accuracy by Cross Validation Top Down Approach vs Bottom Up Approach Running Backtests We now go ahead and come up with the overall approach we want to follow. Some of the questions we would ask during this step are how much data we have, how clean is the data, and how to parse the data to train/test datasets. Additionally, during this stage we also come up with the various models we want to try out and a metric we would consider for benchmarking these models against each other in the form of backtests. Sometimes we could also want to combine the results of multiple models together in an ensemble approach. A Top-down vs bottom-up approach is the methodology of determining whether we want to forecast at the higher level and decompose the forecast or if we want to forecast at a product level and aggregate it to come up with the overall forecast. Pipeline Overview Data Prep using BigQuery Set up VM Instance to deploy Pipeline Machine Learning Libraries like ScikitLearn, Prophet, Arima etc. Control pipeline using Bash Script Deploy Persistent Machines on Cloud Storage For a Typical forecasting project, we start the project by querying the data and creating a dataset that we can use as a data input. In certain cases, we would also go ahead and further create features that make the most sense to use in our training (for example converting categorical variables to Numeric) and then run a model after defining our target variable. Models like Facebook Prophet, Arima, MVAR, etc can all be used during this step. We then look at the results by charting them as well as aggregating the data to multiple granularities (like converting daily forecasts to weekly & monthly). Look at the results. Benchmarking Cross Validation Errors() MAPE = Mean Absolute Percentage Error Comparison (Compare to other metrics) After running a model, we then go ahead and cross-validate the data, using a metric like to understand the model performance. We optimize the forecast using this metric and run backtests and then perform hyperparameter tuning to derive the best parameters that fit the model. MAPE Automation Forecasting at Scale Predict the same data at hourly, daily, or weekly aggregations Run for Multiple Merchants in Parallel using the same Models Test Models in Production Once the core models and forecasting approaches have been selected, the subsequent step involves devising a scalable approach to run the pipeline. This can be achieved through the use of tools such as Airflow to schedule the job or simple chron jobs to create a schedule. During this process, it is important to understand the various files that need to be executed and the models that need to be run. An effective pipeline can be created by following a sequence of steps, starting with Data Preparation, followed by data cleaning and model preparation before executing the model run. Subsequently, data validation and scoring can be carried out before rerunning the model and back-testing the results. The results can then be written into a data table, which can be used to feed a dashboard or outputted as an excel sheet to test assumptions and gain a better understanding of the forecast. Conclusion In conclusion, forecasting is an integral part of business decision-making. It provides organizations with insights into market trends, potential risks, and growth opportunities. It is crucial to ensure that any forecasting automation implemented adds value to the organization by considering factors such as forecast accuracy, data reliability, and potential impact on the business. By doing so, we can provide effective and efficient forecasting automation that supports the organization’s goals and objectives, enabling them to make informed decisions that drive success. Some of the questions we ask ourselves by automating this forecast would be: Who Asked for the forecast? Man Hours Saved Business Impact Operational Impact Lead Image generated with stable diffusion.

Ensemble Time Series Forecasting at Scale

If you’re a marketing department and you want to reach the widest audience possible, text messaging is the best channel to do that with. Most people check their texts more often than email, and text messages have a 90%+ open rate within three minutes of reception. To send SMS and MMS messages at scale you need a cloud communications platform — a.k.a. a communications platform as a service (CPaaS). Peer-to-peer review site G2 lists more than 100 of them, with Plivo ranked number one for customer satisfaction. Throughout this article we’ll use Plivo and Twilio, the CPaaS market share leader, as examples. What’s Your Number? Once you’ve settled on a CPaaS, you have another decision to make: What number type should you use to send your messages? g follows different rules from person-to-person (P2P) texting. Businesses have three options: Commercial application-to-person (A2P) messagin 10-digit long codes (10DLC), toll-free numbers, and short codes. 10DLC Numbers Long codes are normal 10-digit phone numbers — the kind that individuals use every day to engage in P2P conversational messaging. is a service offered by major US carriers to explicitly allow commercial A2P SMS traffic over long code phone numbers. 10DLC is available only in the US. 10DLC You can turn long codes into 10DLC numbers by registering them with a carrier, which you can do through your CPaaS provider. Provisioning a 10DLC number takes less than a week. A vetting process determines the maximum throughput a carrier will allow for a 10DLC number. Throughput is the number of messages they can send in a given time period — messages per second, for example. Toll-Free are used by businesses that want their customers to reach them at no cost. In North America, 800, 888, 877, 866, 855, 844, and 833 denote toll-free numbers. Toll-free numbers Businesses provide toll-free numbers so that incoming calls are free of charge for the calling party, but they can also be enabled for SMS messaging through your carrier or CPaaS, generally in about one business day. Short Codes are five- or six-digit numbers used to send and receive SMS messages. Short codes Unlike long code numbers, shortcodes are not tied to an area code, so they can be used across a country to provide a nationwide brand identity under one easy-to-remember number. Short codes take the longest to acquire — up to 16 weeks. Throughput and Cost Two major differences between number types are their maximum throughput and cost. Throughput Limits Carriers impose varying maximum throughput on different number types. Generally speaking, short codes support the highest throughput and toll-free numbers the lowest. 10DLC numbers sit in the middle. Carrier-imposed throughput for 10DLC numbers depends on vetting scores assigned by an independent organization called The Carrier Registry. Cost Considerations There are three kinds of costs involved in messaging: one-time setup costs, number rental costs, and cost per message. All pricing numbers that we quote below were taken from the two representative vendors’ pricing pages. Short codes incur the highest setup costs: $1,500 for Plivo and $500 for Twilio. 10DLC numbers incur $44 charges for brand and campaign registration and vetting. There are no one-time charges for toll-free numbers. Number rental costs for long codes and toll-free numbers are negligible compared to usage costs for organizations that send a lot of messages. Short code rental costs, however, are significant. Cost per message depends on several factors: whether you’re sending or receiving messages, whether you’re using SMS or MMS, and which carrier you use. Your CPaaS vendor’s pricing page can break it down for you. For comparison purposes, we put together a table showing pricing for sending SMS messages, which for many businesses is the most common use case, and arbitrarily chose T-Mobile as the carrier for which to show surcharges in this table; the “plus” numbers are the T-Mobile carrier surcharges. Run the Numbers All of the different options make it difficult to figure out which number type makes the most economic sense. The best way to decide is to create a spreadsheet that includes the number of SMS and MMS messages you plan to send and receive per month, and factor in your carrier’s surcharges. Regardless of which number type you use, message senders must obtain opt-in consent before texting consumers — and be aware of other considerations. Plivo’s SMS best practices guide lays out those considerations for businesses that want to take advantage of the best channel for business communications — text messaging.

How to Send SMS and MMS Messages at Scale: What is Your Number?

Cómo enviar mensajes SMS y MMS a escala: ¿Cuál es su número?

Comment envoyer des messages SMS et MMS à grande échelle : quel est votre numéro ?

बड़े पैमाने पर एसएमएस और एमएमएस संदेश कैसे भेजें: आपका नंबर क्या है?

SMS および MMS メッセージを大規模に送信する方法: あなたの番号は何ですか?

Como enviar mensagens SMS e MMS em escala: Qual é o seu número?

Cách gửi tin nhắn SMS và MMS ở quy mô lớn: Số của bạn là gì?

如何大规模发送 SMS 和 MMS 消息：您的电话号码是多少？

Recently at work, I came across an interesting method to handle encryption at scale called envelope encryption. First of all, it increases security and helps you ease out the management of encryption keys. But it's also a highly recommended pattern by PCI-DSS (Security Standard for Credit Card Processing) and results in much stronger data privacy and data protection of Personally Identifiable Information (PII). When we think of data there are 3 places we can think of encrypting data: At Rest - On hardware storage devices like on a disk or in your devices In Transit - In moving data between different locations like server to server through API calls In Use - While it's being used by a server (New concept and still being researched) We will be dealing primarily with encryption at rest and envelope encryption is a popular pattern recommended for it. So What is Envelope Encryption? 🤔 This is a type of encryption that involves encrypting your data with a Data Encryption Key, then encrypting the Data Encryption Key (DEK) with a Customer Master Keys (CMK). You then store both the encrypted data and the encrypted DEK alongside each other in the database. This practice of using a wrapping key to encrypt data keys is known as envelope encryption. Like mentioned there are 2 keys you need to understand first before we see how the encryption process takes place: Customer Master Key (CMK) Data Encryption Key (DEK) Customer Master Keys/Root Keys/Key Encryption Keys (CMK) These are symmetric keys used to encrypt, decrypt, and re-encrypt data. It can also generate Data Encryption Keys that you can use outside of the KMS system. They follow the below: Access to these must be restricted to the least endpoints Access to these should be secured through ACL These keys must be stored in a location that is secure like a KMS of a Hardware Security Module (to comply with ) FIPS 140-2 In systems like Google Cloud Key Management Service, you have a hierarchy of keys as seen below with more information to be found . here Data Encryption Keys (DEK) Data keys are encryption keys you can use to encrypt data, including large amounts of data and other data encryption keys. Unlike CMK's, which can't be downloaded, data keys are returned to you for use outside of the KMS. Some of the best practices for DEKs: Generate DEKs locally When stored, always ensure DEKs are encrypted at rest For easy access, store the DEK near the data that it encrypts Generate a new DEK every time you write the data. This means you don't need to rotate the DEKs Do not use the same DEK to encrypt data from two different users Use a strong algorithm such as 256-bit Advanced Encryption Standard (AES) Encryption Process API request is sent to KMS to generate Data key using CMK KMS returns a response with Plain Data key and Encrypted Data key (using CMK) Data is encrypted using Plain Data key Plain Data key is removed from memory Encrypted Data and Encrypted Data Key is packaged together as an envelope and stored Decryption Process Encrypted Data key is extracted from the envelope API request is sent to KMS using Encrypted Data key which has information about CMK to be used in KMS for decryption KMS returns a response with Plain Data Key Encrypted Data is decrypted using Plain Data key Plain Data Key is removed from memory How is Envelope Encryption Different From Other Encryption Patterns? 🤔 Every service you build requires encryption at some point. This could be passwords or PII in a database, credentials for an external service, or even files in a filesystem. Configuration Files You can easily handle some of these situations with a configuration file but they pose their own security risks like: Proper planning is needed to keep the data secure Multiple formats are present e.g - YAML, JSON and XML to name a few Exact storage locations may be hard-coded in the app, making deployment potentially problematic Parsing of the config files can be problematic. Symmetric Encryption You can encrypt data using a symmetric key but they suffer from a major issue which is Key Management. You need to find a way to get the key to the party with whom you are sharing data and if someone gets their hands on a symmetric key, they can decrypt everything encrypted with that key. Asymmetric Encryption You can encrypt data using Asymmetric Encryption which is considered as a standard now a days. Some of the cons of it are: It is a slow process which makes its not suitable for decrypting bulk messages When you lose your private key, your received messages will not be decrypted If your private key is identified by an attacker, all of your messages can be read by him/her Envelope Encryption Some of the benefits offered by it are: - The data is encrypted using a DEK which follows symmetric encryption. The DEK is encrypted by a CMK which follows asymmetric encryption. By using asymmetric encryption, encrypted DEKs can be shared and unencrypted only by those with access to the CMK, mitigating the key exchange problem of symmetric algorithms. A combination of benefits from symmetric and asymmetric encryption - Multiple DEKs can be encrypted under a singular root key and ease the management of keys in a KMS. You can also do more secure key maintenance by rotating your root keys, instead of rotating and re-encrypting all of your DEKs. Easier key management - Because we encrypt the data key with the CMK, we don't have to worry about storing the encrypted data key. Thus, we can safely store the encrypted data key alongside the encrypted data. Data key protection Key Management Systems & Why it Works at Scale? 🤔 The biggest reason for Envelope Encryption and KMSs working at scale is . Like we mentioned before Asymmetric Encryptions are typically slow and Symmetric Encryptions are very fast but the management of keys is the issue. Performance So in Envelope Encryption for a large quantity of data, you quickly encrypt it using symmetric encryption using a random key. Then just the key is encrypted using asymmetric encryption. This gives the benefits of asymmetric encryption, with the performance of symmetric encryption. Key Management Systems like AWS KMS, Azure Key Vault, and Google Cloud Key Management Service gives you a fully managed service to store and manage encryption keys. These use envelope encryption internally, and they’re used by default in a lot of services that support encryption in cloud infrastructure providers like AWS, GCP, Azure, and others. An ideal key management system should be highly available, it should control access to the master key(s), it should audit the key(s) usage, and finally, it should manage key(s) lifecycle. Thus by having the above characteristics and by using envelope encryption internally, Key Management Systems are ideal to handle encryption at scale. Summary Envelope Encryption is one of the most trusted application security design patterns used at scale. It is the default encryption method used in services like AWS S3, GCP, and others. Hopefully, this enables you to understand how you can encrypt/decrypt a large amount of data using the envelope encryption method at scale in a more trusted setup. Thanks for reading! I really hope that you find this article useful. I invite you to participate in the discussion in the comments below, I'm always interested to know your thoughts and happy to answer any questions you might have in your mind. If you think this post was useful, please like the post to help promote this piece to others. Thanks for reading! :) P.S Do feel free to connect with me on or LinkedIn Twitter Appendix This article leans heavily on the following material: Google Cloud Data Encryption - Jayendra's Cloud Certification Blog - AWS KMS concepts - AWS AWS KMS and Envelope Encryption - Manish Pandit Cloud Architecture Pattern: Envelope Encryption (or Digital Envelope) with Public Cloud Providers Part 1 - Nilay Parikh AWS KMS Envelope Encryption - Chirag Modi Protecting data with envelope encryption - IBM Envelope encryption - GCP Encryption at rest in Google Cloud - GCP Managing encryption of data in the cloud (Google Cloud Next '17) - Maya Kaczorowski

Managing Encryption at Scale with Envelope Encryption

There are some common, simple elements that I’ve observed when agile works at scale across the variety of medium-to-large organizations I’ve had the good fortune of working with. Making agile work at scale is ultimately about changing behavior, not just for the individual contributors involved, but also for the leaders. Changing individual behavior is hard, so trying to do change at scale is, by definition, even harder. This difficulty leads us as individuals to complex solutions. “It’s hard, so we must need a complex way to solve it”. It’s what we like to do as humans. I’m guilty of it more often than I’d like. So, I’ve tried to distill the research, discussions, opinions, and practical observations that I’ve come across over the years into a simple view of what makes agile work at scale. The standout elements are: Engaged leaders Empowered teams Sharing what’s working Time These are the elements that have been present when agile has worked at scale. When it doesn’t work, there are a multitude of different specific reasons, but you can usually trace it back to one of these elements not being present. I’ll unpack each of these elements below and then walk through what I’ve omitted and why. Common Elements When Agile Works at Scale 1. Engaged Leaders When agile works, it has engaged leaders. Leaders that understand and model the values and principles from the agile manifesto. Leaders that clear the obstacles and make the organizational changes necessary to empower teams to deliver outcomes. It’s also just as important for leaders to be explicit about how agile will be adopted, particularly the parts that won’t be adopted or will be challenging. This sets expectations with the teams. 2. Empowered Teams Teams need to be empowered to deliver outcomes. Bringing this back to the Agile Manifesto, it’s usually about allowing teams to work the way they want rather than imposing processes on them. It’s about allowing them to focus on outcomes over powerpoints reporting on output. It’s about allowing teams to adapt to what they think the customer and the business needs over fixing them to a plan. The teams themselves are also best placed to bring about change, especially the change that is right for them. Most teams know their problems and have ideas on how to solve them. The teams themselves also have the power to change their ways of working, to change their own mindsets, and to influence each other. 3. Sharing What’s Working Often organizations already have pockets of teams and leaders that are shining lights on better practices (doesn’t even need to be agile-related). But, these teams and their mindset, ways of working, and insights often don’t make it beyond their immediate surroundings. Allowing and supporting teams to share knowledge helps put practices into context and helps demonstrate that “yes, it can be done here, here is an example”. 4. Time Lastly, the other often overlooked element when agile works at scale is time. Change takes time and patience. Time to get the right people in the right roles, time for behavior to change, time to iterate towards the right working model for your organization. I’m guessing it doesn’t show up much in frameworks or models because we can’t control it. What Wasn’t Common There are a few ideas that are often put forward as essential to agile working at scale. However, in my observations and research, these weren’t what made agile work. The ideas that aren’t essential to success are: is common to agile working at scale (think SAFe, Scrum, or the Spotify Model). In fact, most organizations that I’ve seen running agile at scale use a variety, and some leave it to the teams. No single process, methodology, or framework There are organizations that have used great coaches to great effect. There are organizations that have made agile work at scale and have done so without coaches, looking instead to their senior leaders and team leaders to mentor their teams. Agile coaches. Most teams that made agile work at scale used a variety of tools and technology. You don’t need to be using the latest technology, although it helps. And you don’t need a specific toolset, there are plenty of options. Technology and tools. Closing Thoughts Broadly, these elements suggest that approaching agile at scale from the top-down bottom-up with the patience to allow time for change to occur might be the right framing to come from. and On reflection, these elements don’t seem unique to agile and could easily be applied to the use of other values systems. Agile isn’t the right solution for every organization and every situation. It’s perfectly okay to adopt the principles and values that work best for your situation. But be honest and clear about this, don’t ignore it. I’d love to see some research that simplifies the common elements; if you have any, please send it through. The book probably has the closest so far. Accelerate The main reason for publishing this was to collect my own thoughts after the discussion generated by my article ‘ Agile is Dead, McKinsey Killed It ’.

ATSCALE

atscale.com

-- employees

Since n.d.

COMPANY RANKING

ATSCALE

EVERGREEN INDEX #1195

Representing this company? Click here to claim and customize this page!

AtScale WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

ATSCALE

atscale.com

-- employees

Since n.d.

COMPANY RANKING

ATSCALE

EVERGREEN INDEX #1195

RELATED COMPANIESRANK

STORIES ABOUT ATSCALE

Representing this company? Click here to claim and customize this page!

AtScale WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!