TD Synnex (SNX) A multinational firm based in Fremont, California, TD Synnex (NYSE:SNX) provides information technology (IT) services to businesses. Fundamentally, the negative catalyst impacting SNX centers on the mass layoffs that companies have ...

7 De-Risked Tech Stocks to Pick Up for Cheap

Software developers are in high demand and learning on the job is back in vogue. Here's why experts say some programming certifications are still worth your time.

Do programming certifications still matter?

Datech, the specialist design software business of TD SYNNEX, has set out plans to expand its solution portfolio.

TD SYNNEX Datech Broadens Its Design Solutions Portfolio With Additional Specialist Vendors

Companies in the Technology sector have received a lot of coverage today as analysts weigh in on ACM Research (ACMR – Research Report),

Analysts Offer Insights on Technology Companies: ACM Research (ACMR), Exlservice Holdings (EXLS) and SPS Commerce (SPSC)

Trump's Pardon of General Flynn Under Scrutiny After Sidney Powell Flips Here's what 'wealthy' means in 2023 America, in five numbers How Micron is building the biggest chip fab in U.S. history despite a China ban and smartphone slump Peanuts by Charles ...

Sintx Technologies files to sell 8.9M units

The Computer and Technology group has plenty of great stocks, but investors should always be looking for companies that are outperforming their peers. Has NetEase (NTES) been one of those stocks this year?

Has NetEase (NTES) Outpaced Other Computer and Technology Stocks This Year?

Verinext and 11 Event Sponsors Support FOCUS' Efforts to Help Make Everyday Life Better for Children with Disabilities and their Families

Verinext Annual Charity Golf Classic Raises $12,000 for FOCUS

TD SYNNEX (NYSE: SNX), a leading global distributor and solutions aggregator for the IT ecosystem, has relaunched its MyShop e-commerce platform for U.S. partners with industry-leading enhancements and updates, including virtual commerce (v-commerce) and artificial intelligence (AI) features.

TD SYNNEX MyShop E-Commerce Platform Relaunched with Industry Leading Enhancements and Technology

For nearly a decade, Cloudflare has been investing in Latin America, deploying our infrastructure in the region," said Michelle Zatlyn, Co-founder, President & COO of Cloudflare. "Across sectors, cybersecurity and scaling operations are at the forefront.

Cloudflare Expands Operations in Mexico Amid Growth Across Its Infrastructure and Customer Base

TD SYNNEX (NYSE: SNX), a leading global distributor and solutions aggregator for the IT ecosystem, has relaunched its MyShop e-commerce platform for U.S. partners with industry-leading enhancements and updates,

Synnex

synnex

The response to our previous post, , was extraordinary - we’ve fielded dozens of calls from enterprises asking us for repatriation advice. We have aggregated those responses into this new post, where we dig a little deeper into the costs and savings associated with repatriation to make it easier for you to put together your own analysis. Migration of data is a daunting task for many. In practice, they target new data to come to MinIO and take their sweet time to migrate old data from the cloud or leave it in place and not grow. How to Repatriate From AWS S3 to MinIO Repatriation Overview To repatriate data from AWS S3, you will follow these general guidelines: Determine the specific buckets and objects that need to be repatriated from AWS S3. Make sure you understand business needs and compliance requirements on a bucket-by-bucket basis. Review Data Requirements: You’ve already decided to repatriate to MinIO, now you can choose to run MinIO in an on-premises data center or at another cloud provider or colocation facility. Using the requirements from #1, you will select hardware or instances for forecasted storage, transfer and availability needs. Identify Repatriation Destination: Plan and execute the transfer of data from AWS S3 to MinIO. Simply use MinIO's built-in Batch Replication or mirror using the MinIO Client (see for details). There are several additional methods you can use for data transfer, such as using AWS DataSync, AWS Snowball or , or directly using AWS APIs. Data Transfer: How to Repatriate From AWS S3 to MinIO TD SYNNEX data migration Ensure that appropriate access controls and permissions are set up for the repatriated data on a per-bucket basis. This includes IAM and bucket policies for managing user access, authentication, and authorization to ensure the security of the data. Data Access and Permissions: It is critical to preserve the object lock retention and legal hold policies after the migration. The target object store has to interpret the rules in the same way as Amazon S3. If you are unsure, ask for the on the target object store implementation. Object Locks: Cohasset Associates Compliance Assessment Define and implement a data lifecycle management strategy for the repatriated data. This includes defining retention policies, backup and recovery procedures, and data archiving practices on a per-bucket basis. Data Lifecycle Management: Validate the transferred data to ensure its integrity and completeness. Perform necessary checks and tests to ensure that the data has been successfully transferred without any corruption or loss. After the transfer, the object name, ETag and metadata, checksum and the number of objects all match between the source and destination. Data Validation: The good news is that if you follow cloud-native principles to build your applications, then all you will have to do is reconfigure them for the new MinIO endpoint. However, if your applications and workflows were designed to work with the AWS ecosystem, make the necessary updates to accommodate the repatriated data. This may involve updating configurations, reconfiguring integrations or in some cases modifying code. Update Applications and Workflows: Continuously monitor and optimize the repatriated data environment to ensure optimal performance, cost-efficiency, and adherence to data management best practices. Monitor and Optimize: Repatriation Steps There are many factors to consider when budgeting and planning for cloud repatriation. Fortunately, our engineers have done this with many customers and we’ve developed a detailed plan for you. We have customers that have repatriated everything from a handful of workloads to hundreds of petabytes. The biggest planning task is to think through choices around networking, leased bandwidth, server hardware, archiving costs for the data not selected to be repatriated, and the human cost of managing and maintaining your own cloud infrastructure. Estimate these costs and plan for them. Cloud repatriation costs will include data egress fees for moving the data from the cloud back to the data center. These fees are intentionally high enough to compel cloud lock-in. Take note of these high egress fees - they substantiate the economic argument to leave the public cloud because, as the amount of data you manage grows, the egress fees increase. Therefore, if you’re going to repatriate, it pays to take action sooner rather than later. We’re going to focus on data and metadata that must be moved – this is eighty percent of the work required to repatriate. Metadata includes bucket properties and policies (access management based on access/secret key, lifecycle management, encryption, anonymous public access, object locking and versioning). Let’s focus on data (objects) for now. For each namespace you want to migrate, take inventory of the buckets and objects you want to move. It is likely that your DevOps team already knows which buckets hold important current data. You can also use . At a high level, this will look something like: Amazon S3 Inventory Namespace Total Buckets Total Object Count Total Object Size (GB) Daily Total Upload (TB) Daily Total Download (TB) ns-001 166 47,751,258 980,014.48 50.04 14.80 ns-002 44 24,320,810 615,033.35 23.84 675.81 ns-002 648 88,207,041 601,298.91 328.25 620.93 ns-001 240 68,394,231 128,042.16 62.48 12.45 The next step is to list, by namespace, each bucket and its properties for every bucket you’re going to migrate. Note the application(s) that store and read data in that bucket. Based on usage, classify each bucket as hot, warm or cold tier data. In an abridged version, this will look something like Bucket Name Properties App(s) Hot/Warm/Cold Tier A Copy and paste JSON here Spark, Iceberg, Dremio Hot B Copy and paste JSON here Elastic Warm C Copy and paste JSON here Elastic (snapshots) Cold You have some decisions to make about data lifecycle management at this point and pay close attention because here’s a great way to save money on AWS fees. Categorize objects in each bucket as hot, warm or cold based on how frequently they are accessed. A great place to save money is to migrate cold tier buckets directly to S3 Glacier – there’s no reason to incur egress fees to download just to upload again. Depending on the amount of data you’re repatriating, you have a few options to choose how to migrate. We recommend that you load and work with new data on the new MinIO cluster while copying hot and warm data to the new cluster over time. The amount of time and bandwidth needed to copy objects will, of course, depend on the number and size of the objects you’re copying. Here’s where it will be very helpful to calculate the total data that you’re going to repatriate from AWS S3. Look at your inventory and total the size of all the buckets that are classified as hot and warm. Total Hot and Warm Tier Data = 1,534,096.7 GB Available bandwidth = 10 Gbps Minimum Transfer Time required (total object size / available bandwidth) = 14.2 days Calculate data egress fees based on the above total. I’m using , but your organization may qualify for a discount from AWS. I’m also using 10 Gbps as the connection bandwidth, but you may have more or less at your disposal. Finally, I’m working from the assumption that one-third of S3 data will merely be shifted to S3 Glacier Deep Archive. list price Total Data Tiered to S3 Glacier = 767,048.337 GB S3 to S3 Glacier transfer fees ($0.05/1000 objects) = $3,773.11 S3 Glacier Deep Archive monthly storage fee = $760 Don’t forget to budget for S3 Glacier Deep Archive usage moving forward. Total Data to be Transferred = 1,534,096.7 GB First 10 TB at $0.09/GB = $900 Next 40 TB at $0.085/GB = $3,400 Next 100 TB at $0.07/GB = $70,000 Additional over 150 TB at $0.05/GB = $69,205 Total Egress Fees = $143,504 For the sake of simplicity, the above calculation includes neither the fee for per object operations ($0.40/1m) nor the cost of LISTing ($5/1m). For very large repatriation projects, we can also compress objects before sending them across the network, saving you some of the cost of egress fees. Another option is to use AWS Snowball to transfer objects. Snowball devices are each 80TB, so we know up front that we need 20 of them for our repatriation effort. The per-device fee includes 10 days of use, plus 2 days for shipping. Additional days are available for $30/device. 20 Snowball Devices Service Fee ($300 ea) = $6,000 R/T shipping (3-5 days at $400/device) = $8,000 S3 data out ($0.02/GB) = $30,682 Total Snowball Fees = $38,981.93 AWS will charge you standard request, storage, and data transfer rates to read from and write to AWS services including and . There are further considerations when working with . For S3 export jobs, data transferred to your Snow Family device from S3 are billed at standard S3 charges for operations such as LIST, GET, and others. You are also charged standard rates for Amazon CloudWatch Logs, Amazon CloudWatch Metrics, and Amazon CloudWatch Events. Amazon S3 AWS Key Management Service (KMS) Amazon S3 storage classes Now we know how long it will take to migrate this massive amount of data and the cost. Make a business decision as to which method meets your needs based on the combination of timing and fees. At this point, we also know the requirements for the hardware needed to run MinIO on-prem or at a colocation facility. Take the requirement above for 1.5PB of storage, estimate data growth, and consult our page and . Recommended Hardware & Configuration Selecting the Best Hardware for Your MinIO Deployment The first step is to recreate your S3 buckets in MinIO. You’re going to have to do this regardless of how you choose to migrate objects. While both S3 and MinIO store objects using server-side encryption, you don’t have to worry about migrating encryption keys. You can connect to your KMS of choice using . This way, new keys will be automatically generated for you as encrypted tenants and buckets are created in MinIO. MinIO KES to manage encryption keys You have multiple options to copy objects: Batch Replication and . My previous blog post, included detailed instructions for both methods. You can copy objects directly from S3 to on-prem MinIO, or use a temporary MinIO cluster running on EC2 to query S3 and then mirror to on-prem MinIO. mc mirror How to Repatriate From AWS S3 to MinIO Typically, customers use tools we wrote combined with AWS Snowball or TD SYNNEX’s data migration hardware and services to move larger amounts of data (over 1 PB). MinIO recently partnered with Western Digital and TD SYNNEX to field a Snowball alternative. Customers can schedule windows to take delivery of the Western Digital hardware and pay for what they need during the rental period. More importantly, the service is not tied to a specific cloud - meaning the business can use the service to move data into, out of, and across clouds - all using the ubiquitous S3 protocol. Additional details on the service can be found on the page on the TD SYNNEX site. Data Migration Service Bucket metadata, including policies and bucket properties, can be read using and then set up in MinIO. When you sign up for MinIO SUBNET, our engineers will work with you to migrate these settings from AWS S3: access management based on access key/secret key, lifecycle management policies, encryption, anonymous public access, immutability and versioning. One note about versioning, AWS version ID isn’t usually preserved when data is migrated because each version ID is an internal UUID. This is largely not a problem for customers because objects are typically called by name. However, if AWS version ID is required, then we have an extension that will preserve it in MinIO and we’ll help you enable it. get-bucket S3 API calls Pay particular attention to . S3 isn’t going to be the only part of AWS’s infrastructure that you leave behind. You will have a lot of service accounts for applications to use when accessing S3 buckets. This would be a good time to list and audit all of your service accounts. Then you can decide whether or not to recreate them in your identity provider. If you choose to automate, then use Amazon Cognito to share IAM information with external OpenID Connect IDPs and AD/LDAP. IAM and bucket policies Pay particular attention to Data Lifecycle Management, such as object retention, object locking and archive/tiering. Run a on each bucket to obtain a human-readable JSON list of lifecycle rules. You can easily recreate AWS S3 settings using MinIO Console or MinIO Client (mc). Use commands such as and to pinpoint objects that require special security and governance treatment. get-bucket-lifecycle-configuration get-object-legal-hold get-object-lock-configuration While we’re on the subject of lifecycle, let’s talk about backup and disaster recovery for a moment. Do you want an additional MinIO cluster to replicate to, for backup and disaster recovery? After objects are copied from AWS S3 to MinIO, it’s important to validate data integrity. The easiest way to do this is to use the MinIO Client to run against old buckets in S3 and new buckets on MinIO. This will compute the difference between the buckets and return a list of only those objects that are missing or different. This command takes the arguments of the source and target buckets. For your convenience, you may want to create for S3 and MinIO so you don’t have to keep typing out full addresses and credentials. For example: mc diff aliases mc diff s3/bucket1 minio/bucket1 The great news is that all you have to do is point existing apps at the new MinIO endpoint. Configurations can be rewritten app by app over a period of time. Migrating data in object storage is less disruptive than a filesystem, just change the URL to read/write from a new cluster. Note that if you previously relied on AWS services to support your applications, those won’t be present in your data center, so you’ll have to replace them with their open-source equivalent and rewrite some code. For example, Athena can be replaced with Spark SQL, Apache Hive and Presto, Kinesis with Apache Kafka, and AWS Glue with Apache Airflow. If your S3 migration is part of a larger effort to move an entire application on-prem, then chances are you used to call downstream services when new data arrived. If this is the case, then do not fear - MinIO supports as well. The most straightforward migration here would be to implement a custom webhook to receive the notification. However, if you need a destination that is more durable and resilient, then use messaging services such as Kafka or RabbitMQ. We also support sending events to databases such as PostgreSQL and MySQL. S3 event notifications event notification Now that you’ve completed repatriating, it’s time to turn your attention to storage operation, monitoring and optimization. The good news is that no optimization is needed for MinIO – we’ve built optimization right into the software so you know you’re getting the best performance for your hardware. You’ll want to start monitoring your new MinIO cluster to assess resource utilization and performance on an ongoing basis. MinIO exposes via a Prometheus endpoint that you can consume in your . For more on monitoring, please see and . metrics monitoring and alerting platform of choice Multi-Cloud Monitoring and Alerting with Prometheus and Grafana Metrics with MinIO using OpenTelemetry, Flask, and Prometheus With , we have your back when it comes to with MinIO. Subscribers gain access to built-in automated troubleshooting tools to keep their clusters running smoothly. They also get unlimited, direct-to-engineer support in real-time via our support portal. We also help you future-proof your object storage investment with an annual architecture review. SUBNET Day 2 operations Migrate and Save It’s far from a secret that the days of writing blank checks to cloud providers are gone. Many businesses are currently evaluating their cloud spend to find potential savings. Now you have everything you need to start your migration from AWS S3 to MinIO, including concrete technical steps and a financial framework. If you get excited about the prospect of repatriation cost savings, then please reach out to us at . hello@min.io Also appears . here

All You Need to Know to Repatriate from AWS S3 to MinIO

AWS S3 থেকে MinIO-তে প্রত্যাবাসন করতে আপনার যা জানা দরকার

Alles, was Sie zur Repatriierung von AWS S3 zu MinIO wissen müssen

Todo lo que necesita saber para repatriar de AWS S3 a MinIO

Tout ce que vous devez savoir pour rapatrier d'AWS S3 vers MinIO

AWS S3 से MinIO में स्थानांतरण के लिए आपको जो कुछ भी जानना चाहिए

AWS S3 から MinIO への復帰に必要なすべての情報

AWS S3에서 MinIO로 송환하기 위해 알아야 할 모든 것

Tudo o que você precisa saber para repatriar do AWS S3 para MinIO

Все, что вам нужно знать для репатриации с AWS S3 на MinIO

AWS S3'ten MinIO'ya Geri Dönmek İçin Bilmeniz Gereken Her Şey

Tất cả những điều bạn cần biết để hồi hương từ AWS S3 về MinIO

从 AWS S3 迁移到 MinIO 所需了解的一切

As the world becomes increasingly digital, nearly every company is looking for to code—no matter the industry. The phenomenon has been great news for software engineers and programmers who can now command high salaries for their hot, in-demand talent. software engineers Perhaps it is no surprise that software engineers in the San Francisco Bay Area, home to Silicon Valley and its many tech giants, can command an even higher premium with their pay packages. According to the more than 5 million verified professionals on the professional social network Blind, these are the top 17 best-paying cities for software engineers in the San Francisco Bay Area. Blind considered the annual base pay and the total compensation package, including the annual salary and any stock-based compensation or bonuses a company might offer. 1. Los Gatos, California n: $500,449 Average Total Compensatio : $475,000 Median Total Compensation : $470,015 Average Salary : $451,000 Median Salary Los Gatos is the best-paying city for software engineers in the U.S., according to Blind. The small town is home to Netflix, which prefers to hire more senior or experienced professionals and pays accordingly. The video-streaming giant typically pays large salaries and gives few employees stock-based compensation. 2. Menlo Park, California : $328,423 Average Total Compensation : $297,000 Median Total Compensation : $167,004 Average Salary : $165,000 Median Salary Menlo Park might be home to only 34,000 people, but its population can swell in size with tech workers during the workweek. Menlo Park is home to the headquarters of Meta, the owner of Facebook and one of the highest-paying technology companies. Google and Roblox were also founded in the city. 3. Cupertino, California n: $279,391 Average Total Compensatio : $255,850 Median Total Compensation : $165,753 Average Salary : $165,000 Median Salary Cupertino is home to Apple and its futuristic Apple Park corporate campus. Software engineers at Apple have enjoyed large paychecks recently, as the iPhone maker briefly topped a jaw-dropping $3 trillion market cap early in 2022. 4. San Bruno, California : $278,368 Average Total Compensation : $225,900 Median Total Compensation : $161,269 Average Salary : $149,500 Median Salary YouTube moved its headquarters to San Bruno in 2007 and has become the city’s largest employer. The Google-owned video streaming service employs more than 2,000 professionals. 5. Mountain View, California : $275,181 Average Total Compensation : $242,000 Median Total Compensation : $160,790 Average Salary : $156,000 Median Salary Mountain View is synonymous with Silicon Valley. Long before startups were birthed in San Francisco Bay Area garages, the city was home to silicon semiconductor producers, which gave the region its name. Of course, the city is now well-known as the headquarters of Google, but other tech companies with prominent presences include 23andMe, Intuit, Mozilla and Synopsys. 6. Oakland, California : $266,869 Average Total Compensation : $197,750 Median Total Compensation : $192,240 Average Salary : $168,750 Median Salary Oakland has become a hub for many financial-technology companies. Credit Karma moved its headquarters from San Francisco to downtown Oakland in 2020. Block and Marqeta also have major offices in “The Town.” 7. San Francisco, California : $265,050 Average Total Compensation : $225,000 Median Total Compensation : $165,042 Average Salary : $160,200 Median Salary San Francisco might be the crown jewel of the technology industry, as the home of too many tech startups and companies to count. The city by the bay is home to Airbnb, DoorDash, Lyft, Twitter, Uber and more, but its top employer is . The cloud giant employs more than 9,000 employees, and its headquarters has made a permanent contribution to the downtown skyline. Salesforce 8. Sunnyvale, California : $260,048 Average Total Compensation : $238,000 Median Total Compensation : $160,499 Average Salary : $160,000 Median Salary Sunnyvale maintains its Silicon Valley roots as home to the chip-maker AMD and advanced technology and networking companies, including Juniper Networks, Microsoft-owned LinkedIn and NetApp. 9. Palo Alto, California : $234,739 Average Total Compensation : $200,000 Median Total Compensation : $157,488 Average Salary : $154,000 Median Salary The tony city of Palo Alto is one of the major cities of Silicon Valley. Cloudera and VMware are some of the largest technology companies based in the city. 10. San Mateo, California : $234,066 Average Total Compensation : $187,875 Median Total Compensation : $158,629 Average Salary : $155,500 Median Salary Companies that call San Mateo home include Coupa, Roblox and Snowflake. Before being acquired by Zendesk, Momentive, which was founded as SurveyMonkey, was headquartered in the city. 11. Pleasanton, California : $225,058 Average Total Compensation : $202,000 Median Total Compensation : $150,887 Average Salary : $150,000 Median Salary Pleasanton is home to many business parks, but the suburb’s top employer is the HR-technology giant Workday. 12. Foster City, California : $220,293 Average Total Compensation : $190,000 Median Total Compensation : $158,319 Average Salary : $148,500 Median Salary Foster City is the headquarters site of the biopharmaceutical giant Gilead Sciences and the enterprise software company Zuora, among other companies. 13. Fremont, California : $218,913 Average Total Compensation : $180,000 Median Total Compensation : $144,638 Average Salary : $142,500 Median Salary Notable companies with headquarters in Fremont include Corsair, Synnex and more, although the city’s No. 1 employer is Tesla. 14. Santa Clara, California : $214,168 Average Total Compensation : $197,000 Median Total Compensation : $153,885 Average Salary : $150,000 Median Salary AMD, Intel, NVIDIA and Palo Alto Networks are some of the most well-known tech companies headquartered in Santa Clara. 15. San Jose, California n: $210,801 Average Total Compensatio : $183,000 Median Total Compensation : $153,390 Average Salary : $145,000 Median Salary The largest city by population in the San Francisco Bay Area, San Jose is home to Adobe, Cisco, eBay, PayPal, Zoom and more. 16. Redwood City, California : $207,955 Average Total Compensation : $190,000 Median Total Compensation : $156,248 Average Salary : $151,000 Median Salary Redwood City is the headquarters location for several global technology companies, such as Box, Carbon, Electronic Arts and Informatica. 17. Milpitas, California : $188,146 Average Total Compensation : $165,750 Median Total Compensation : $154,073 Average Salary : $138,000 Median Salary Some of the top employers in Milpitas are tech employers, including Cisco, FireEye, SanDisk and more. Methodology Blind analyzed the base salaries and total compensation of software engineers in the San Francisco Bay Area on its platform. Cities were ranked by the average total compensation, including annual base salary and, if applicable, stock-based compensation and cash bonuses. All data is self-reported by verified professionals on the professional social network Blind. Also published on Teamblind's blog

17 Highest-Paying Cities in the San Francisco Bay Area for Software Engineers

Las 17 ciudades que mejor pagan en el área de la bahía de San Francisco para ingenieros de software

17 villes les mieux rémunérées de la région de la baie de San Francisco pour les ingénieurs logiciels

सॉफ्टवेयर इंजीनियरों के लिए सैन फ्रांसिस्को खाड़ी क्षेत्र में 17 उच्चतम-भुगतान वाले शहर

ソフトウェア エンジニアにとって、サンフランシスコ ベイエリアで最も給料の高い 17 の都市

17 cidades mais bem pagas na área da baía de São Francisco para engenheiros de software

17 thành phố trả lương cao nhất trong khu vực vịnh San Francisco cho các kỹ sư phần mềm

旧金山湾区软件工程师薪酬最高的 17 个城市

How to Build and Manage Projects From the Ground Up When managing a project, you will rely on tools like Trello, Jira, or your own backlog in Excel. You may build a mindmap that shows how many branches your product has. But, as a product manager who is also doing some full-stack development and working on a complex project, it’s sometimes difficult to have a good overview of what you are doing. Sure, we create our epics, our user stories, add tasks, we have sketches, and more. But these are not methods that visualize how we build an app from zero to hero. And sometimes, it’s necessary to see what relies on what. There are ways to show us this, of course, but it can be done better just to help us. I have been working with Masonries. A masonry is a visual bottom-up approach for project structures that I have developed to better display what tasks I have to complete before a parent task is completed and how it affects the user story or its epic. It has some similarities to Treemaps. Fictional Example In a fictional example, I want to show how I build and use a Masonry. In my example, we want to build an app that allows authors to write interesting blog articles. To make it not too complex, let’s focus on just two user stories. The image above shows us two user stories. The first is the author who wants to write and publish an article. The second one is the author who wants to view and edit or delete a published article. What do these stories need? Apart from the second which needs a way to display all articles, we want to Open the article editor. Enable markdown syntax in the editor Enable image import Publish the article. The colors here indicate (I apologize if you are red-green blind) that a green-marked task, enabling image import, is done. Publishing an article is in progress while enabling markdown syntex in red is yet to be done. The parent task, opening the editor, so far, is affected as it’s yellow or “in progress.” This again affects the next task or user story. It could, of course, be possible to have a wider brick layer below the tasks, e.g., if you need authentication for a specific feature. You can also, as seen in the example image above, reuse tasks. Like a mason, we put brick on brick, and the user story is nothing without the completed task; the epic is nothing without the completed user stories. In a project, you can have multiple Masonries to build a House where the house is your project. Pros and Cons of Masonry The pros and cons of Masonry are so far clear, I guess. Pros: better visualization of dependencies from-scratch build-up method easy to read Cons: Takes some time to build (no software yet) No time estimates or story points No assignments For paper, you need to cut out paper The concept of Masonry is just new. But it helped me more than mind mapping to have an overview of complex projects and makes a more efficient product manager. What’s your opinion on Masonry? Can you imagine using this approach? If you do, share your experiences. If not, please share your reasons why. Also published here

Masonry — An Approach You Should Try Out

By: David Zhu Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under storage systems, thus making it simple for users to retrieve the latest version of data from Alluxio. Meanwhile, understanding the internal process is important in order to tune the performance. This article describes the design and the implementation in Alluxio to keep metadata synchronized. Why is Metadata Sync Critical in Alluxio In Alluxio, metadata refers to the information of files and directories in the Alluxio file system, including information of their owners, groups, permission, creation and modification time, etc. Metadata is independent of their content — even if a file or directory is empty, it still has associated metadata. Alluxio maintains a copy of the file system or the object store namespace of the under storage system. Metadata consistency is important in Alluxio, especially when different clusters write/read data in the data pipeline and make changes outside of Alluxio. A typical scenario in the above diagram is a data pipeline with the combination of Spark ETL and Presto SQL. The ETL cluster (without Alluxio) writes data, followed by the analytics cluster with Alluxio reading the transformed data. Since Alluxio maintains a copy of the metadata from under storage and manages the metadata, when data changes in under storage with the ETL step, the instance of Alluxio on the analytics cluster must be made aware and kept consistent with the metadata in the underlying storage system to proceed correctly. How Does Metadata Sync Work in Alluxio Alluxio provides a file system abstraction in a unified namespace on one or more under storage systems. Accessing files or directories through Alluxio is expected to get the same result as directly accessing the under storage. For example, if the under storage mounted to Alluxio root is , listing the directory in Alluxio yields the same result as listing objects in and printing in Alluxio should return the same result as . s3://bucket/data "/" s3://bucket/data "/file" s3://bucket/data/file In Alluxio, metadata is only stored and served from Alluxio masters, whereas the content of individual files is served from Alluxio workers. By default, Alluxio loads the metadata from the under storage on demand. In the above example, an Alluxio master starting from empty does not have any information about after startup. This file is only recognized when some user lists directory in Alluxio or tries to access . This “lazy” behavior prevents unnecessary effort and significantly improves performance because metadata operations in under storage can be slow. s3://bucket/data/file "/" "/file" Note that updating metadata can be bidirectional. If all modifications to the file system occur through Alluxio, Alluxio only needs to scan under storage once to retrieve the initial state, then apply changes in both Alluxio and under storage synchronously as a part of the file system RPC calls. This will provide users with a consistent view of under storage. In reality, however, changes are often made to the under storage outside of Alluxio. As a result, Alluxio master must discover the addition, removal, and updates to files and directions in under storage, and apply the changes in the Alluxio file system. This process to synchronize the two namespaces is called metadata sync. How to Trigger Metadata Sync When applications make changes to the metadata of an Alluxio file, and this file is persisted, the change will always be propagated to under storage synchronously, and there is no need to trigger metadata sync. When applications update under storage files without letting Alluxio know, there are two approaches to govern the timing of metadata sync. 1. Rely on a time-based automatic sync One can set a sync Interval as an Alluxio property key “ ”. alluxio.user.file.metadata.sync.interval When the value is -1 (current default value), Alluxio will never resync with under storage after initial load. When its value is set to 0, Alluxio will always resync with under storage whenever metadata is accessed. When the value is positive (default unit is ms), Alluxio will (best effort) not re-sync a path within that time interval. Note that using this approach, if a path in Alluxio has never been accessed, it will never trigger a sync. Once the path is accessed after the interval of sync expires, Alluxio will sync with the under storage again. For example, in a Presto job, the query planning stage lists all the files necessary for the job, which will trigger a sync if those paths have not been accessed recently. However, subsequent stages of the job will not sync unless the job duration exceeds the sync interval. So we can technically resync more frequently than the sync interval in this case. This property key can be customized with a new global default value (when set in ) or on a directory basis recursively to apply all its children. alluxio-site.properties 2. Sync manually using LoadMetadata flags If sync metadata does not happen because of the sync interval, most Alluxio operations will continue to execute with the current metadata in the Alluxio file system with some exceptions: For most of the users, Alluxio CLI “ ” is the easiest way to trigger sync manually. For example, one can run “ ” to force update the metadata of Alluxio path “ “. loadMetadata bin/alluxio fs loadMetadata /path/to/sync /path/to/sync For applications built on Alluxio file system (Java), there are two API methods and which retrieve the metadata for a path or a directory. When calling these methods, there is an additional field of in the option for each call, which may trigger a “loadMetadata” process by the master on the Alluxio path being inquired. This process can be considered as a simplified version of the sync by only loading file metadata from the under storage but not modifying a file’s metadata if it is already in the Alluxio. if is set to , then nothing will be loaded, and if the file does not already exist, a FileNotFound exception will be thrown. When is , we will only load the metadata once for each directory. This affects these two file system calls only, and this option is only considered when sync does not happen. SDK getStatus listStatus LoadMetadataPType LoadMetadataPType NEVER LoadMetadataPType ONCE How is Metadata Sync Implemented Alluxio master potentially triggers metadata sync on an Alluxio path when the Alluxio master receives an RPC request to retrieve the metadata of this path. Rather than having a dedicated service to traverse the entire file system inode tree and keep it in sync, this effort is amortized by each individual Alluxio file system operation on master. The high-level process for the syncing inside the RPC request is: Given the Alluxio path, determine if it is not consistent with the corresponding under storage path.  This means the under storage path does not exist, or has metadata that differs from Alluxio. This part is done using the RPC thread. The steps from 1 should populate the sync queue, and we iterate through the sync queue and process each path in a worker thread from a separate thread pool. The order of traversal is a BFS order, because additional paths are added at the end of the queue. Concurrency and executors will be discussed in the parallelism section in more detail. This part is carried out by the sync threads, and under storage, information is read using the under storage prefetch thread. The reason for this is to overlap communication with computation. Sync threads need to manipulate inode tree, and under storage, prefetch can start as soon as we determine that we need the information at some point in the future. Prefetching threads load the under storage status information into the under storage status cache, which is discussed in the cache section. Note that the metadata sync process can be relatively expensive and block other operations if they touch the same part of the inode tree. This is because the sync process may write-lock the part of file system metadata it is updating. Particularly, when syncing a particular path in a tree, the RPC handling thread will first acquire the read lock on the entire path to the file. Because the syncing thread needs the ability to create paths as well, it needs a write lock for the sync root path. As the syncing thread processes each path under the root path, additional locks are acquired. A write lock to the file path is acquired by the syncing thread and released as soon as the path is processed. Performance Optimization Tuning parallelism One can tune the level of parallelism to sync metadata by controlling three configuration parameters: This indicates the number of individual files to sync concurrently within a single metadata sync request (e.g., on a directory). alluxio.master.metadata.sync.concurrency.level This indicates the number of concurrent threads for all sync operations alluxio.master.metadata.sync.executor.pool.size This indicates the number of concurrent threads that can perform under storage prefetch operations for all sync operations alluxio.master.metadata.sync.ufs.prefetch.pool.size Caching result There are three types of different caches, with different goals and uses in the metadata syncing process. Here is a quick summary of all of them. is a negative cache, used to avoid checking the under storage for those paths that have been known to not exist. It uses prefix matching to determine if a path is in the under storage or not. For example, if the path is in the absent cache, we know must not exist on the under storage either. In addition, Absent Cache entries have a timestamp attached. So that we know the time it was last checked in the under storage. This is useful when the sync interval is some time period, we use the timestamp to determine whether we need to recheck the under storage for the existence of the file or directory. AbsentCache /a/b /a/b/c is a cache used to prefetch under storage statuses during the sync process. Instead of fetching the path information when needed, we can often prefetch some file statuses while we process the current directory. UfsStatusCache is a positive cache and contains paths that have been synced with the under storage recently. When we receive a metadata operation, we will check with this cache to determine whether we need to sync a particular path. UfsSyncPathCache Summary Metadata synchronization is one of the most important features in Alluxio. There are multiple different ways to trigger synchronization with different performance tradeoffs. Inside Alluxio master, there is a list of optimizations applied to speed up the synchronization. More Info Documentation of Alluxio Unified Namespace Two Ways to Keep Files in Sync Between Alluxio and HDFS Also Published Here

Metadata Synchronization: Design, Implementation and Optimization

The web is thirty years old this month, counting from the date the . I first saw it emerge on August 6th, 1991, when the web was publicly announced. But after 30 years, I think we've lost sight of its original purpose. Here's what I mean. first web server was launched on Christmas Day 1990 In 1991 I was writing for magazine, which was a lot of fun because I got to work with, and write about, people who were doing fascinating things with computer technology. Working for led to writing the book and a short stint at Steve Jobs's as a technical writer. NeXTWORLD NeXTWORLD Taking the Next Step NeXT The most innovative project I encountered that year was not NeXT. Rather, it was something developed by Tim Berners-Lee, a researcher at CERN, the European particle physics laboratory. He used a NeXT computer to develop a networked information system for high-energy physicists. If I hadn't been reading the Usenet newsgroups carefully, I would have missed the announcement of "a hypertext editor for the NeXT." comp.sys.next Not many people used "WWWNeXTStepEditor version 0.12." The NeXT community was small, and many NeXT computer users did not yet have Internet access, which was required to access Tim Berners-Lee's experimental "WorldWideWeb" documents. To me, the name WorldWideWeb seemed a bit of a conceit since the only users at the time were high-energy physicists at a few research institutes. Nonetheless, as soon as I'd installed and tried the software, I wanted to get involved. I told everyone about it, including a fellow editor at , . Barlow, a lyricist for the Grateful Dead, and co-founder of the (then-new) Electronic Frontier Foundation, wrote to his friend to say, here's "Something called World Wide Web, which sounds rather like Project Xanadu emerging from the Matrix almost without design. This could be cool." NeXTWORLD John Perry Barlow Mitch Kapor If you don't recognize Barlow's references, was a visionary hypertext project that predated the Internet. The Matrix is not the movie universe created by the Wachowski's, but rather the network of computer networks that existed before the commercialization of the Internet (as described by the researcher ). Project Xanadu John Quarterman Subsequently, I joined the and also contacted Tim Berners-Lee directly. I wanted to know if he was interested in technology I'd heard about from friends at Adobe Systems. Adobe's editable PostScript specification (which came to be named PDF and became the basis for a product named "Acrobat") seemed to me a better match for distributed hypertext than the very-limited SGML-based "HTML tags" that Tim Berners-Lee was using. HTML offered only minimal layout and styling possibilities; was a better page description language. www-talk mailing list PostScript Here's an excerpt... Tim Berners-Lee was reluctant to publicize that code for the web was available for free, as management at CERN had not yet agreed to release it publicly (that didn't happen until ). Plus, the editors at were not interested in a product that was only useful to a few NeXT users with Internet access, so I wasn't able to write about it for the magazine (too bad, as I could have been the first journalist to write about the web!). April 1993 NeXTWORLD A year later, editable PostScript was released, and in 1993, of CERN visited me in California. I arranged a meeting between Cailliau and management at Adobe Systems. He met with John Kunze of Adobe at the Stanford Linear Accelerator Center (SLAC) and demonstrated the WorldWideWeb — the first time management at Adobe had seen the web. By that time, HTML was firmly entrenched as the native language of the new web. Adobe later invested in Netscape Communications and produced an Acrobat plug-in for the Netscape web browser, establishing the basis for viewing richly formatted PDF pages in web browsers. Robert Cailliau For some years afterward, I regretted that the Adobe/WWW meeting had not taken place the previous year - I imagined the richer PDF language might have eclipsed the sparse and limited HTML language as the primary syntax of web pages. With the adoption of the (from 1999 forward), my views changed. I've come to appreciate that the technical underpinnings of the web are optimal, reflecting both the genius of Berners-Lee's original vision and the rich contributions of the worldwide developer community. Cascading Style Sheets specification Thirty years later, it is easy to overlook the web's origins as a tool for sharing knowledge. Key to Tim Berners-Lee's vision were open standards that reflected his belief in the , a principle that choosing the simplest and least powerful language for a given purpose allows you to do more with the data stored in that language (thus, HTML is easier for humans or machines to interpret and analyze than PostScript). Along with open standards and the Rule of Least Power, Tim Berners-Lee wanted to make it easy for anyone to publish information in the form of web pages. His first web browser, named Nexus, was both a browser and editor. The NeXTSTEP SDK made editing easy to implement; editing was abandoned in subsequent non-NeXT web browsers. Rule of Least Power I think we've lost sight of that original vision. Facebook or Wix makes it easy to author web pages but at the cost of lock-in to proprietary systems. And when developers choose complex software such as React or Rails to build simple websites, I cringe at the violation of Rule of Least Power. I still have a "deep case of net religion" (as John Perry Barlow said in 1991), and I'm now writing about ways to build websites without frameworks and build tools (see my articles, ). My current open source project, , is a do-it-yourself website builder that uses new web standards (including web components) to help people build simple sites. Thirty years on, we still have a lot of work to do to realize the web's original goals. I hope I'm not the only one who sees this. Build Websites the Yax Way Yax.com I never anticipated that Berners-Lee's NeXT hypertext project would become a medium to supplant print and television. In 1991, during idle moments, I imagined the world's books and magazines might be reborn as hypertext (many others had imagined this earlier, including Ted Nelson, with his Xanadu project). But I never honestly believed that Berners-Lee's software project could change the world. By 1999, as web addresses appeared on billboards and the sides of buses, I started to accept that Berners-Lee naming his project "the WorldWide Web" was not vainglorious but merely prescient. Still, I hope its utility as a platform for commerce does not eclipse our original vision of the web as a means for sharing the world’s knowledge.

Stories, News & Mentions about SYNNEX

Synnex’s Stories on HackerNoon

Synnex's News & Mentions