We're solving one of the hardest problems on the internet - establishing trust between two parties

Trinsic

ZK is anticipated as one of the technologies that will undoubtedly change Web3. When it comes to ensuring the privacy, security and integrity of blockchain applications ZK looks quite promising. Trust is backbone of Blockchain and ZK would be helpful in securing user identity and data, so that they can authenticate themselves in a trustless world. In today’s world privacy is the main point of concern. An individual won’t want to reveal his complete identity to get a small job done, but at the same time the verifier also needs the information to confirm if the things are fine to proceed. The answer to this scenario is Zero Knowledge. Understanding Zero Knowledge with an example The classic example gives deeper insights about the ZK concept. But for beginners it might be confusing and unclear to get started with understanding the concept of ZK. So let’s discuss another interesting example. Ali Baba cave When you visit to a club, at the entry you are asked to show a proof that you are 18+ in age. Normally you share your ID card which contains your photo, date of birth and other private details like your residential address or your work place details. It’s not desired that just to verify an individual’s age all other private details would also be revealed. Now what’s the solution to this ? The answer is ZK proofs. In the above scenario, the individual can generate a ZK proof that he/she is 18+ in age which can be stored as a claim. Now any organisation or individual who wants to verify the claim can do so easily without having a need to look at ID card and check the date of birth. It’s a win-win situation for both the parties as the prover don’t need to reveal his ID card and the verifier can verify is the prover is 18+ in age or not. Now, for the online world a user has his/her identity too. Many platforms demand gmail authentication and unnecessary access to mails and other details. Same is the case with web3 platforms, once the user connects his wallet to the platform, indirectly his entire on-chain data is revealed to that platform via his blockchain wallet address. Existing ZK and Identity Solutions and Tools Polygon ID is the first ever to be powered by zero-knowledge (ZK) cryptography, a privacy and blockchain scaling technology. It provides Blockchain-based ID for decentralized and self-sovereign models backed by zero-knowledge native protocols for ultimate user privacy. Polygon ID trinsic You can leverage ZK proofs using t platform to build your end-to-end identity solution. Trinsic’s API makes it easy to build identity products that helps users manage their data and identity. rinsic iden3 is a next-generation private access control based on self-sovereign identity, designed for decentralised and trust-minimised environments. The main idea of the iden3 protocol is that each identity is self-soverign and can issue claims on another identity. The iden3 protocol is based on circuits which is the backbone of the protocol. iden3 Circom Semaphore consists of a smart contract and zero-knowledge proof components which work in tandem. The on-chain proof verification and identity state is maintained and handled by smart contract. The zero-knowledge components work off-chain to create identities, manage groups and generate proofs. Semaphore Conclusion The above discussion would have established a curiosity and a fundamental understanding of zero knowledge proofs and related tools and platforms built over it. Find me on following socials, I would love to hear from you! LinkedIn : https://www.linkedin.com/in/rishikeshkale Twitter : Rishikesh Kale (@0xrishikesh) / Twitter Also published here.

An Intro to Zero-Knowledge Proofs (ZKP) and Identity

Imagine you're developing on your laptop, using Docker to containerize your applications for consistency and ease of deployment. Your current project involves using MinIO for object storage, and you've set it up beautifully in Docker. But here's the twist: your workflow requires MinIO to interact with a Flask application running on your localhost for processing events and executing functions (referred to as and in MinIO's context). Event Notifications Object Lambdas One of the common hurdles you might encounter in this setup is a networking-related error, typically manifesting as “ error (flask-notification:webhook): dial tcp 127.0.0.1:5000”…. This error is indicative of a fundamental networking misalignment between the Docker container where MinIO is running and the Flask application hosted on your localhost. Essentially, when MinIO inside Docker tries to send event notifications to the Flask app, it attempts to connect to localhost:5000. However, within the Docker context, localhost refers to the Docker container itself, not the host machine where your Flask app is running. This results in MinIO attempting to establish a connection with a service on localhost:5000 inside its container, where no such service exists, leading to the connection being refused. This scenario underscores the need for a clear understanding of Docker networking, particularly how localhost is interpreted differently inside a Docker container. This guide will subsequently delve into effective strategies to realign this communication, ensuring that MinIO inside Docker can successfully send event notifications to the Flask app running on your host machine. By addressing this challenge, we create a development environment that closely simulates a production setup, improving the reliability and effectiveness of your workflow. This walkthrough will guide you through the process of setting up this environment. We'll start by setting up MinIO within Docker, ensuring it's configured correctly to emit events. Then, we'll move on to the Flask application, designed to respond to these events. The crucial part of this guide will focus on enabling communication between the MinIO container and the Flask app running on your localhost. We'll tackle common hurdles such as Docker networking, address resolution and data consistency. By the end of this guide, you'll have a functional local development setup where , mimicking a production-like environment on your laptop. This scenario is not just about making things work; it's about creating a development environment that's as close to production as possible, enhancing the reliability of your development and testing processes. MinIO events trigger actions in your Flask app Understanding Docker Networking in Local Dev Strategies As we progress towards integrating our Flask application, it's imperative to first comprehend a crucial aspect of Docker networking - the communication dynamics between Docker containers and the host machine. This knowledge is particularly indispensable for developers orchestrating local development environments, such as on laptops, where there is a need for Docker containers and host services, like our Flask application, to interact without hiccups. The term ' ' within the context of a Docker container has a unique implication. Contrary to its usual reference to the host machine, inside a Docker container, 'localhost' points to the container itself. This distinction is of paramount importance, especially when Dockerized services, such as MinIO, attempt to reach services on the host machine. localhost To illustrate this point, let's consider an error scenario as shown in the following image: When MinIO, running inside a Docker container, fails to connect to the localhost – in this case, the Flask app on the host. The connection error we encountered in our Docker-MinIO screenshot, characterized by the message , initially appears complex. However, a closer analysis reveals its underlying cause. The segment flask-notification:webhook identifies the source of the issue – it's MinIO’s mechanism to interact with our Flask application. The phrase signifies MinIO's effort to establish a TCP connection with the Flask app, presumably located at . The critical part, , indicates that MinIO is unable to access the Flask app at the specified address, underscoring a fundamental challenge in containerized networking. (flask-notification:webhook): dial tcp 127.0.0.1:5000: connect: connection refused dial tcp 127.0.0.1:5000 localhost:5000 connection refused To correct this, a clear and direct communication pathway between MinIO and Flask is essential. For scenarios where Flask is hosted on the same machine but external to Docker, the solution involves configuring MinIO to connect to . This special DNS address acts like a direct line, enabling MinIO within its container to reach out to the Flask application on the host. host.docker.internal:5000 This offers a solution for Mac and Windows users through , a DNS name that allows containers to correctly reference the host, thus facilitating the desired interaction between containerized services and host-based applications. Understanding and utilizing this Docker feature is critical for ensuring that local development setups mirror production environments as closely as possible, thereby enhancing development reliability and efficiency. host.docker.internal Overview of Assets The foundation of our guide lies in the prepared assets available at the in the directory. Here I have provided the source code for anything discussed in this guide, at the heart of this repository, you'll find: GitHub minio/blog-assets repository docker_to_localhost This contains the relevant commands to preform the actions taken in Smooth Sailing from Docker to Localhost(this article). README File: README.md : This file constitutes the Flask application, serving as a crucial component in our setup. It is tailor-made to interact efficiently with the MinIO service. Flask App Files main.py : This is the blueprints for creating your Docker container. It defines the environment in which your Flask applications will be built and ran, ensuring consistency across different setups. Docker File for Flask App dockerfile : Acting as the orchestrator, this define how multiple containers (like MinIO and Flask) coexist and communicate within your Docker environment. Docker Compose File docker-compose.yaml Deploying Flask App for MinIO Event Handling In the context of integrating a Flask application with a Dockerized MinIO service, let's delve into the configuration of the Flask app. Specifically, I've set up a Flask endpoint with an identifier of that is intended to serve as the receiver for MinIO's webhook events via , as highlighted in the following screenshot: “flask-minio-event” http://host.docker.internal:5000/minio_event Below is the code for our demonstration Flask App that runs on our local machine and serves out a route of on : /minio_event port 5000 from flask import Flask, request, jsonify
import logging
import os
from dotenv import load_dotenv

app = Flask(__name__)

load_dotenv()
logging.basicConfig(level=logging.INFO)
I
@app.route('/minio_event', methods=['POST'])
def log_bucket_event():
    """
    Logs events received from MinIO to the Python logger.
    """
    event_data = request.json
    logging.info(f"Event received: {event_data}")
    return jsonify({'message': 'Event logged successfully'})

if __name__ == '__main__':
    port = int(os.getenv('FLASK_PORT', 5000))
    debug_mode = os.getenv('FLASK_DEBUG', 'False') == 'True'
    app.run(host='0.0.0.0', port=port, debug=debug_mode) In the previous section, we explored how Docker containers interpret 'localhost', which is a significant consideration in ensuring that the Flask app correctly receives notifications from a Dockerized MinIO instance. This Flask application in more specific detail, is configured to receive and log events from MinIO. The endpoint /minio_event is set up to listen for POST requests, which are expected to be the event notifications from MinIO. Note that the application runs on port 5000 and is configured to be accessible from all network interfaces (0.0.0.0), which is crucial for receiving requests from MinIO running inside a Docker container. Deploying MinIO Container with Docker Networking Deploying the MinIO container to interact with localhost services like a Flask API requires a specific Docker network configuration. This setup is crucial for enabling seamless communication between the container and your local machine, particularly for local development or testing scenarios. To achieve this, the MinIO container must be configured to use the host's network. This approach allows the container to directly interact with services running on the host machine, such as a Flask application. It's especially beneficial in development environments where the Flask app needs to access or store data in MinIO. The command for this configuration is straightforward yet powerful: docker run --name minio --network="host" minio/minio server /data Using the option removes the complexities of Docker's default network bridging and port forwarding. It's a simpler, more direct way to ensure that the MinIO container can access any service running on localhost, not just limited to the Flask app. This makes the MinIO container versatile for a variety of local services, streamlining the development and testing processes. --network="host" Deploying MinIO and Flask App Containers with Docker-Compose You can use Docker networking features, such as a custom network or Docker Compose, to facilitate communication between containers. By placing both containers on the same network, they can communicate using their container names as hostnames. Example using Docker Compose to facilitate the communication between a MinIO container named minio, and a Flask app container named flaskapp specified below as services: version: '3.8'
services:
  minio:
    image: minio/minio
    command: server /data
    ports:
      - "9000:9000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - mynetwork

  flaskapp:
    image: python:3.9 # Use an official Python image
    command: >
      sh -c "pip install Flask
      && FLASK_APP=main.py FLASK_RUN_HOST=0.0.0.0 flask run" # Install Flask and run the app
    volumes:
      - ./app:/app # Mount your Flask app directory
    working_dir: /app # Set working directory to your app directory
    ports:
      - "5000:5000"
    depends_on:
      - minio
    networks:
      - mynetwork

networks:
  mynetwork:
    driver: bridge This setup maps host.docker.internal to the host machine's IP address, allowing containers to access services on the host by adding the extra_hosts parameter. This setup allows MinIO to recognize host.docker.internal as the host machine, bridging the container-to-host communication gap. It's a smart workaround for a common Docker challenge, ensuring that MinIO can connect to services on the host, such as our Flask app. Additionally by including the Flask installation in the docker-compose file we will make the app listen on 0.0.0.0. This change is vital for accessibility from other containers and the host, broadening the communication scope of our Flask app. It’s an essential step for setups where Flask might also be containerized. The depends_on directive in the Flask service ensures an orderly startup sequence. By starting the Flask app only after MinIO is operational, we avoid potential timing issues. This arrangement mimics production environments where service dependencies are carefully managed, adding robustness to our local development setup. These examples are tailored for a Dockerized MinIO setup, ensuring effective communication with services running on the host. Be mindful of the security implications, especially when adjusting network settings or exposing ports, and choose the solution that aligns with your security and architectural requirements. Both methods will allow the MinIO service to communicate with your Flask application, depending on whether the Flask app is running on the host or inside another Docker container. Seamless Integration in a Dockerized World We've journeyed through the nuances of Docker networking, unraveling how to efficiently bridge Docker containers with localhost environments so a Dockerized MinIO service can effectively communicate with a Flask application running on your host machine. These insights are crucial for developers looking to harness the power of Docker without compromising on the flexibility and convenience of local development. Understanding Docker networking is more than a technical necessity; it's a step toward mastering containerized environments. The ability to set up and manage these interactions is a requirement in today's cloud-native development landscape. Whether you're developing on a laptop or deploying on a global scale, these skills ensure your applications remain robust, scalable, and secure. It's important to understand and implement best practices when developing cloud-native software. The Docker ecosystem is ever-evolving. You can quickly tap into the latest and greatest cloud-native technologies with an architecture based on containers and object storage. Stay curious and keep exploring. MinIO enables this exploration. Containerized, Kubernetes-native and S3 API compatible, MinIO frees you to write code that runs anywhere, consistently and reliably. Good luck on your journey with Docker and MinIO. May your development be robust, your solutions secure, and your learning continuous. If you have any questions we’re here to help, at or by joining the . hello@min.io Slack community

Smooth Sailing: Transitioning from Docker to Localhost

স্মুথ সেলিং: ডকার থেকে লোকালহোস্টে রূপান্তর

Reibungsloses Segeln: Übergang von Docker zu Localhost

Navegación tranquila: transición de Docker a Localhost

Navigation fluide : transition de Docker vers Localhost

सहज सेलिंग: डॉकर से लोकलहोस्ट में संक्रमण

順風満帆: Docker から Localhost への移行

원활한 항해: Docker에서 Localhost로 전환

Navegação suave: transição do Docker para o Localhost

Плавный ход: переход с Docker на Localhost

Sorunsuz Yolculuk: Docker'dan Localhost'a Geçiş

Thuận buồm xuôi gió: Chuyển đổi từ Docker sang Localhost

顺利进行：从 Docker 过渡到 Localhost

The job market is ever-evolving, shaped by technological advancements, economic shifts, and global trends. Analyzing the current job market reveals a dynamic landscape. Industries such as technology, healthcare, and sustainable energy are witnessing significant growth, offering many opportunities. Conversely, traditional sectors might be experiencing a decline. Understanding these trends provides a foundation for career change strategies in 2024. As we enter 2024, understanding the current job market trends is crucial for those considering a career change. I will share some tips on successfully transitioning to a new industry or role here. Analyzing Job Market Trends for Industry Transition Guide Self-Assessment Before Transitioning Before embarking on a career change journey, conducting a self-assessment is crucial. Running a personal skills and interests audit involves introspection and a realistic evaluation of one's strengths, weaknesses, passions, and skills. This self-awareness serves as a compass, guiding individuals toward cross-industry skill transfer. Reskilling for Career Shifts Technology and automation are across industries. While they bring forth efficiency and innovation, they also alter the skills landscape. Jobs involving creativity, critical thinking, and emotional intelligence are gaining prominence, highlighting the need for individuals to adapt and acquire relevant skills to stay competitive in the job market. reshaping job roles Aligning personal values with career goals is fundamental for long-term satisfaction and success. Entering a new industry requires . The journey of developing new skills begins with a clear roadmap. In the digital age, numerous resources facilitate skill acquisition. acquiring new skills From online courses on platforms like LinkedIn Learning and Coursera to intensive boot camps and industry-recognized certifications, individuals can enhance their skill set. Leveraging Technology to Market Skills Transferable skills play a pivotal role in making a successful career transition. Understanding the is essential. Crafting resumes and cover letters that showcase transferable skills is an art. transferability of skills Tailoring applications to highlight relevant experiences and skills is necessary, increasing the chances of securing interviews in a new industry. is also an effective networking strategy for career pivot planning. Building a professional network requires strategic efforts. Social media is critical in professional networking in this digital age. Job seekers must leverage platforms like LinkedIn to expand one's professional network and create meaningful connections in the new industry. Networking Building a Personal Brand for the New Industry is essential when transitioning to a new industry. The strategy requires creating strategies for updating online profiles, crafting personal branding statements, and communicating the value of previous experience in the context of the new industry. Rebranding oneself An online presence is often the first impression employers have. Job seekers must update their profiles on LinkedIn, personal websites, and other online platforms to align with the new industry practices. Effectively communicating the value of previous experience is a crucial aspect of rebranding. Individuals must know how to articulate their skills and achievements in a way that resonates with the requirements of the new industry. Acquiring temporary positions, internships, and volunteering opportunities can be valuable for gaining industry experience. To ensure that you receive maximum benefits, it is essential to identify relevant options that can help you gain practical experience and build credibility in your chosen field. These opportunities can provide insights into the day-to-day workings of the industry. Volunteering and seeking out can facilitate a smoother transition to a new career path. mentors Navigating the Emotional Challenges of Career Changes Career change often comes with emotional challenges. It is essential to address the psychological aspects of transition, creating strategies for dealing with uncertainty and fear of change and maintaining a positive mindset throughout the journey. Uncertainty is a common emotion during a . You should know effective coping strategies for navigating uncertainties and confidently making informed decisions. A positive mindset is a crucial driver of success. career transition Maintaining optimism, resilience, and a forward-looking attitude is essential, even in the face of challenges encountered during a career change. Interviewing Techniques for Career Changers Interviewing for a new industry requires specific preparation. During the interview, you should explain the reasons for a career transition to potential employers. During interviews, an essential part is framing the career change as a strength rather than a weakness. You must communicate transferable skills and highlight relevant experiences to project your image as valuable assets to potential employers. Effectively explaining the reasons for a career transition is crucial. It would be best if you communicated the decision to switch industries in a way that emphasizes growth, adaptability, and a genuine passion for the new field. Planning for Financial Stability During Career Transitions Embarking on a career change may come with financial implications. You must budget for potential gaps in employment and address the financial considerations of retraining and potentially starting in a new industry at a different position or salary level. Planning for potential gaps in employment is a prudent financial strategy. It is essential to gain skills related to budgeting, saving, and creating a financial safety net to navigate transition periods. Retraining for a new career often requires a financial investment. It is essential to carefully consider the financial implications of pursuing additional education or certifications. To navigate a career change in 2024, taking a multifaceted approach that includes self-assessment, upskilling, networking, and strategic planning is essential. By implementing these strategies, individuals can successfully transition into a new industry and thrive in the changing job market. Remember, a career change is not just a shift in employment; it presents an opportunity for personal and professional growth. About the Author is a marketing and growth expert with extensive experience scaling venture-backed hyper-growth startups such as Tynker, Roku, IMVU, Texture, and TrustedID. He is also the author of " ," a book that is part of the bestselling "The Lean Startup" series by Eric Ries. Lomit Patel Lean AI

Your Career Path: A Step-by-Step Guide to Navigating Industry and Role Transitions in 2024

আপনার কর্মজীবনের পথ: 2024 সালে নেভিগেট শিল্প এবং ভূমিকা পরিবর্তনের জন্য একটি ধাপে ধাপে নির্দেশিকা

Ihr Karriereweg: Eine Schritt-für-Schritt-Anleitung zur Bewältigung von Branchen- und Rollenwechseln im Jahr 2024

Su trayectoria profesional: una guía paso a paso para navegar por la industria y las transiciones de roles en 2024

Votre cheminement de carrière : un guide étape par étape pour naviguer dans les transitions d'industrie et de rôle en 2024

आपका कैरियर पथ: 2024 में उद्योग और भूमिका परिवर्तन को नेविगेट करने के लिए एक चरण-दर-चरण मार्गदर्शिका

あなたのキャリアパス: 2024 年の業界と役割の変遷をナビゲートするためのステップバイステップ ガイド

진로: 2024년 업계 및 역할 전환을 위한 단계별 가이드

Seu plano de carreira: um guia passo a passo para navegar na indústria e nas transições de funções em 2024

Ваш карьерный путь: пошаговое руководство по смене отраслей и ролей в 2024 году

Kariyer Yolunuz: 2024'te Endüstri ve Rol Geçişlerinde Yön Bulmanıza Yönelik Adım Adım Kılavuz

Con đường sự nghiệp của bạn: Hướng dẫn từng bước để định hướng chuyển đổi ngành và vai trò vào năm 2024

您的职业道路：2024 年行业和角色转变的分步指南

Introduction As Kubernetes continues to mature as the leading container orchestration platform, security remains a paramount concern for organizations deploying containerized applications. Among the fundamental tools in Kubernetes' security arsenal, once played a pivotal role. PSP allowed administrators to meticulously define and enforce security constraints on Pods, ensuring that only containers meeting specific security standards could operate within the cluster. However, the landscape of Kubernetes security is evolving rapidly, and it's important to note that PSP has now been deprecated, starting with Kubernetes 1.25 (see link to Kubernetes 1.25 Urgent Release notes). This change stems from various factors, including the complexity of managing and maintaining PSP policies, often leading to operational challenges for users. In response to the deprecation, organizations are now looking for modern alternatives to PSP, ones that not only address security requirements but also streamline the process of securing workloads in . In this comprehensive guide, we embark on a journey to navigate this shift in Kubernetes security practices, focusing on the transition to Pod Security Admission (PSA). PSA is one of the most robust alternatives available, and this article explores it in-depth. However, it's worth mentioning that two other alternatives, Kyverno and OPA Gatekeeper, will be covered in separate articles within this series. Throughout this article, you will find detailed instructions for setting up and installing PSA, step-by-step migration guides to transition from PSP to PSA smoothly, and precise commands for transferring existing PSP rules to PSA. Additionally, you will gain the knowledge to assess namespaces for migration readiness using dry-run commands tailored to PSA. By the end of this guide, you will gain a comprehensive understanding of the strengths and limitations of PSA, equipping you to make an informed decision based on your organization's specific security requirements and Kubernetes environment. With this guide, you can confidently modernize your Kubernetes security practices, ensuring that your workloads remain protected as you bid farewell to the era of Pod Security Policies. Pod Security Policies (PSP) Kubernetes Pod Security Admission (PSA) PSA ensures containers adhere to Kubernetes' native security standards, which are defined by the Pod Security Standards. These standards classify security policies into three distinct profiles, each with a different level of restrictiveness: PSA Enforces Kubernetes Security Standards: : The Privileged policy is intentionally open and entirely unrestricted. Typically, this policy targets system- and infrastructure-level workloads managed by trusted users. It is characterized by an absence of restrictions. While allow-by-default mechanisms like Gatekeeper may inherently be Privileged, for deny-by-default mechanisms such as Pod Security Policy (PSP), the Privileged policy should deactivate all restrictions. Privileged : The Baseline policy is designed for easy adoption by common containerized workloads while preventing known privilege escalations. It caters to application operators and developers of non-critical applications. Baseline : The Restricted policy is focused on enforcing rigorous Pod hardening best practices, albeit at the potential expense of some compatibility. It primarily targets operators and developers of security-critical applications, as well as lower-trust users. For a comprehensive list of controls that should be enforced or disallowed under each profile, you can refer to the official documentation. Restricted Unlike some other solutions, PSA does not offer a straightforward method for directly migrating or modifying Pod Security Policy (PSP) rules. PSA's primary focus is on validating pods against the established , including the profiles mentioned above. No Direct PSP Rule Transfer: Kubernetes security standards : While PSA is effective at validation, it cannot modify or customize pod specifications as PSP could. PSA's main purpose is to enforce predefined Kubernetes security standards and does not include features for altering or mutating pod specifications. It primarily focuses on validating pods against these established standards. No Mutations Setup and Installation Since PSA is a Kubernetes Native component, to make it work, you just need to ensure that the Pod Security Admission (PSA) controller is enabled in your Kubernetes cluster. You can do that by running  the following command: kubectl api-versions | grep admission Prep Steps for Migration Assess Namespace Permissions Control of Pod Security Admission is influenced by namespace labels. This implies that individuals with the capability to update, patch, or create namespaces also have the authority to modify the Pod Security settings for those namespaces. This modification could potentially bypass stricter security policies. Before proceeding, it is essential to verify that namespace permissions are assigned exclusively to trusted and privileged users. It is advisable to refrain from granting these elevated permissions to users who do not require such access. If additional constraints are needed for configuring Pod Security labels on Namespace objects, consider utilizing an admission webhook to enforce those restrictions. Streamline and Normalize PodSecurityPolicies Before migrating to Pod Security Admission (PSA), it is beneficial to normalize your PodSecurityPolicies (PSP): : Eliminate options not covered by the Pod Security Standards. These options include: Remove Unsupported Fields .spec.allowedHostPaths

.spec.allowedFlexVolumes

.spec.allowedCSIDrivers

.spec.forbiddenSysctls

.spec.runtimeClass : Initiate the process by removing fields that solely have a mutating effect and do not impact the validation policy. These fields, as also mentioned in the "Mapping PodSecurityPolicies to Pod Security Standards" reference, include: Eliminate Purely Mutating Fields .spec.defaultAllowPrivilegeEscalation
.spec.runtimeClass.defaultRuntimeClassName
.metadata.annotations['seccomp.security.alpha.kubernetes.io/defaultProfileName']
.metadata.annotations['apparmor.security.beta.kubernetes.io/defaultProfileName']
.spec.defaultAddCapabilities Important Note: Removing these fields may lead to workloads lacking necessary configurations, which could potentially cause operational issues. It is crucial to ensure that workloads can function correctly with the simplified policies. Determine the Suitable Pod Security Level When determining the appropriate Pod Security level for your namespace, you have several methods to consider: By Security Requirements for the Namespace If you are familiar with the expected access level and security requirements for the namespace, you can select an appropriate Pod Security level based on those specific requirements. This approach is similar to how you would approach security settings in a new cluster. By Existing PodSecurityPolicies: Utilize the reference to map each of your existing Pod Security Policies (PSPs) to a corresponding Pod Security Standard level. If your PSPs are not originally based on the Pod Security Standards, you may need to make a decision. Choose a Pod Security level that is at least as permissive as your PSPs, or opt for a level that is at least as restrictive. To identify which PSPs are in use for pods within a specific namespace, use the following command: "Mapping PodSecurityPolicies to Pod Security Standards" kubectl get pods -n $NAMESPACE -o jsonpath="{.items[*].metadata.annotations.kubernetes\.io\/psp}" | tr " " "\n" | sort -u By Existing Pods: Conduct a dry run and apply the label command from the next section to test both the Baseline and Restricted Pod Security levels. This helps you assess whether these levels are sufficiently permissive for your existing workloads. Choose the least-privileged valid level that aligns with your existing workloads. It's important to note that the mentioned options are based on existing pods, which may not account for workloads that are not currently running. This includes CronJobs, scale-to-zero workloads, or other workloads that have not yet been deployed. Testing the Pod Security Level Once you've determined the Pod Security level for your namespace (except for the Privileged level), it's important to validate the chosen policy. Pod Security offers testing options to ensure a smooth rollout and compliance with security standards. Dry Run Testing: Use this method to assess the impact of the selected policy without immediate enforcement. To perform a dry run, use the following command, which will highlight any existing pods that do not meet the specified policy level: kubectl label --overwrite ns $NAMESPACE pod-security.kubernetes.io/enforce=$LEVEL Audit Mode Testing: Audit mode allows you to record policy violations without enforcing them. Violating pods are logged in the audit records for later review. Enable audit mode with the command: kubectl label --overwrite ns $NAMESPACE pod-security.kubernetes.io/audit=$LEVEL If unexpected policy violations arise during testing, you can: Update non-compliant workloads to align with the policy. Adjust the Pod Security level for the namespace if necessary to accommodate your workload requirements. Enforcing the Pod Security Level Once you are confident that the selected Pod Security level is appropriate for your namespace, you can proceed to enforce it. This step ensures that the desired security standards are actively applied to your workloads within the namespace. To enforce the desired Pod Security level on the namespace, use the following command: kubectl label --overwrite ns $NAMESPACE pod-security.kubernetes.io/enforce=$LEVEL Executing this command will activate and enforce the specified Pod Security level, enhancing the security posture of your namespace. Bypassing PSP To effectively bypass PodSecurityPolicy at the namespace level, you can bind a fully privileged Pod Security Policy (PSP) to all service accounts in the namespace. This process ensures that pods within the namespace are no longer subjected to modifications or restrictions imposed by PodSecurityPolicy. You can do that at a cluster level or at an individual namespace level. Cluster-Scoped Setup (Needed Only Once for the whole cluster): Create a fully privileged Pod Security Policy (PSP) by applying a YAML configuration file, such as . This PSP should grant all necessary privileges to pods and create a cluster role named privileged-psp to allow the use of Pod Security Policies (PSPs) and associate it with the privileged PSP: privileged-psp.yaml kubectl apply -f privileged-psp.yaml
kubectl create clusterrole privileged-psp --verb use --resource podsecuritypolicies.policy --resource-name privileged Per-Namespace Disablement: Create a role binding in the target namespace to associate the privileged-psp cluster role with the group. This binding effectively grants all service accounts within the namespace access to the fully privileged PSP, bypassing PodSecurityPolicy: system:serviceaccounts:$NAMESPACE kubectl create -n $NAMESPACE rolebinding disable-psp --clusterrole privileged-psp --group system:serviceaccounts:$NAMESPACE With this setup, the privileged PSP is non-mutating, and the PodSecurityPolicy admission controller always prioritizes non-mutating PSPs. As a result, pods in this namespace will no longer be modified or restricted by PodSecurityPolicy. One advantage of this approach is its reversibility. If any issues arise, you can easily roll back the change by deleting the RoleBinding associated with disabling PodSecurityPolicy. Ensure that pre-existing Pod Security Policies remain in place during this process. Undo PodSecurityPolicy Disablement: To revert the PodSecurityPolicy disablement for the namespace, simply delete the RoleBinding created earlier: kubectl delete -n $NAMESPACE rolebinding disable-psp Revisit Namespace Creation Processes With existing namespaces updated to enforce Pod Security Admission, it's essential to review and update your processes and policies for creating new namespaces. This ensures that new namespaces are configured with an appropriate Pod Security profile from the outset. Consider the following steps to enhance namespace creation processes: : Update your organization's policies and procedures for creating new namespaces to include the selection and application of the desired Pod Security level. Ensure that security standards are established right from the creation stage. Adjust Namespace Creation Policies You can statically configure the Pod Security admission controller to define default enforcement, audit, and/or warning levels for unlabeled namespaces. This approach ensures that namespaces lacking explicit Pod Security labels still adhere to your specified security standards by default. Static Configuration: By revisiting your namespace creation processes, you can seamlessly integrate Pod Security standards into y and maintain a consistent and secure approach across all namespaces, old and new. our Kubernetes environment Disable PodSecurityPolicy Once you are confident that the PodSecurityPolicy (PSP) admission controller is no longer needed and that Pod Security Admission (PSA) has been successfully implemented and validated, you can proceed to disable the PSP admission controller. Here are the steps to disable the PSP admission controller: Modify kube-apiserver Configuration: Open the configuration file for your kube-apiserver, typically located at /etc/kubernetes/manifests/kube-apiserver.yaml. Add the flag to disable the PSP admission controller. Ensure that it's removed from the list of active admission plugins. --disable-admission-plugins kube-apiserver --disable-admission-plugins=PodSecurityPolicy Save the configuration file. Restart kube-apiserver: Restart the kube-apiserver to apply the changes. You can usually achieve this by restarting the Kubernetes control plane or the kube-apiserver pod itself. systemctl restart kubelet Verification: Ensure that the PSP admission controller is disabled by checking the kube-apiserver logs or its status. After sufficient "soak time" to be confident that you won't need to roll back to PSPs, proceed to the next step. Cleanup PSP Resources: With the PSP admission controller disabled and PSA in place, you can safely delete your existing PodSecurityPolicies, as well as any associated Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings. kubectl delete podsecuritypolicies --all
kubectl delete roles,clusterroles,rolebindings,clusterrolebindings --selector=rbac.authorization.kubernetes.io/autoupdate=true This cleanup step ensures that there are no residual PSP configurations in your cluster. By deactivating the PSP admission controller and eliminating PSP-related resources, you simplify your cluster's security architecture, completing the transition to Pod Security Admission. Summary In this comprehensive guide, we've explored the essential steps and considerations for smoothly transitioning your from Pod Security Policies (PSP) to Pod Security Admission (PSA). This migration ensures that your workloads continue to run securely while aligning with Kubernetes' evolving security standards. Kubernetes cluster's security framework Pros and Cons of using Pod Security Admission (PSA) Pros: : PSA is an integral part of Kubernetes, eliminating the need for third-party tool installations. It leverages the platform's built-in security features. Native Kubernetes Component : PSA aligns with Kubernetes' native security standards, ensuring compliance with the platform's best practices. Enforces Kubernetes Standards Cons: : PSA may not provide the same level of customization as PSP, especially for complex security policies that require mutation of pod specs. Limited Customization : Existing workloads must be updated to include security context settings, which may require modification of YAML files. Retrofitting Required This guide equips you with the knowledge and steps required to migrate confidently from PSP to PSA, ensuring a secure and efficient transition while safeguarding your Kubernetes workloads. Stay tuned for upcoming articles, where I will explore alternative migration paths to Kyverno and OPA Gatekeeper.

Migrating from Pod Security Policies: A Comprehensive Guide (Part 1: Transitioning to PSA)

পড নিরাপত্তা নীতি থেকে স্থানান্তর: একটি ব্যাপক নির্দেশিকা (পর্ব 1: PSA-তে রূপান্তর)

Migration von Pod-Sicherheitsrichtlinien: Ein umfassender Leitfaden (Teil 1: Übergang zu PSA)

Migración desde políticas de seguridad de pod: una guía completa (Parte 1: Transición a PSA)

Migration à partir des politiques de sécurité des pods : un guide complet (Partie 1 : Transition vers PSA)

पॉड सुरक्षा नीतियों से माइग्रेट करना: एक व्यापक मार्गदर्शिका (भाग 1: पीएसए में परिवर्तन)

ポッド セキュリティ ポリシーからの移行: 包括的なガイド (パート 1: PSA への移行)

포드 보안 정책에서 마이그레이션: 종합 가이드(1부: PSA로 전환)

Migração de políticas de segurança de pod: um guia abrangente (Parte 1: transição para PSA)

Миграция с политик безопасности Pod: подробное руководство (Часть 1: Переход на PSA)

Pod Güvenlik Politikalarından Geçiş: Kapsamlı Bir Kılavuz (Bölüm 1: PSA'ya Geçiş)

Di chuyển từ Chính sách bảo mật Pod: Hướng dẫn toàn diện (Phần 1: Chuyển sang PSA)

从 Pod 安全策略迁移：综合指南（第 1 部分：过渡到 PSA）

OpenAI, the world-renowned artificial intelligence research laboratory, made headlines when it announced its decision to transition from a nonprofit organization to a for-profit company in . From 2015 to 2018, the company allegedly remained non-profit. Today, the company has a staggering valuation of . 2019 $29 billion The move, which raised eyebrows and sparked debates, represented a seismic shift for the organization which has long been regarded as a bastion of development by those who made it and came amid growing demand for advanced AI technology. But as OpenAI seeks to balance its lofty ideals with the realities of the market, questions remain about how the company will navigate the complex landscape of AI development and ensure that its technology is used for the greater good. ethical and responsible AI In this story, we delve deep into OpenAI's transformation and explore the implications of its bold move. The Early Days of OpenAI: A Nonprofit Research Laboratory Once upon a time, in a galaxy far, far away...just kidding, it was in Silicon Valley, OpenAI was founded as a nonprofit research lab with a mission to save the world with artificial intelligence. Their plan? Use machine learning to smoke cancer, build self-driving cars, and end world hunger straight out of a lab in San Francisco. No biggie. As a nonprofit research laboratory, OpenAI focused on developing cutting-edge AI technologies and conducting groundbreaking research in the field. The pretext? The company announced they were throwing a patent party and everyone was invited! declaring that it would "freely collaborate" with other organizations and researchers by making its patents and research available to the general public. When the company by a group of high-profile entrepreneurs and researchers – among them were Sam Altman, Reid Hoffman, Jessica Livingston, Elon Musk, Ilya Sutskever, Peter Thiel, and Olivier Grabias, its mission was clear: started in 2015 , unconstrained by a need to generate a financial return.” “to advance artificial intelligence in a way that would benefit society as a whole A list of the "top researchers in the area" was reportedly created by Greg Brockman, AI researcher and co-founder of OpenAI, after a meeting with Yoshua Bengio, a Canadian computer scientist, who is allegedly one of the "founding fathers" of the deep learning movement, according to . In December 2015, Brockman was able to hire nine of them as the company’s first employees, and the promise and mission of OpenAI were said to be what attracted these researchers to the company. Wired Wojciech Zaremba, a former Intern at both Google and Facebook who later came to work at  OpenAI said he joined the company partly because of the people and the mission, even though OpenAI could not match the offers he was made somewhere else. These offers, he said, were two or three times his market value. According to him, “OpenAI was the best place to be." Because the company provided something more valuable, such as the opportunity to investigate future-focused research and eventually share the majority, if not all, of this study for free with anyone who requests it. https://twitter.com/sama/status/1610666441027772416?embedable=true OpenAI’s research director, Ilya Sutskever, who left Google to join the company, said about his transition from his former position at Google to OpenAI, "They did make it very compelling for me to stay, so it wasn't an easy decision, but in the end, I decided to go with OpenAI, partly because of the very strong group of people and, to a very large extent, because of its mission." The founding of the company was accompanied by a commitment of in funding from the founding members, which also included YC leader Paul Graham, as well as corporate sponsors like AWS and Infosys, with an aim to create a safety net that would allow researchers to explore the full potential of AI without compromising safety or ethics. $1 billion One of the first batches of AI software OPenAI released was, , a toolkit for building artificially intelligent systems by way of a technology called "reinforcement learning." Exactly what Brockman said the company will be starting with.  The same year, a software platform for assessing and honing an AI's general intelligence via a variety of games, websites, and other applications, was also introduced. OpenAI Gym "Universe," However, during this nascent stage of the company's ascendancy, Elon Musk left his board seat in 2018 on account of a plausible and prospective conflict of interest with Tesla's Artificial Intelligence (AI) research and development endeavors concerning autonomous driving, though he remained a donor to the company. But is it so? According to the most recent reports, there was more to the “conflict of interest” than we know, and we will soon find out. The Challenge of Funding a Nonprofit As a nonprofit organization, OpenAI faced the constant struggle of balancing its research goals with its budget constraints. The researchers had to learn how to stretch a dollar like a rubber band, while still managing to develop cutting-edge AI technologies that would impress their investors. The company has that an open nonprofit organization was the most effective way to accomplish its objectives. said before “Since our research is free from financial obligations, we can better focus on a positive human impact. We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible.” But, a few years later, they came to the conclusion that being an open and nonprofit organization would make it more difficult to carry out this purpose. So, In the epoch of 2019, OpenAI experienced a monumental inflection point. In a rather unorthodox move for a non-profit, OpenAI made an , a distinct entity that operates as a "capped-profit" corporation, which marked a transformative shift in its foundational structure. announcement of creating OpenAI LP Motivations Behind the Transition to For-Profit When OpenAI made the bold decision to transition to a for-profit model. Some say it was because they were tired of eating ramen noodles for every meal, while others argue it was because they wanted to make more money than Elon Musk. Regardless of the reason, it was a big change for the nonprofit research lab. In reality, this decision was motivated by a desire to increase financial sustainability and pursue commercial applications of its technology, and, apparently, this move was not without controversy, as some critics argued that it could compromise the organization's commitment to ethical AI development. First, the company that it needed to raise "billions of dollars" and pay out enormous signing bonuses to recruit top talent. claimed “We’ll need to invest billions of dollars in upcoming years into large-scale cloud compute, attracting and retaining talented people, and building AI supercomputers.” The company also thought it was also dangerous to keep OpenAI open. According to a Vox , the safety team at OpenAI came to the conclusion that making all of their work open-source might, rather than serving humanity's best interests, invite trouble. So, when they created GPT-2, they withheld it from the public due to concerns that it might be easily abused for plagiarism, bots, fake Amazon reviews, and spam. report But while the company went through this transition, it insisted that it would continue to pursue the same goals and set financial restrictions on both investors and employees. Despite the organization's new objective to amass a fortune, it remained cognizant of its moral compass and claimed that it was determined to refrain from any unethical practices. As a result, the idea was to establish OpenAI LP as a "capped-profit" firm, a combination of a for-profit and nonprofit organization. “We want to increase our ability to raise capital while still serving our mission, and no pre-existing legal structure we know of strikes the right balance.” The aforementioned "capped-profit" framework alludes to the limitation placed on first-round investors, whereby they are only authorized to receive a maximum return of 100 times their original investment. Balancing Profitability with Ethical Considerations The transition from nonprofit to for-profit required OpenAI to balance its desire for profit with its commitment to ethical AI development. It was kind of like trying to juggle while riding a unicycle, except with more existential questions about the nature of humanity. Information on the company’s stated that the lab operates under a non-profit governance structure ( The OpenAI LP will be “governed” by OpenAI Inc. which is the Nonprofit arm) that prioritizes the betterment of humanity over for-profit interests, which also allows it to emphasize safety concerns and undertake significant initiatives, such as sponsoring the most comprehensive Universal Basic Income (UBI) experiment, and the ability to cancel equity obligations to shareholders if necessary. website It said that the cap on the profits that its stockholders can receive prevents them from being enticed to try and grab wealth without limits and risk deploying something that could be very much harmful. Regardless, Elon Musk maintains his reservations. On March 15th, he took to Twitter, the social media platform which he now owns, to express his grievances. https://twitter.com/elonmusk/status/1636047019893481474?embedable=true Apparently, the meme billionaire invested  $100 million in the LLM giant, way before it became a for-profit company. But, while Elon seems to be concerned about the legal implications of OpenAI’s transition, his past show how much he is rather really concerned about the potential dangers of artificial intelligence. advocations While the Tesla boss cited that he left OpenAI simply because of a conflict of interest, a new report by show that Elon's real reason for leaving was that his proposal to take over the company and run it himself was rejected by both Sam Altman and other founders. The reason he proposed this idea was that he felt that the venture had significantly lagged behind Google and that he was the Messiah who could turn things around. Semafor Elon made it clear that his objective was to establish the AI lab as a competing force against Google. https://twitter.com/elonmusk/status/1626516035863212034?embedable=true With his leaving, he also held back from making a large intended donation. OpenAI's indicated that Musk would continue to provide financial support to the organization, but he did not follow through with the commitment. statement So, really, the conflict of interest was a power struggle that Elon lost? As per Semafor's report, it appears that only a small number of individuals at OpenAI shared the belief that Musk's departure was due to potential conflicts of interest. Furthermore, the speech that he gave during his departure from OpenAI's office, which primarily centered on this topic, did not receive favorable feedback from the majority of employees who remained skeptical of the explanation provided. Fortunately, after Elon’s departure, Microsoft $1 billion in OpenAI, gaining exclusive licensing rights to the company's GPT-3 , while also becoming its exclusive cloud provider. The investment was instrumental in allowing OpenAI to pursue further research endeavors. Through a collaborative effort between the AI lab and its new investor Microsoft, a supercomputer was built for training large-scale models, which ultimately led to the creation of ChatGPT and the image generator DALL-E. invested model a year later The Future of OpenAI The company's aim, , is to develop AI systems that have a positive impact on society, as well as to explore profitable commercial applications of its technology with no stated intentions to compete, but instead, seeks to collaborate with other research and policy institutions to promote safety in the final stages of AGI development. as it noted itself Nevertheless, the competition continues to intensify. While Elon, for example, might present a facade of not being an adversary, https://twitter.com/elonmusk/status/1639200036578885632?embedable=true In truth, he has actually joined the bandwagon of direct competitors of OpenAI. He is actively pursuing the of a new startup to square off against OpenAI. Obviously, Elon has not left the battle ring; therefore, the power struggle is still on. creation Despite Sam Altman's uncommon decision to forego equity in the new for-profit entity as a way to stay aligned with the original mission, Elon Musk remains suspicious of OpenAI's shift from an open to a closed organization. https://twitter.com/elonmusk/status/1599291104687374338?embedable=true Looking ahead, the company continues to push the boundaries of artificial intelligence through innovative research and the development of new technologies. Who knows what they'll come up with next? Maybe a robot that can fly humans to Mars or a machine that can predict the end of the world. Either way, it looks super interesting.

How OpenAI Transitioned from a Nonprofit to a $29B For-Profit Company

কিভাবে OpenAI একটি অলাভজনক থেকে $29B লাভের কোম্পানিতে রূপান্তরিত হয়েছে৷

Wie OpenAI von einer gemeinnützigen Organisation zu einem gewinnorientierten Unternehmen mit einem Umsatz von 29 Milliarden US-Dollar wurde

Cómo OpenAI hizo la transición de una organización sin fines de lucro a una empresa con fines de lucro de $ 29 mil millones

Comment OpenAI est passé d'une organisation à but non lucratif à une entreprise à but lucratif de 29 milliards de dollars

कैसे OpenAI एक गैर-लाभकारी संस्था से $29B की लाभकारी कंपनी में परिवर्तित हुआ

OpenAI が非営利団体から 290 億ドル規模の営利企業に移行した方法

OpenAI가 비영리에서 290억 달러 규모의 영리 기업으로 전환한 방법

Como a OpenAI fez a transição de uma organização sem fins lucrativos para uma empresa com fins lucrativos de US$ 29 bilhões

OpenAI Kâr Amacı Gütmeyen Bir Kuruluştan 29 Milyar Dolarlık Kâr Amaçlı Bir Şirkete Nasıl Geçti?

Cách OpenAI chuyển đổi từ một tổ chức phi lợi nhuận thành một công ty vì lợi nhuận trị giá 29 tỷ đô la

OpenAI 如何从非营利组织转变为 $29B 的营利性公司

Three years ago, on 25 November 2019, at 15:35 (UTC+1), the world ran out of IPv4 addresses....and nothing happened. RIPE NCC (the institution overseeing global internet resources) from the last remaining addresses in their available pool. have announced that they made a final IPv4 allocation This was supposed to be a big deal. It was the end of an era. Or so we thought. The truth is, the IPv4 address depletion was not nearly as catastrophic as many thought it would be. The birth of IPv6 The top-level exhaustion occurred as far as 31 January 2011. That's when the last /8 address blocks were allocated. So in June 2012, the world-leading Internet service providers (ISPs), home networking equipment manufacturers, and Web companies came together to save the Internet. The problem they were solving was the impending exhaustion of the IPv4 addresses. IPv4 has only 4.3 billion addresses. You can see how in the world of over 7.5 billion humans and countless internet-connected devices, each requiring a unique IPv4 address to connect to the Internet, that might be a problem. The solution the industry came up with was to switch to a new version of the Internet Protocol, IPv6, which has over 340 undecillion addresses, or 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses. (That's enough for every person to have addresses.) 47 octillion In an inspiring effort, the industry giants have committed to making sure that their products and services would work with both IPv4 and IPv6. This was no small feat, as it required a lot of coordination and hard work. And it will eventually pay off, with a future-proof Internet that can keep growing for many years to come. By 2019, all five Regional Internet Registries (RIRs) had run out of unallocated address space to hand out to ISPs, who would then distribute it to end users. This should have been the end of IPv4. Yet, 10 years later, we still haven't transitioned to IPv6. Why? I can think of several reasons why we haven't transitioned to IPv6 yet: CGNAT is working well enough that there is no immediate need to switch. IPv6 is not backward-compatible with IPv4. Reuse and reallocation of IPv4 addresses prolonged the time before we needed to switch CGNAT is working well enough that there is no incentive to switch. The solution that was supposed to be temporary, Carrier-Grade NAT (CGN) or Large-Scale NAT (LSN), is working so well that there is no immediate need to switch to IPv6. In 1994, when the problem of IPv4 exhaustion loomed on the horizon, a new, "temporary" technology was proposed to keep the Internet growing without depleting IP addresses as quickly. To keep the Internet running, a stopgap measure, carrier-grade was born. , Network Address Translation (NAT) NAT allows multiple devices on a network to share a single public IPv4 address. It takes a small pool of public IPv4 addresses and shares them among a large number of users on an internal network. It does this by translating the private addresses of devices on a network to a single public address and then back again when it receives data from the Internet. It's kind of like an apartment complex where everyone has their own apartment, but they share a single front door (a public IPv4 address). As a result, a single public IPv4 address can be shared among many devices on a local network. NAT was intended as a temporary fix before a better alternative - the later-developed IPv6 - could be implemented. However, this "temporary patch" has been so successful that this stopgap has become a semi-permanent solution lasting for over two decades and counting. IPv6 is not backward-compatible with IPv4 Many devices only work with IPv4, so a transition would require replacing a lot of hardware. So to make the transition from IPv4 to IPv6, everyone would need to upgrade their devices and software to be compatible with IPv6. Today, only about 40% of devices are IPv6-compatible. This presents a big challenge for companies that want to transition, as they need to support both IPv4 and IPv6 devices during the transition period. Source IPv6 adoption is also not equal worldwide, with some countries trailing behind. This creates a "digital divide" where some users can access certain content and services that others can't. This presents a challenge for companies trying to transition, as they need to support both IPv4 and IPv6 devices during the transition period. Source: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption Reuse and reallocation of IPv4 addresses Another reason for the delay in transitioning to IPv6 is that there are secondary markets for IPv4 addresses. IP address blocks turned into a commodity. For example, an ISP might have a /16 address block, which contains 65,536 addresses. But the ISP might only use a small fraction of those addresses - say, 1,000. The rest of the addresses in that block can be sold or leased to other companies. As a result, the same IPv4 address might be used by multiple companies at different times. ISPs can also reallocate their IPv4 addresses, for example, when a customer with a /24 address block (256 addresses) cancels their service. The ISP can then give that /24 address block to another customer. This address reuse and reallocation has helped to prolong the time before we need to switch to IPv6. The bottom line - it works well enough still Taken together, these factors - the difficulty of transitioning, the need to support both IPv4 and IPv6, and address reuse and reallocation - have all contributed to a delay in the transition to IPv6. So while IPv6 will eventually become the new standard, for now, IPv4 is here to stay. It's working well enough. Maybe our kids and grandkids will actually use IPv6 addresses.

Why We Are Not Likely to Transition to IPv6 Any Time Soon

Por qué no es probable que hagamos la transición a IPv6 en el corto plazo

Pourquoi nous ne sommes pas susceptibles de passer à IPv6 de si tôt

हम जल्द ही किसी भी समय IPv6 में परिवर्तन की संभावना क्यों नहीं रखते हैं

すぐに IPv6 に移行する可能性が低い理由

Por que provavelmente não faremos a transição para o IPv6 tão cedo

Tại sao chúng ta không có khả năng sớm chuyển đổi sang IPv6

为什么我们不太可能很快过渡到 IPv6

This article is co-written by me and my colleague Kai Dai. We are both data platform engineers at Tencent Music (NYSE: TME), a music streaming service provider with a whopping 800 million monthly active users. To drop the number here is not to brag but to give a hint of the sea of data that my poor coworkers and I have to deal with everyday. What We Use ClickHouse For The music library of Tencent Music contains data of all forms and types: recorded music, live music, audios, videos, etc. As data platform engineers, our job is to distill information from the data, based on which our teammates can make better decisions to support our users and musical partners. Specifically, we do all-round analysis of the songs, lyrics, melodies, albums, and artists, turn all this information into data assets, and pass them to our internal data users for inventory counting, user profiling, metrics analysis, and group targeting. We stored and processed most of our data in Tencent Data Warehouse (TDW), an offline data platform where we put the data into various tag and metric systems and then created flat tables centering each object (songs, artists, etc.). Then we imported the flat tables into ClickHouse for analysis and Elasticsearch for data searching and group targeting. After that, our data analysts used the data under the tags and metrics they needed to form datasets for different usage scenarios, during which they could create their own tags and metrics. The data processing pipeline looked like this: The Problems with ClickHouse When working with the above pipeline, we encountered a few difficulties: : Partial update of columns was not supported. Therefore, any latency from any one of the data sources could delay the creation of flat tables, and thus undermine data timeliness. Partial Update : Data under different tags and metrics was updated at different frequencies. As much as ClickHouse excelled in dealing with flat tables, it was a huge waste of storage resources to just pour all data into a flat table and partition it by day, not to mention the maintenance cost coming with it. High storage cost : Architecturally speaking, ClickHouse was characterized by the strong coupling of storage nodes and compute nodes. Its components were heavily interdependent, adding to the risks of cluster instability. Plus, for federated queries across ClickHouse and Elasticsearch, we had to take care of a huge amount of connection issues. That was just tedious. High maintainence cost Transition to Apache Doris , a real-time analytical database, boasts a few features that are exactly what we needed in solving our problems: Apache Doris : Doris supports a wide variety of data models, among which the Aggregate Model supports real-time partial update of columns. Building on this, we can directly ingest raw data into Doris and create flat tables there. The ingestion goes like this: Firstly, we use Spark to load data into Kafka; then, any incremental data will be updated to Doris and Elasticsearch via Flink. Meanwhile, Flink will pre-aggregate the data so as to release burden on Doris and Elasticsearch. Partial update : Doris supports multi-table join queries and federated queries across Hive, Iceberg, Hudi, MySQL, and Elasticsearch. This allows us to split the large flat tables into smaller ones and partition them by update frequency. The benefits of doing so include a relief of storage burden and an increase of query throughput. Storage cost : Doris is of simple architecture and is compatible with MySQL protocol. Deploying Doris only involves two processes (FE and BE) with no dependency on other systems, making it easy to operate and maintain. Also, Doris supports querying external ES data tables. It can easily interface with the metadata in ES and automatically map the table schema from ES so we can conduct queries on Elasticsearch data via Doris without grappling with complex connections. Maintenance cost What's more, Doris supports multiple data ingestion methods, including batch import from remote storage such as HDFS and S3, data reads from MySQL binlog and Kafka, and real-time data synchronization or batch import from MySQL, Oracle, and PostgreSQL. It ensures service availability and data reliability through a consistency protocol and is capable of auto debugging. This is great news for our operators and maintainers. Statistically speaking, these features have cut our storage cost by 42% and development cost by 40%. During our usage of Doris, we have received lots of support from the open source Apache Doris community and timely help from the SelectDB team, which is now running a commercial version of Apache Doris. Further Improvement to Serve Our Needs Introduce a Semantic Layer Speaking of the datasets, on the bright side, our data analysts are given the liberty of redefining and combining the tags and metrics at their convenience. But on the dark side, high heterogeneity of the tag and metric systems leads to more difficulty in their usage and management. Our solution is to introduce a semantic layer in our data processing pipeline. The semantic layer is where all the technical terms are translated into more comprehensible concepts for our internal data users. In other words, we are turning the tags and metrics into first-class citizens for data definement and management. Why would this help? For data analysts, all tags and metrics will be created and shared at the semantic layer so there will be less confusion and higher efficiency. For data users, they no longer need to create their own datasets or figure out which one is applicable for each scenario but can simply conduct queries on their specified tagset and metricset. Upgrade the Semantic Layer Explicitly defining the tags and metrics at the semantic layer was not enough. In order to build a standardized data processing system, our next goal was to ensure consistent definition of tags and metrics throughout the whole data processing pipeline. For this sake, we made the semantic layer the heart of our data management system: How does it work? All computing logics in TDW will be defined at the semantic layer in the form of a single tag or metric. The semantic layer receives logic queries from the application side, selects an engine accordingly, and generates SQL. Then it sends the SQL command to TDW for execution. Meanwhile, it might also send configuration and data ingestion tasks to Doris and decide which metrics and tags should be accelerated. In this way, we have made the tags and metrics more manageable. A fly in the ointment is that since each tag and metric is individually defined, we are struggling with automating the generation of a valid SQL statement for the queries. If you have any idea about this, you are more than welcome to talk to us. Give Full Play to Apache Doris As you can see, Apache Doris has played a pivotal role in our solution. Optimizing the usage of Doris can largely improve our overall data processing efficiency. So in this part, we are going to share with you what we do with Doris to accelerate data ingestion and queries and reduce costs. What We Want? Currently, we have 800+ tags and 1300+ metrics derived from the 80+ source tables in TDW. When importing data from TDW to Doris, we hope to achieve: : In addition to the traditional T+1 offline data ingestion, we require real-time tagging. Real-time availability : Each source table generates data through its own ETL task at various paces and involves only part of the tags and metrics, so we require the support for partial update of columns. Partial update : We need a response time of only a few seconds in group targeting, analysis and reporting scenarios. High performance : We hope to reduce costs as much as possible. Low costs What We Do? Generate Flat Tables in Flink Instead of TDW Generating flat tables in TDW has a few downsides: : TDW has to maintain an extra flat table apart from the discrete 80+ source tables. That's huge redundancy. High storage cost : Any delay in the source tables will be augmented and retard the whole data link. Low real-timeliness : To achieve real-timeliness would require extra development efforts and resources. High development cost On the contrary, generating flat tables in Doris is much easier and less expensive. The process is as follows: Use Spark to import new data into Kafka in an offline manner. Use Flink to consume Kafka data. Create a flat table via the primary key ID. Import the flat table into Doris. As is shown below, Flink has aggregated the five lines of data, of which "ID"=1, into one line in Doris, reducing the data writing pressure on Doris. This can largely reduce storage costs since TDW no long has to maintain two copies of data and KafKa only needs to store the new data pending for ingestion. What's more, we can add whatever ETL logic we want into Flink and reuse lots of development logic for offline and real-time data ingestion. Name the Columns Smartly As we mentioned, the Aggregate Model of Doris allows partial update of columns. Here we provide a simple introduction to other data models in Doris for your reference: : This is applicable for scenarios requiring primary key uniqueness. It only keeps the latest data of the same primary key ID. (As far as we know, the Apache Doris community is planning to include partial update of columns in the Unique Model, too.) Unique Model : This model stores all original data exactly as it is without any pre-aggregation or deduplication. Duplicate Model After determining the data model, we had to think about how to name the columns. Using the tags or metrics as column names was not a choice because: Ⅰ. Our internal data users might need to rename the metrics or tags, but Doris 1.1.3 does not support modification of column names. Ⅱ. Tags might be taken online and offline frequently. If that involves the adding and dropping of columns, it will be not only time-consuming but also detrimental to query performance. Instead, we do the following: For flexible renaming of tags and metrics, we use MySQL tables to store the metadata (name, globally unique ID, status, etc.). Any change to the names will only happen in the metadata but will not affect the table schema in Doris. For example, if a is given an ID of 4, it will be stored with the column name of a4 in Doris. Then if the is involved in a query, it will be converted to a4 in SQL. song_name song_name For the onlining and offlining of tags, we sort out the tags based on how frequently they are being used. The least used ones will be given an offline mark in their metadata. No new data will be put under the offline tags but the existing data under those tags will still be available. For real-time availability of newly added tags and metrics, we prebuild a few ID columns in Doris tables based on the mapping of name IDs. These reserved ID columns will be allocated to the newly added tags and metrics. Thus, we can avoid table schema change and the consequent overheads. Our experience shows that only 10 minutes after the tags and metrics are added, the data under them can be available. Noteworthily, the recently released Doris 1.2.0 supports Light Schema Change, which means that to add or remove columns, you only need to modify the metadata in FE. Also, you can rename the columns in data tables as long as you have enabled Light Schema Change for the tables. This is a big trouble saver for us. Optimize Date Writing Here are a few practices that have reduced our daily offline data ingestion time by 75% and our CUMU compaction score from 600+ to 100. Flink pre-aggregation: as is mentioned above. Auto-sizing of writing batch: To reduce Flink resource usage, we enable the data in one Kafka Topic to be written into various Doris tables and realize the automatic alteration of batch size based on the data amount. Optimization of Doris data writing: fine-tune the the sizes of tablets and buckets as well as the compaction parameters for each scenario: max_XXXX_compaction_thread
max_cumulative_compaction_num_singleton_deltas Optimization of the BE commit logic: conduct regular caching of BE lists, commit them to the BE nodes batch by batch, and use finer load balancing granularity. Use Dori-on-ES in Queries About 60% of our data queries involve group targeting. Group targeting is to find our target data by using a set of tags as filters. It poses a few requirements for our data processing architecture: Group targeting related to APP users can involve very complicated logic. That means the system must support hundreds of tags as filters simultaneously. Most group targeting scenarios only require the latest tag data. However, metric queries need to support historical data. Data users might need to perform further aggregated analysis of metric data after group targeting. Data users might also need to perform detailed queries on tags and metrics after group targeting. After consideration, we decided to adopt Doris-on-ES. Doris is where we store the metric data for each scenario as a partition table, while Elasticsearch stores all tag data. The Doris-on-ES solution combines the distributed query planning capability of Doris and the full-text search capability of Elasticsearch. The query pattern is as follows: SELECT tag, agg(metric) 
   FROM Doris 
   WHERE id in (select id from Es where tagFilter)
   GROUP BY tag As is shown, the ID data located in Elasticsearch will be used in the sub-query in Doris for metric analysis. In practice, we find that the query response time is related to the size of the target group. If the target group contains over one million objects, the query will take up to 60 seconds. If it is even larger, a timeout error might occur. After investigation, we identified our two biggest time wasters: I. When Doris BE pulls data from Elasticsearch (1024 lines at a time by default), for a target group of over one million objects, the network I/O overhead can be huge. II. After the data pulling, Doris BE needs to conduct Join operations with local metric tables via SHUFFLE/BROADCAST, which can cost a lot. Thus, we make the following optimizations: Add a query session variable that specifies whether to enable optimization. es_optimize In data writing into ES, add a BK column to store the bucket number after the primary key ID is hashed. The algorithm is the same as the bucketing algorithm in Doris (CRC32). Use Doris BE to generate a Bucket Join execution plan, dispatch the bucket number to BE ScanNode and push it down to ES. Use ES to compress the queried data; turn multiple data fetch into one and reduce network I/O overhead. Make sure that Doris BE only pulls the data of buckets related to the local metric tables and conducts local Join operations directly to avoid data shuffling between Doris BEs. As a result, we reduce the query response time for large group targeting from 60 seconds to a surprising 3.7 seconds. Community information shows that Doris is going to support inverted indexing since version 2.0.0, which is soon to be released. With this new version, we will be able to conduct full-text search on text types, equivalence or range filtering of texts, numbers, and datetime, and conveniently combine AND, OR, NOT logic in filtering since the inverted indexing supports array types. This new feature of Doris is expected to deliver 3~5 times better performance than Elasticsearch on the same task. Refine the Management of Data Doris' capability of cold and hot data separation provides the foundation of our cost reduction strategies in data processing. Based on the TTL mechanism of Doris, we only store data of the current year in Doris and put the historical data before that in TDW for lower storage cost. We vary the numbers of copies for different data partitions. For example, we set three copies for data of the recent three months, which is used frequently, one copy for data older than six months, and two copies for data in between. Doris supports turning hot data into cold data so we only store data of the past seven days in SSD and transfer data older than that to HDD for less expensive storage. Conclusion Thank you for scrolling all the way down here and finishing this long read. We've shared our cheers and tears, lessons learned, and a few practices that might be of some value to you during our transition from ClickHouse to Doris. We really appreciate the help from the Apache Doris community and the SelectDB team, but we might still be chasing them around for a while since we attempt to realize auto-identification of cold and hot data, pre-computation of frequently used tags/metrics, simplification of code logic using Materialized Views, and so on and so forth.

Tencent Music Transitions from ClickHouse to Apache Doris

ক্লিকহাউস থেকে অ্যাপাচি ডরিসে টেনসেন্ট মিউজিক ট্রানজিশন

Tencent Music wechselt von ClickHouse zu Apache Doris

Transiciones de Tencent Music de ClickHouse a Apache Doris

Tencent Music passe de ClickHouse à Apache Doris

क्लिकहाउस से अपाचे डोरिस तक टेनसेंट म्यूजिक ट्रांजिशन

TRINSIC

trinsic.id

-- employees

Since n.d.

COMPANY RANKING

TRINSIC

EVERGREEN INDEX #10694

Representing this company? Click here to claim and customize this page!

Trinsic WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

TRINSIC

trinsic.id

-- employees

Since n.d.

COMPANY RANKING

TRINSIC

EVERGREEN INDEX #10694

RELATED COMPANIESRANK

STORIES ABOUT TRINSIC

Representing this company? Click here to claim and customize this page!

Trinsic WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!