Open Meta

openmeta

Introduction Introduction In my most recent installments, I’ve been looking at Apache Iceberg, Apache Kafka, and Apache Flink. This led me to think about what might be a helpful extension in that space to look at, which led me to think about navigating the complexities of modern data ecosystems. Managing metadata effectively ensures data discoverability, governance, and collaboration. Enter OpenMetadata, an open-source platform designed to streamline metadata management, offering a robust data discovery, observability, and governance solution. In this blog post, we’ll dive into the technical underpinnings of OpenMetadata, explore its architecture, key features, and use cases, and provide visual aids to help you understand why it’s gaining traction in the data engineering community. With that preamble, let’s dive in! Apache Iceberg Apache Iceberg Apache Kafka Apache Kafka Apache Flink Apache Flink OpenMetadata OpenMetadata What is OpenMetadata? What is OpenMetadata? OpenMetadata is a unified, open-source metadata platform that empowers organizations to manage their data assets efficiently. Launched in 2021 and inspired by lessons from Uber’s metadata infrastructure, it provides a centralized repository for metadata, enabling data discovery, lineage tracking, quality monitoring, and team collaboration. With over 300 contributors and adoption across diverse industries, OpenMetadata stands out for its simplicity, extensibility, and vibrant community. It’s built to address the challenges of fragmented data ecosystems, where metadata often becomes a bottleneck for scalability and governance. Unlike traditional metadata tools that rely on complex graph databases or proprietary systems, OpenMetadata adopts a streamlined architecture with a schema-first approach. It supports over 90 connectors for ingesting metadata from databases, data warehouses, pipelines, and dashboards, making it a versatile choice for modern data stacks. Its user-friendly interface caters to technical and non-technical users, fostering a data-driven culture. Why OpenMetadata Matters Why OpenMetadata Matters Reading a database and producing a report was pretty straightforward in the olden days. You did some joins, some filtering, some formatting, and bang, you were done. Now you have complex pipelines that grab data from various sources and types. When, not if, something goes south with the results, it’s not easy to trace where it went wrong, and this is where OpenMetadata comes in. It’s a critical asset for understanding data lineage, ensuring quality, and enabling collaboration, addressing several pain points: Fragmented Data Sources: Organizations often use multiple tools (e.g., Snowflake, dbt, Metabase), leading to siloed metadata. OpenMetadata centralizes this metadata into a unified graph.
Data Discoverability: Finding relevant data assets can be time-consuming. OpenMetadata’s search capabilities and metadata enrichment make discovery intuitive.
Governance and Compliance: OpenMetadata supports robust governance without excessive manual effort through features like metadata versioning and automated workflows.
Scalability: Its lightweight architecture and extensive connector support suit enterprises of all sizes. Fragmented Data Sources: Organizations often use multiple tools (e.g., Snowflake, dbt, Metabase), leading to siloed metadata. OpenMetadata centralizes this metadata into a unified graph. Fragmented Data Sources Data Discoverability: Finding relevant data assets can be time-consuming. OpenMetadata’s search capabilities and metadata enrichment make discovery intuitive. Data Discoverability Governance and Compliance: OpenMetadata supports robust governance without excessive manual effort through features like metadata versioning and automated workflows. Governance and Compliance Scalability: Its lightweight architecture and extensive connector support suit enterprises of all sizes. Scalability Architecture of OpenMetadata Architecture of OpenMetadata Based on the time I spent, it appears that OpenMetadata is comprised of four core components: Metadata Store: A central repository that stores the metadata graph, connecting data assets, users, and tool-generated metadata. It uses a relational database (e.g., MySQL, Postgres) for storage, avoiding the complexity of graph databases like Neo4j.
Ingestion Framework: A pluggable framework that ingests metadata from over 90 sources, including databases (e.g., BigQuery, Snowflake), data lakes (e.g., S3, Iceberg), and BI tools (e.g., Power BI). Connectors are written in Python and support custom extensions.
Metadata Schemas: JSON-based schemas define metadata entities (e.g., tables, dashboards) and relationships. These schemas are extensible, allowing organizations to tailor metadata to their needs.
User Interface: A web-based UI built with React, offering search, lineage visualization, and collaboration tools. It integrates with Elasticsearch for full-text search and supports CMD + K shortcuts for quick navigation. Metadata Store: A central repository that stores the metadata graph, connecting data assets, users, and tool-generated metadata. It uses a relational database (e.g., MySQL, Postgres) for storage, avoiding the complexity of graph databases like Neo4j. Metadata Store Ingestion Framework: A pluggable framework that ingests metadata from over 90 sources, including databases (e.g., BigQuery, Snowflake), data lakes (e.g., S3, Iceberg), and BI tools (e.g., Power BI). Connectors are written in Python and support custom extensions. Ingestion Framework Metadata Schemas: JSON-based schemas define metadata entities (e.g., tables, dashboards) and relationships. These schemas are extensible, allowing organizations to tailor metadata to their needs. Metadata Schemas User Interface: A web-based UI built with React, offering search, lineage visualization, and collaboration tools. It integrates with Elasticsearch for full-text search and supports CMD + K shortcuts for quick navigation. User Interface The simplicity of the setup reduces deployment overhead. For example, setting up a local environment takes minutes, and the platform supports cloud deployments on AWS, Azure, and Google Cloud. Key Features OpenMetadata offers a rich set of features that, based on my experience in the space, really cover what people need/want to do. Here’s a breakdown of the most impactful ones that I gleaned from the documentation: Data Discovery
The full-text search engine, powered by Elasticsearch, indexes entity names, descriptions, tags, and even conversation threads. Users can refine searches with filters or use advanced queries to explore tables, dashboards, pipelines, and more.


Data Lineage
Lineage tracking provides column-level visibility into data flows across pipelines and tools. For example, you can trace how data moves from a PostgreSQL table through a dbt transformation to a Power BI dashboard. Lineage can be exported as PNG or PDF for documentation.


Data Quality and Profiling
Includes no-code data quality tests and profiling tools. Users can define test suites, monitor data health, and view results in an interactive dashboard. AutoPilot, an AI-driven feature, automates metadata extraction and profiling for new services, reducing onboarding time.


Collaboration
The platform fosters collaboration through conversation threads, task creation, and event notifications. Data producers and consumers can communicate directly on data assets, reducing silos.


Governance
Supports metadata versioning, tagging, and ownership assignment, enabling compliance with data governance policies. Its two-way metadata synchronization pushes enriched metadata (e.g., tags) back to source systems like Snowflake, ensuring consistency.


Extensibility The schema-first approach and REST APIs allow developers to extend metadata entities and integrate with custom tools. The ingestion framework supports community-contributed connectors, ensuring flexibility. Data Discovery
The full-text search engine, powered by Elasticsearch, indexes entity names, descriptions, tags, and even conversation threads. Users can refine searches with filters or use advanced queries to explore tables, dashboards, pipelines, and more. Data Discovery Data Discovery The full-text search engine, powered by Elasticsearch, indexes entity names, descriptions, tags, and even conversation threads. Users can refine searches with filters or use advanced queries to explore tables, dashboards, pipelines, and more. Data Lineage
Lineage tracking provides column-level visibility into data flows across pipelines and tools. For example, you can trace how data moves from a PostgreSQL table through a dbt transformation to a Power BI dashboard. Lineage can be exported as PNG or PDF for documentation. Data Lineage Data Lineage Lineage tracking provides column-level visibility into data flows across pipelines and tools. For example, you can trace how data moves from a PostgreSQL table through a dbt transformation to a Power BI dashboard. Lineage can be exported as PNG or PDF for documentation. Data Quality and Profiling
Includes no-code data quality tests and profiling tools. Users can define test suites, monitor data health, and view results in an interactive dashboard. AutoPilot, an AI-driven feature, automates metadata extraction and profiling for new services, reducing onboarding time. Data Quality and Profiling Data Quality and Profiling Includes no-code data quality tests and profiling tools. Users can define test suites, monitor data health, and view results in an interactive dashboard. AutoPilot, an AI-driven feature, automates metadata extraction and profiling for new services, reducing onboarding time. Collaboration
The platform fosters collaboration through conversation threads, task creation, and event notifications. Data producers and consumers can communicate directly on data assets, reducing silos. Collaboration Collaboration The platform fosters collaboration through conversation threads, task creation, and event notifications. Data producers and consumers can communicate directly on data assets, reducing silos. Governance
Supports metadata versioning, tagging, and ownership assignment, enabling compliance with data governance policies. Its two-way metadata synchronization pushes enriched metadata (e.g., tags) back to source systems like Snowflake, ensuring consistency. Governance Governance Supports metadata versioning, tagging, and ownership assignment, enabling compliance with data governance policies. Its two-way metadata synchronization pushes enriched metadata (e.g., tags) back to source systems like Snowflake, ensuring consistency. Extensibility The schema-first approach and REST APIs allow developers to extend metadata entities and integrate with custom tools. The ingestion framework supports community-contributed connectors, ensuring flexibility. Extensibility The schema-first approach and REST APIs allow developers to extend metadata entities and integrate with custom tools. The ingestion framework supports community-contributed connectors, ensuring flexibility. Extensibility Use Cases Use Cases The flexible design makes it applicable across many industries. Here are a few scenarios to consider:: Data Centralization: A retail company uses OpenMetadata to consolidate metadata from Snowflake, dbt, and Metabase, providing a single source of truth for analytics teams.
Governance Automation: A financial institution leverages AutoPilot to automate metadata tagging and enforce data masking policies in BigQuery, ensuring compliance with GDPR.
Data Discovery for AI: A SaaS provider uses OpenMetadata to standardize metadata for diverse customer datasets, enabling seamless integration into AI model pipelines. Data Centralization: A retail company uses OpenMetadata to consolidate metadata from Snowflake, dbt, and Metabase, providing a single source of truth for analytics teams. Data Centralization Governance Automation: A financial institution leverages AutoPilot to automate metadata tagging and enforce data masking policies in BigQuery, ensuring compliance with GDPR. Governance Automation Data Discovery for AI: A SaaS provider uses OpenMetadata to standardize metadata for diverse customer datasets, enabling seamless integration into AI model pipelines. Data Discovery for AI Collaboration Across Teams: An e-commerce platform uses OpenMetadata’s collaboration tools to bridge gaps between business analysts and data engineers, improving dashboard creation efficiency. Getting Started Getting Started There are multiple ways to get started with OpenMetadata, and the options are clearly described here. I don’t want to replicate them in this blog, because they can change over time. What shouldn’t change, though, is the UI address, so: here here Access the UI: Navigate to http://localhost:8585 to access the web interface.
Configure Connectors: Connect to your data sources using the ingestion framework. As I've said a few times, there are over 90 to choose from at the time of this writing. The documentation provides step-by-step guides for popular tools. Access the UI: Navigate to http://localhost:8585 to access the web interface. Access the UI Configure Connectors: Connect to your data sources using the ingestion framework. As I've said a few times, there are over 90 to choose from at the time of this writing. The documentation provides step-by-step guides for popular tools. Configure Connectors You can avoid the installation step and try the OpenMetadata Sandbox at sandbox.open-metadata.org. The community Slack channel is also an excellent resource for support and feature discussions. sandbox.open-metadata.org Slack Slack Challenges and Considerations Challenges and Considerations While OpenMetadata is powerful, it has limitations. For modern data lakehouse architectures (e.g., Delta Lake), connector functionality may be limited, and ingestion processes (e.g., Athena) can incur costs if not optimized, so don’t just blindly point it at a massive data lake and fire it off. Check connector compatibility and test the platform in a proof-of-concept before full deployment. Summary Summary What the heck is OpenMetadata, then? It is an open-source project that provides a unified metadata management solution. Its lightweight architecture, extensive connector support, and focus on collaboration make it a compelling choice for modern data stacks. By centralizing metadata, enhancing discoverability, and automating governance, OpenMetadata empowers organizations to unlock the full potential of their data assets. Whether you’re building a data-driven culture or tackling compliance challenges, OpenMetadata is worth exploring. Want to read more in my “What the Heck is???” series? A handy list is below: What The Heck Is DuckDB?
What the Heck Is Malloy?
What the Heck is PRQL?
What the Heck is GlareDB?
What the Heck is SeaTunnel?
What the Heck is LanceDB?
What the heck is SDF?
What the Heck is Paimon?
What the Heck is Proton?
What the Heck is PuppyGraph?
What the Heck is GPTScript?
What the Heck is WarpStream?
What the Heck is DeltaStream? What The Heck Is DuckDB? What The Heck Is DuckDB? What The Heck Is DuckDB? What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is PRQL? What the Heck is PRQL? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is LanceDB? What the Heck is LanceDB? What the Heck is LanceDB? What the heck is SDF? What the heck is SDF? What the heck is SDF? What the heck is SDF? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Proton? What the Heck is Proton? What the Heck is Proton? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is WarpStream? What the Heck is WarpStream? What the Heck is WarpStream? What the Heck is WarpStream? What the Heck is DeltaStream? What the Heck is DeltaStream? What the Heck is DeltaStream?

What the Heck is OpenMetadata?

Какво представлява Open Metadata?

Šta je Open Metadata?

¿Qué es Open Metadata?

¿Que é o Open Metadata?

Apa yang dimaksud dengan Open Metadata?

オープンメタデータとは何ですか?

តើវាជា metadata ដែលមានអត្ថប្រយោជន៍?

Yé Yé Yé Yé Yé Yé Yé

ຄວາມຄິດເຫັນທີ່ Heck ແມ່ນ Open Metadata?

खुल्ला डाटा र खुल्ला डाटा किन हुन्छ ?

Wat is Open Metadata eigenlijk?

د Open Metadata په اړه څه دي؟

Cary, North Carolina, May 14th, 2025/CyberNewsWire/--INE Security, a global leader in hands-on cybersecurity training and certifications, today highlighted how ongoing real-world practice with the latest CVEs (Common Vulnerabilities and Exposures) is essential for transforming security teams from reactive to proactive defenders. INE Security INE Security With over 26,000 new CVEs documented in the past year, security teams are drowning in vulnerability alerts while facing exploit windows that have compressed to hours in many cases. "Reading CVE bulletins is not the same as knowing how to stop the attack," said Dara Warn, CEO at INE Security. "Our Skill Dive platform gives practitioners hands-on experience with real vulnerabilities in contained environments, cutting incident response times when these same issues hit production. This practical approach delivers far more value than traditional security certifications alone." "Reading CVE bulletins is not the same as knowing how to stop the attack," said Dara Warn, CEO at INE Security. "Our Skill Dive platform gives practitioners hands-on experience with real vulnerabilities in contained environments, cutting incident response times when these same issues hit production. This practical approach delivers far more value than traditional security certifications alone." Skill Dive Skill Dive Skill Dive is INE Security's risk-free technical environment featuring exclusive labs not found in learning paths and courses. Skill Dive’s Vulnerabilities Lab Collection offers a continuously updated library of labs specifically designed to provide hands-on practice with actual CVEs, allowing security practitioners, including those preparing for pentester certifications, to experience both the exploitation and mitigation of current real-world threats in a safe environment. Skill Dive is INE Security's risk-free technical environment featuring exclusive labs not found in learning paths and courses. Skill Dive is INE Security's risk-free technical environment featuring exclusive labs not found in learning paths and courses. Vulnerabilities Lab Collection Vulnerabilities Lab Collection CVEs: From Bulletin to Defense CVEs are the standard identifiers for known vulnerabilities, but many security teams struggle to implement effective mitigations at scale, even those with Sec+ and other entry-level certifications. Common Challenges include: Risk prioritization across hundreds of monthly CVEs
Testing mitigations without impacting production
Adapting defenses to diverse system configurations
Building response muscle memory that works under pressure
Getting ahead of the threat curve instead of constantly reacting Risk prioritization across hundreds of monthly CVEs Testing mitigations without impacting production Adapting defenses to diverse system configurations Building response muscle memory that works under pressure Getting ahead of the threat curve instead of constantly reacting Practice Today's Threats. Prevent Tomorrow's Breaches. INE Security’s Skill Dive Vulnerabilities Lab Collection delivers: INE Security’s Skill Dive Vulnerabilities Lab Collection INE Security’s Skill Dive Vulnerabilities Lab Collection Exclusive vulnerability labs not available in standard security training
Monthly CVE updates focusing on high-impact vulnerabilities
Isolated practice environment for both offensive and defensive techniques
Complete severity coverage from critical zero-days to common misconfigurations
Practical exploitation and defense experience that transfers directly to production incidents Exclusive vulnerability labs not available in standard security training Monthly CVE updates focusing on high-impact vulnerabilities Isolated practice environment for both offensive and defensive techniques Complete severity coverage from critical zero-days to common misconfigurations Practical exploitation and defense experience that transfers directly to production incidents "When a critical CVE drops, you don't have time to theorize," said Tracy Wallace, Director of Content at INE Security. "Teams with hands-on practice respond significantly faster because they've seen similar attack patterns before. Log4Shell (CVE-2021-44228) was a perfect example – practitioners who had experience with JNDI injection attacks were able to implement effective mitigations within hours, while others took days or even weeks to fully remediate." "When a critical CVE drops, you don't have time to theorize," said Tracy Wallace, Director of Content at INE Security. "Teams with hands-on practice respond significantly faster because they've seen similar attack patterns before. Log4Shell (CVE-2021-44228) was a perfect example – practitioners who had experience with JNDI injection attacks were able to implement effective mitigations within hours, while others took days or even weeks to fully remediate." Real Benefits for Security Teams Skill Dive delivers immediate advantages for practitioners: Develop attack pattern recognition that speeds incident response
Understand attack chains beyond what bulletins describe
Practice team coordination for high-pressure security events
Identify defensive gaps before attackers find them
Build skills that directly translate to career advancement Develop attack pattern recognition that speeds incident response Understand attack chains beyond what bulletins describe Practice team coordination for high-pressure security events Identify defensive gaps before attackers find them Build skills that directly translate to career advancement SecOps teams, security analysts, and IT admins get exactly what certification courses miss: hands-on practice with real-world vulnerabilities. "Security professionals who regularly drill on current vulnerabilities become exponentially more valuable to their organizations," said Wallace. "The best defenders understand both the attack and defense sides of the equation." "Security professionals who regularly drill on current vulnerabilities become exponentially more valuable to their organizations," said Wallace. "The best defenders understand both the attack and defense sides of the equation." High-Impact CVEs in the Skill Dive Collection The platform features hands-on labs for the most actively exploited vulnerabilities in enterprise environments, including: OpenMetadata Authentication Bypass (CVE-2024-28255): Exploit the target machine running OpenMetadata by bypassing the authentication and gaining remote code execution (RCE)
Calibre RCE (CVE-2024-6782): Exploit the remote code execution vulnerability in Calibre, leading to unauthorized system access
Log4Shell (CVE-2021-44228): Practice identifying and remediating this critical remote code execution vulnerability that continues to plague Java applications across multiple sectors
Spring4Shell (CVE-2022-22965): Gain hands-on experience with this widely exploited RCE vulnerability affecting Spring Framework applications OpenMetadata Authentication Bypass (CVE-2024-28255): Exploit the target machine running OpenMetadata by bypassing the authentication and gaining remote code execution (RCE) OpenMetadata Authentication Bypass (CVE-2024-28255) OpenMetadata Authentication Bypass (CVE-2024-28255) Calibre RCE (CVE-2024-6782): Exploit the remote code execution vulnerability in Calibre, leading to unauthorized system access Calibre RCE (CVE-2024-6782) Calibre RCE (CVE-2024-6782) Log4Shell (CVE-2021-44228): Practice identifying and remediating this critical remote code execution vulnerability that continues to plague Java applications across multiple sectors Log4Shell (CVE-2021-44228) Log4Shell (CVE-2021-44228) Spring4Shell (CVE-2022-22965): Gain hands-on experience with this widely exploited RCE vulnerability affecting Spring Framework applications Spring4Shell (CVE-2022-22965) Spring4Shell (CVE-2022-22965) "We continuously track which vulnerabilities are most actively exploited," said Wallace. "Our collection prioritizes CVEs with the highest real-world impact, not just theoretical severity ratings." "We continuously track which vulnerabilities are most actively exploited," said Wallace. "Our collection prioritizes CVEs with the highest real-world impact, not just theoretical severity ratings." Proactive Security Through Deliberate Practice The Skill Dive approach includes: Monthly updates aligned with emerging threat patterns
Realistic environments mirroring production systems
Practical documentation focused on effective mitigations
Continuous evolution based on real-world attack trends Monthly updates aligned with emerging threat patterns Realistic environments mirroring production systems Practical documentation focused on effective mitigations Continuous evolution based on real-world attack trends Recent lab additions include other top-exploited vulnerabilities such as Cacti Import Packages RCE (CVE-2024-25641), Gradio Path Traversal (CVE-2024-1561), Calibre Arbitrary File Read (CVE-2024-6781), Graylog Information Exposure (CVE-2024-24824), and Navidrome SQL Injection (CVE-2024-47062). "Security teams that regularly practice with new vulnerabilities stop more breaches, period," said Wallace. "Practice transforms defense from constant firefighting into strategic advantage." "Security teams that regularly practice with new vulnerabilities stop more breaches, period," said Wallace. "Practice transforms defense from constant firefighting into strategic advantage." Availability Individual subscriptions to Skill Dive are available now. Enterprise packages for team training are also available. For more information, users can visit ine.com/cyber-ranges ine.com/cyber-ranges ine.com/cyber-ranges About INE Security INE Security is the premier provider of online networking and cybersecurity training and cybersecurity certifications. INE Security INE Security Harnessing a powerful hands-on lab platform, cutting-edge technology, a global video distribution network, and world-class instructors, INE Security is the top training choice for Fortune 500 companies worldwide for cybersecurity training in business and for IT professionals looking to advance their careers. INE Security’s suite of learning paths offers an incomparable depth of expertise across cybersecurity. The company is committed to delivering advanced technical training while also lowering the barriers worldwide for those looking to enter and excel in an IT career. Contact Kathryn Brown INE Security kbrown@ine.com This story was published as a press release by Cybernewswire under HackerNoon’s Business Blogging Program. This story was published as a press release by Cybernewswire under HackerNoon’s Business Blogging Program. Program Program

Continuous CVE Practice Closes Critical Gap Between Vulnerability Alerts and Effective Defense

La práctica continua de CVE cierra la brecha crítica entre las alertas de vulnerabilidad y la defensa efectiva

A práctica continua de CVE pecha a brecha crítica entre as alertas de vulnerabilidade e a defensa efectiva

継続的なCVE実践は、脆弱性警告と効果的な防衛の間の重要なギャップを満たす

ການຝຶກອົບຮົມ CVE Continuous Closes The Critical Gap Between Vulnerability Alerts and Effective Defense

Nepārtraukta CVE prakse aizver kritisko plaisu starp neaizsargātības brīdinājumiem un efektīvu aizsardzību

Continu CVE-praktijk sluit kritische kloof tussen kwetsbaarheidswaarschuwingen en effectieve verdediging

Practica CVE continuă închide decalajul critic între avertismentele de vulnerabilitate și apărarea eficientă

Continuous CVE Practice Closes Critical Gap Between Vulnerability Alerts and Effective Defence - விரைவில் CVE Practice Closes Critical Gap Between Vulnerability Alerts and Effective Defence

Продовжена практика CVE заповнює критичний розрив між попередженнями про уразливість та ефективною обороною

Men o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z o‘z

Thực hành CVE liên tục thu hẹp khoảng cách quan trọng giữa cảnh báo lỗ hổng và phòng thủ hiệu quả

持续的CVE实践缩小了漏洞警报和有效防御之间的关键差距

Last week, two remarkable events occurred: Meta announced its Llama 3, the most capable and highly anticipated open large language model (LLM) to date, and Boston Dynamics introduced its new fully electric Atlas robot platform — a stark departure from their hydraulic robots! At first glance, these developments may seem unrelated, but they are actually deeply interconnected, with the potential to synergistically drive each other forward and reshape how we work with and implement AI. At the heart of this connection is the transformative power of advanced AI. Breakthroughs in natural language processing and machine learning, exemplified by Llama 3, extend beyond language alone. These techniques, from deep neural networks to reinforcement learning, are also propelling significant advancements in computer vision, motion planning, and robot control. As language models like Llama 3 continue to expand the boundaries of AI’s ability to understand and interact with the world, they also lay the groundwork for more intelligent and capable robots. The implications, of course, extend beyond communication. The same techniques used to train LLMs on vast amounts of text data are also applicable to learning from massive datasets of sensor readings, images, and simulations for robots. Moreover, the open-source nature of models like Llama 3 democratizes access to state-of-the-art AI, enabling a broader spectrum of researchers and companies to integrate these capabilities into their robotic systems. At least start playing on that field! Every successful (and unsuccessful) attempt by Atlas to manipulate an object, navigate a cluttered factory floor, or assist a human worker becomes a valuable data point for the model, just as it does for Llama 3. Using this innovative model, Meta developed a standalone app, meta.ai, and embedded it into WhatsApp, Instagram, and Facebook, collecting vast and highly personalized data. This could lead us toward synthetic social networks or — which is more likely — an extremely personalized, embodied AI experience. As Llama 3 and the electric Atlas converge, they may accelerate each other’s development, blurring the lines between software and hardware, bits and atoms. As language models become more proficient at understanding and interacting with the world, robots will become more capable of applying that knowledge. This doesn’t necessarily mean evil. More capable and advanced robots, which can be fine-tuned and improved with your personal data and specifics, could take over mundane tasks at home. Imagine, instead of hundreds of text editors, finally having a robot that can load and unload the dishwasher and do the laundry. Currently, launches like Llama 3 are seen as enhancing AI’s understanding and processing capabilities, but in the long term, they will be one of the milestones in building and deploying machines that are finely attuned and aligned with us to assist in our daily lives. https://youtu.be/29ECwExc-_M?si=UyexFH1jOfd5TGY9&embedable=true Other Impressive Models: (I didn’t send you the FOD digest last Monday because I was at a conference dedicated to citizen diplomacy. Therefore, today, we have an extensive list of recently launched models that are worth checking out, along with other relevant research papers.) Mixtral-8x22B: Mistral AI introduced a scalable sparse mixture of expert models optimizing cost and latency by selectively using parameters during inference, offering a high capacity and efficiency model for further training and applications →read the paper
Rerank 3: Launched by Cohere, this model enhances enterprise search and Retrieval Augmented Generation systems, improving accuracy in document retrieval across multiple languages and data formats →read the paper
Idefics2: Hugging Face's model that excels in integrating text and image data, significantly improving on OCR and visual question-answering tasks →read the paper
Reka Core, Flash, and Edge: A series of multimodal language models from Reka that process text, images, video, and audio, demonstrating high performance across diverse tasks →read the paper
Ferret-UI: Developed by Apple, this model specializes in mobile UI interaction, enhancing user experience by accurately performing tasks tailored to the unique properties of UI screens →ead the paper
Zamba: Zyphra's compact and efficient SSM Hybrid model is designed for performance with reduced training data needs, optimized for consumer hardware →read the paper
YaART: Yandex's advanced text-to-image diffusion model that optimizes training efficiency and quality with smaller, high-quality datasets →read the paper
RHO-1: A novel approach by Xiamen University focusing on Selective Language Modeling to enhance efficiency by prioritizing useful tokens during training →read the paper
RecurrentGemma: Google DeepMind's model that moves past traditional transformers by incorporating recurrences for more efficient long-sequence processing →read the paper
JetMoE: An economical LLM from the MIT-IBM Watson AI Lab using a mixture-of-experts architecture to achieve high performance at reduced costs →read the paper News from The Usual Suspects © Google open-sourced Gemini Cookbook
and is merging its Research, DeepMind, and Responsible AI teams to accelerate its “capacity to deliver capable AI”
DeepMind published an article about the ethics of advanced AI assistants, stressing the significance of their integration into daily life Hugging Face https://x.com/QGallouedec/status/1782430246957994422?embedable=true Stanford published the 2024 AI Index report, reflecting the escalating impact of AI on society. It contains data with new estimates on AI training costs and detailed insights into the responsible AI landscape. It also introduces a new chapter on AI's influence on science and medicine. Highlights include the substantial cost of training state-of-the-art models, such as GPT-4 and Gemini Ultra; the dominance of the U.S. in producing top AI models; and significant investment growth in generative AI despite overall funding declines. Additionally, the report notes a major increase in AI-related regulations in the U.S. and heightened public awareness and nervousness about AI's future impact. Enjoyed This Story? I write a weekly analysis of the AI world in the Turing Post newsletter. We aim to equip you with comprehensive knowledge and historical insights so you can make informed decisions about AI and ML. Last week, two remarkable events occurred: Meta announced its Llama 3 , the most capable and highly anticipated open large language model (LLM) to date, and Boston Dynamics introduced its new fully electric Atlas robot platform — a stark departure from their hydraulic robots! At first glance, these developments may seem unrelated, but they are actually deeply interconnected, with the potential to synergistically drive each other forward and reshape how we work with and implement AI. Llama 3 Llama 3 electric Atlas robot electric Atlas robot At the heart of this connection is the transformative power of advanced AI. Breakthroughs in natural language processing and machine learning, exemplified by Llama 3, extend beyond language alone. These techniques, from deep neural networks to reinforcement learning, are also propelling significant advancements in computer vision, motion planning, and robot control. As language models like Llama 3 continue to expand the boundaries of AI’s ability to understand and interact with the world, they also lay the groundwork for more intelligent and capable robots. The implications, of course, extend beyond communication. The same techniques used to train LLMs on vast amounts of text data are also applicable to learning from massive datasets of sensor readings, images, and simulations for robots. Moreover, the open-source nature of models like Llama 3 democratizes access to state-of-the-art AI, enabling a broader spectrum of researchers and companies to integrate these capabilities into their robotic systems. At least start playing on that field! Every successful (and unsuccessful) attempt by Atlas to manipulate an object, navigate a cluttered factory floor, or assist a human worker becomes a valuable data point for the model, just as it does for Llama 3. Using this innovative model, Meta developed a standalone app, meta.ai , and embedded it into WhatsApp, Instagram, and Facebook, collecting vast and highly personalized data. This could lead us toward synthetic social networks or — which is more likely — an extremely personalized, embodied AI experience. As Llama 3 and the electric Atlas converge, they may accelerate each other’s development, blurring the lines between software and hardware, bits and atoms. As language models become more proficient at understanding and interacting with the world, robots will become more capable of applying that knowledge. meta.ai meta.ai synthetic social networks synthetic social networks This doesn’t necessarily mean evil. More capable and advanced robots, which can be fine-tuned and improved with your personal data and specifics, could take over mundane tasks at home. Imagine, instead of hundreds of text editors, finally having a robot that can load and unload the dishwasher and do the laundry. Currently, launches like Llama 3 are seen as enhancing AI’s understanding and processing capabilities, but in the long term, they will be one of the milestones in building and deploying machines that are finely attuned and aligned with us to assist in our daily lives. https://youtu.be/29ECwExc-_M?si=UyexFH1jOfd5TGY9&embedable=true https://youtu.be/29ECwExc-_M?si=UyexFH1jOfd5TGY9&embedable=true Other Impressive Models: (I didn’t send you the FOD digest last Monday because I was at a conference dedicated to citizen diplomacy. Therefore, today, we have an extensive list of recently launched models that are worth checking out, along with other relevant research papers.) (I didn’t send you the FOD digest last Monday because I was at a conference dedicated to citizen diplomacy. Therefore, today, we have an extensive list of recently launched models that are worth checking out, along with other relevant research papers.) Mixtral-8x22B: Mistral AI introduced a scalable sparse mixture of expert models optimizing cost and latency by selectively using parameters during inference, offering a high capacity and efficiency model for further training and applications →read the paper Rerank 3: Launched by Cohere, this model enhances enterprise search and Retrieval Augmented Generation systems, improving accuracy in document retrieval across multiple languages and data formats →read the paper Idefics2: Hugging Face's model that excels in integrating text and image data, significantly improving on OCR and visual question-answering tasks →read the paper Reka Core, Flash, and Edge: A series of multimodal language models from Reka that process text, images, video, and audio, demonstrating high performance across diverse tasks →read the paper Ferret-UI: Developed by Apple, this model specializes in mobile UI interaction, enhancing user experience by accurately performing tasks tailored to the unique properties of UI screens →ead the paper Zamba: Zyphra's compact and efficient SSM Hybrid model is designed for performance with reduced training data needs, optimized for consumer hardware →read the paper YaART: Yandex's advanced text-to-image diffusion model that optimizes training efficiency and quality with smaller, high-quality datasets →read the paper RHO-1: A novel approach by Xiamen University focusing on Selective Language Modeling to enhance efficiency by prioritizing useful tokens during training →read the paper RecurrentGemma: Google DeepMind's model that moves past traditional transformers by incorporating recurrences for more efficient long-sequence processing →read the paper JetMoE: An economical LLM from the MIT-IBM Watson AI Lab using a mixture-of-experts architecture to achieve high performance at reduced costs →read the paper Mixtral-8x22B: Mistral AI introduced a scalable sparse mixture of expert models optimizing cost and latency by selectively using parameters during inference, offering a high capacity and efficiency model for further training and applications →read the paper Mixtral-8x22B: →read the paper Rerank 3: Launched by Cohere, this model enhances enterprise search and Retrieval Augmented Generation systems, improving accuracy in document retrieval across multiple languages and data formats →read the paper Rerank 3: →read the paper Idefics2: Hugging Face's model that excels in integrating text and image data, significantly improving on OCR and visual question-answering tasks →read the paper Idefics2: →read the paper Reka Core, Flash, and Edge: A series of multimodal language models from Reka that process text, images, video, and audio, demonstrating high performance across diverse tasks →read the paper Reka Core, Flash, and Edge: →read the paper Ferret-UI: Developed by Apple, this model specializes in mobile UI interaction, enhancing user experience by accurately performing tasks tailored to the unique properties of UI screens →ead the paper Ferret-UI: →ead the paper Zamba: Zyphra's compact and efficient SSM Hybrid model is designed for performance with reduced training data needs, optimized for consumer hardware →read the paper Zamba: →read the paper YaART: Yandex's advanced text-to-image diffusion model that optimizes training efficiency and quality with smaller, high-quality datasets →read the paper YaART: →read the paper RHO-1: A novel approach by Xiamen University focusing on Selective Language Modeling to enhance efficiency by prioritizing useful tokens during training →read the paper RHO-1: →read the paper RecurrentGemma: Google DeepMind's model that moves past traditional transformers by incorporating recurrences for more efficient long-sequence processing →read the paper RecurrentGemma: →read the paper JetMoE: An economical LLM from the MIT-IBM Watson AI Lab using a mixture-of-experts architecture to achieve high performance at reduced costs →read the paper JetMoE: →read the paper News from The Usual Suspects © Google Google open-sourced Gemini Cookbook and is merging its Research, DeepMind, and Responsible AI teams to accelerate its “capacity to deliver capable AI” DeepMind published an article about the ethics of advanced AI assistants, stressing the significance of their integration into daily life open-sourced Gemini Cookbook Gemini Cookbook and is merging its Research, DeepMind, and Responsible AI teams to accelerate its “ capacity to deliver capable AI ” merging capacity to deliver capable AI DeepMind published an article about the ethics of advanced AI assistants, stressing the significance of their integration into daily life published Hugging Face Hugging Face https://x.com/QGallouedec/status/1782430246957994422?embedable=true https://x.com/QGallouedec/status/1782430246957994422?embedable=true Stanford Stanford published the 2024 AI Index report, reflecting the escalating impact of AI on society. It contains data with new estimates on AI training costs and detailed insights into the responsible AI landscape. It also introduces a new chapter on AI's influence on science and medicine. Highlights include the substantial cost of training state-of-the-art models, such as GPT-4 and Gemini Ultra; the dominance of the U.S. in producing top AI models; and significant investment growth in generative AI despite overall funding declines. Additionally, the report notes a major increase in AI-related regulations in the U.S. and heightened public awareness and nervousness about AI's future impact. published the 2024 AI Index report , reflecting the escalating impact of AI on society. It contains data with new estimates on AI training costs and detailed insights into the responsible AI landscape. It also introduces a new chapter on AI's influence on science and medicine. Highlights include the substantial cost of training state-of-the-art models, such as GPT-4 and Gemini Ultra; the dominance of the U.S. in producing top AI models; and significant investment growth in generative AI despite overall funding declines. Additionally, the report notes a major increase in AI-related regulations in the U.S. and heightened public awareness and nervousness about AI's future impact. the 2024 AI Index report Enjoyed This Story? I write a weekly analysis of the AI world in the Turing Post newsletter. We aim to equip you with comprehensive knowledge and historical insights so you can make informed decisions about AI and ML. Turing Post

AI Industries Converge: Llama 3 and Electric Atlas Have More In Common Than You Think

The United States v Meta Platforms Court Filing October 24, 2023 is part of . You can jump to any part in this filing . This is part 13 of 100. HackerNoon’s Legal PDF Series here 3. Meta designs and deploys features to capture young users’ attention and prolong their time on its Social Media Platforms. 86. Acquiring young users helps secure Meta’s profit stream over time. By capturing users’ attention and engagement when they are young, Meta ensures future engagement and monetization as those young users grow up. 87. Meta thus develops and implements features to attract young users and keep them engaged on its Social Media Platforms for as long as possible. These features include: engagement-based (as opposed to chronological) feeds; infinite scroll; push notifications; ephemeral content; and video-based content. 88. Meta had originally displayed content on a user’s “Feed” chronologically, i.e., in the order the content was posted by people the user elected to follow. Meta moved from chronological Feeds to engagement-based Feeds in 2009 (for Facebook) and 2016 (for Instagram). 89. The engagement-based Feed is different and alters the users’ experience. It algorithmically presents material to users based on several engagement components: posts with more “Likes,” comments, and other indicia of user engagement are displayed to users first. 90. This change was designed to prioritize material most likely to engage users for longer periods of time. 91. In the fall of 2016, Instagram debuted its infinite scroll system. 92. Infinite scroll is characterized by the partial display of additional content at the bottom of the user’s screen, such that the user is typically unable to look at a single post in isolation (without seeing the top portion of the next post in their Feed). 93. The “teasing” of yet-to-be-viewed content continues indefinitely; as the user scrolls down the Feed, new content is automatically loaded and “teased.” 94. This “teasing” feature is intended to keep young users of the Platform engaged and continuing to scroll to the new content. 95. In April 2015, Meta introduced a variety of “push notifications” to Instagram. Push notifications are auditory and visual cues to alert users when accounts they follow add new content. 96. Push notifications allowed Instagram to draw its users back to the Platform at any time of day. [Redacted] 97. Meta also sought to increase engagement through making certain content available to users only temporarily—with notifications and visual design cues indicating that the content would soon disappear forever (ephemeral content). 98. Ephemeral content leads young users to more frequently open Meta’s Social Media Platforms so they do not “miss out” on any new content. This phenomenon is called “Fear of Missing Out,” or “FOMO.” Meta designed ephemeral content features in its Social Media Platforms to induce this sense of FOMO in young users. 99. For example, on August 2, 2016, Meta introduced a feature to Instagram designed to show images and narratives for only a short amount of time before disappearing, known as the “Stories” feature. Meta released a similar feature to Facebook in 2017. 100-101. [Redacted] 102. Another example is “Live,” which gives users the ability to livestream videos to followers or the public. 103. Meta launched Facebook Live on a limited basis to celebrities and other highprofile users in August 2015, with the feature being available to all users by April 2016. Instagram soon followed in November 2016. 104. Live allows users to create video content in real time that their followers can watch and react to, often called “going Live.” 105. When an account goes Live, the Instagram Platform sends out a notification. 106. [Redacted] 107. In addition to video-streaming offered through the Live feature, Meta has also designed and implemented several video features, including “IGTV,” “Instagram Video,” and ultimately “Reels.”[7] As with prior features, Meta focused on teen engagement with these video features. 108. [Redacted] 109. In 2020, when Meta introduced its short-form video feature, “Reels,” to the U.S. market on Instagram, [Redacted] Reels were made available on Facebook in September 2021. 110. Reels are algorithmically presented to users based on a number of factors, including the user’s activity, the popularity of the content, and the viewer’s connection to the creator. 111. Reels display metrics such as Like counts, comments, and views in the video itself, which reduces the need for the user to navigate away from the video. 112-116. [Redacted] [7] IGTV was revamped in October 2021 (in a shift to Instagram Video), and ultimately removed completely from the Platform in March 2022. Reels was merged with and superseded “Instagram Video” in June 2022. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case 4:23-cv-05448 retrieved on October 25, 2023, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. Washingtonpost.com

Meta Intentionally Deploys Features To Keep Young Users Glued to Their Phones

The Release of Meta’s LLAMA-2 Marks A New Era in The History of Humanity

Well, what did we tell you! Billionaire Mark Zuckerberg, the owner of Facebook, Instagram, and WhatsApp, is steaming ahead with plans to create a social network platform that would be similar to Twitter. Meta's take on the social media platform is becoming real by the day and is already trying to woo celebrities, including the Dalai Lama and Oprah Winfrey, to be a part of the first batch of users who get enrolled. Referred to as "Project 92" internally, the platform is aimed at users who want the convenience of Twitter without having to worry if it's being run by a literal slavedriver. “We’ve been hearing from creators and public figures who are interested in having a platform that is sanely run, that they believe that they can trust and rely upon for distribution,” Meta’s chief product officer, Chris Cox, was quoted as telling employees working on the project. No word on when the platform will be released but we do know that a new standalone app for the service will be based on Instagram and integrate with ActivityPub, the decentralized social media protocol, The Verge reported. Instagram ranked #2 on HackerNoon's Tech Company Rankings this week while Facebook was on the #8 spot. Major VC Firm Bets Crypto's Future is in the UK 🪙 While the U.S. seems hell bent on reigning in on cryptocurrency and crypto trading, the U.K. seems to be embracing it. In fact, so bullish is Andreessen Horowitz about the U.K.'s prospect as a crypto hub, that the American venture capital firm — which manages billions and counts multiple success stories in its portfolio, including Facebook — has decided to expand outside the U.S. and open its first ever off-shore office in London. "We believe that the UK is on the right path to becoming a leader in crypto regulation. It is home to more unicorns than Germany, France, and Sweden combined; to some of the world’s largest financial markets and pools of capital; and to highly-sophisticated, world-class regulators. All of these make the UK strongly positioned to lead in web3," the firm, one of the more prominent VCs betting on crypto, said in a statement. Andreessen Horowitz's expansion outside the U.S. makes sense; roughly 90% of crypto trading already takes place outside the U.S. With the U.S. Securities & Exchange Commission coming down hard on crypto, it's no wonder that more forward-minded investors are seeking avenues outside mainland America. Though whether Andreessen Horowitz's bet will pay off, only time will tell. For now, the VC firm left this tidbit in its announcement, perhaps for the naysayers. "A common question we hear is: 'What problems do blockchains solve?' Services built on blockchains solve the same problems that other digital services solve, but with better outcomes." 👋 You’re reading part 2 of HackerNoon's Tech Company News Brief, a weekly collection of tech goodness that combines HackerNoon's proprietary data with internet trends to determine which companies are rising and falling in the public consciousness. Part 1 went live yesterday. Prefer reading the whole thing a day early AND in one go? No problemo! Just subscribe here to receive the complete newsletter in your inbox every Tuesday. In Other News.. 📰 AMD is clawing back at Nvidia, looking to launch its own AI "superchip" to compete against Team Green, which already dominates the AI computing market with 80% to 95% of market share.
The UK will host the first major global summit on AI safety later this year.
Intel was unable to convince the German government to give it more subsidies for the currently in construction advanced semiconductor manufacturing facility near Magdeburg, Germany.
In an apparent FU to Google, Twitter is refusing to pay its Google Cloud bills as its contract comes up for renewal this month, Platformer reports.
Reddit's decision to charge exorbitant prices for third-party apps via its API is not sitting well with its userbase. And that's a wrap! Don't forget to share this newsletter with your family and friends! See y'all next week. PEACE! ☮️ — Sheharyar Khan, Editor, Business Tech @ HackerNoon Well, what did we tell you ! Billionaire Mark Zuckerberg, the owner of Facebook , Instagram , and WhatsApp , is steaming ahead with plans to create a social network platform that would be similar to Twitter . what did we tell you what did we tell you Facebook Facebook Instagram Instagram WhatsApp WhatsApp Twitter Twitter Meta's take on the social media platform is becoming real by the day and is already trying to woo celebrities, including the Dalai Lama and Oprah Winfrey, to be a part of the first batch of users who get enrolled. Referred to as "Project 92" internally, the platform is aimed at users who want the convenience of Twitter without having to worry if it's being run by a literal slavedriver . literal slavedriver literal slavedriver “We’ve been hearing from creators and public figures who are interested in having a platform that is sanely run, that they believe that they can trust and rely upon for distribution,” Meta’s chief product officer, Chris Cox, was quoted as telling employees working on the project. No word on when the platform will be released but we do know that a new standalone app for the service will be based on Instagram and integrate with ActivityPub, the decentralized social media protocol, The Verge reported . The Verge reported reported Instagram ranked #2 on HackerNoon's Tech Company Rankings this week while Facebook was on the #8 spot. Tech Company Rankings Tech Company Rankings Major VC Firm Bets Crypto's Future is in the UK 🪙 Major VC Firm Bets Crypto's Future is in the UK 🪙 While the U.S. seems hell bent on reigning in on cryptocurrency and crypto trading, the U.K. seems to be embracing it. In fact, so bullish is Andreessen Horowitz about the U.K.'s prospect as a crypto hub , that the American venture capital firm — which manages billions and counts multiple success stories in its portfolio, including Facebook — has decided to expand outside the U.S. and open its first ever off-shore office in London. prospect as a crypto hub prospect as a crypto hub "We believe that the UK is on the right path to becoming a leader in crypto regulation. It is home to more unicorns than Germany, France, and Sweden combined; to some of the world’s largest financial markets and pools of capital; and to highly-sophisticated, world-class regulators. All of these make the UK strongly positioned to lead in web3," the firm, one of the more prominent VCs betting on crypto, said in a statement . statement statement Andreessen Horowitz's expansion outside the U.S. makes sense; roughly 90% of crypto trading already takes place outside the U.S. With the U.S. Securities & Exchange Commission coming down hard on crypto, it's no wonder that more forward-minded investors are seeking avenues outside mainland America. 90% of crypto trading 90% of crypto trading Though whether Andreessen Horowitz's bet will pay off, only time will tell. For now, the VC firm left this tidbit in its announcement, perhaps for the naysayers . naysayers naysayers "A common question we hear is: 'What problems do blockchains solve?' Services built on blockchains solve the same problems that other digital services solve, but with better outcomes." 👋 You’re reading part 2 of HackerNoon's Tech Company News Brief, a weekly collection of tech goodness that combines HackerNoon's proprietary data with internet trends to determine which companies are rising and falling in the public consciousness. Part 1 went live yesterday. Prefer reading the whole thing a day early AND in one go? No problemo! Just subscribe here to receive the complete newsletter in your inbox every Tuesday. 👋 You’re reading part 2 of HackerNoon's Tech Company News Brief, a weekly collection of tech goodness that combines HackerNoon's proprietary data with internet trends to determine which companies are rising and falling in the public consciousness. Part 1 went live yesterday. Prefer reading the whole thing a day early AND in one go? No problemo! Just subscribe here to receive the complete newsletter in your inbox every Tuesday. You’re reading part 2 of HackerNoon's Tech Company News Brief, a weekly collection of tech goodness that combines HackerNoon's proprietary data with internet trends to determine which companies are rising and falling in the public consciousness. Part 1 went live yesterday. Prefer reading the whole thing a day early AND in one go? No problemo! Just subscribe here to receive the complete newsletter in your inbox every Tuesday. part 2 Part 1 Part 1 here complete In Other News.. 📰 In Other News.. 📰 AMD is clawing back at Nvidia, looking to launch its own AI "superchip" to compete against Team Green, which already dominates the AI computing market with 80% to 95% of market share. The UK will host the first major global summit on AI safety later this year. Intel was unable to convince the German government to give it more subsidies for the currently in construction advanced semiconductor manufacturing facility near Magdeburg, Germany. In an apparent FU to Google, Twitter is refusing to pay its Google Cloud bills as its contract comes up for renewal this month, Platformer reports. Reddit's decision to charge exorbitant prices for third-party apps via its API is not sitting well with its userbase. AMD is clawing back at Nvidia , looking to launch its own AI "superchip" to compete against Team Green, which already dominates the AI computing market with 80% to 95% of market share. AMD AMD Nvidia Nvidia launch launch The UK will host the first major global summit on AI safety later this year. host host Intel was unable to convince the German government to give it more subsidies for the currently in construction advanced semiconductor manufacturing facility near Magdeburg, Germany. Intel Intel unable unable In an apparent FU to Google , Twitter is refusing to pay its Google Cloud bills as its contract comes up for renewal this month, Platformer reports . Google Google Twitter Twitter reports reports Reddit's decision to charge exorbitant prices for third-party apps via its API is not sitting well with its userbase. third-party apps third-party apps not sitting well not sitting well And that's a wrap! Don't forget to share this newsletter with your family and friends! See y'all next week. PEACE! ☮️ — Sheharyar Khan, Editor, Business Tech @ HackerNoon

Meta is Actually Making a Twitter Clone 

মেটা আসলে একটি টুইটার ক্লোন তৈরি করছে

Meta erstellt tatsächlich einen Twitter-Klon

메타는 실제로 트위터 복제를 만들고 있습니다

Meta Aslında Bir Twitter Klonu Yapıyor

The gaming field is always one of the hottest lands in the metaverse. Some people think that on-chain games would become one of the highest volume market segments in the Web3 space, while others doubt the long-term economic viability of the most common Play To Earn (P2E) business model. Including Axie Infinity and STEPN, the reason why most blockchain gaming projects are unsustainable is that their P2E models lead to an inability to withstand huge sell-offs, and all players have to either actively or passively participate in the death spiral that eventually cannot avoid falling into the ponzinomics. However, the dilemma that most blockchain games are currently facing is not insurmountable. Since 2022, while the flaws of traditional P2E games are gradually being revealed, there have been some projects that have made innovations in business models to try to solve sustainability problems. Some promoting themselves as F2O (free to own), PAE (play and earn), etc. have emerged in the market, transforming and enhancing the traditional P2E model, thus extending the life cycle of the game. Such transformation cannot fundamentally solve the impact of the marginal diminishing effect of the Web3 decentralised profit-sharing model on post-entry players. So does this mean it’s impossible for blockchain games to get rid of this death spiral? The answer is not unqualified. Looking back at the traditional game industry, from the initial time-killing single-player games to the complete eSports industry today, it has gone through several iterations. Today a brand new web3 game operation model built on the esports ecosystem is about to bring a revolution to the whole on-chain game market – it’s time to get into Gamefi 2.0. Why should we focus on Web3 game development? The gaming industry, which is currently around $200 billion, is expected to see robust growth in the years ahead. Accenture’s 2021 estimate of the direct and indirect value of the gaming industry stood at $300 billion. As one group of the earliest inhabitants of the virtual world, game users are more familiar with creating their own virtual profiles than many others. Yat Siu, the co-founder and chairman of Animoca Brands, pointed to the 3.2 billion people who play video games on a regular basis, which he says are already operating in an “enclosed metaverses.” The people playing those games have created virtual identities and acquired digital goods that many players think they own, but in reality they don’t. The gaming industry is effectively renting out those virtual goods to its players. Web3 is essentially a decentralised version of the Internet. It shifts digital ownership from tech companies to users so that they can move freely across platforms in an open meta-world. For gaming users in general, the most immediate change that the Web3 reform in the gaming industry has brought about for them is the ownership of their digital assets. With the support of blockchain technology, all game data held by players' NFTs will be permanently recorded, this data will also increase the value of the NFT itself. As a result, the value of the digital assets of quality players will continue to increase as the number of plays increases. From another perspective, blockchain technology offers more possibilities for gamers to stretch their assets. In traditional online games, the use of players' in-game assets is limited to in-game scenarios only. These assets are difficult to transfer. Even if a few players trade their game accounts with others, such transactions have significant limitations. Most of the time the price of the asset during the P2P transactions is much lower than the cost invested by the player. In the metaverse, players' tokens and NFT assets have more liquidity. Gamefi's inherent financial properties can help investors sell, rent out their gaming assets or place their passes in the pool as liquidity providers (LPs), thus maximising capital efficiency. In the metaverse, people are free to travel through the gaming, social, and artistic worlds unhindered by their digital assets as independent individuals. As the use of NFT expands, the cross-disciplinary NFT will provide more possibilities for people's metaverse lives. So the significance of the construction of web3 games is not only on a technical level but also as a bridge between the future entertainment life and social interaction. How can Esport change the Gamefi ecosystem? Before we talk about Game 2.0, let's go back to the p2e economic model that is now most common in web3 games. The fundamental reason why the P2E economic model is not sustainable in the long term is that the single-structured economic model is plagued by "ponzinomics". P2E projects need a constant stream of new investors to consume the output of the previous investors. When the number of "new" entrants does not cover the number necessary to sustain the existing number, it will collapse instantly. As the shortcomings of P2E became apparent, attempts were made to reverse or ameliorate the difficulty of matching supply with demand in the long term by adding structural pillars to the ecosystem. Stepn, for example, introduces a non-monetary benefit output to the model, which is the health improvement. This makes the return on investment not the only consideration for users, but also an important incentive to improve their health after using Stepn consistently. This non-monetary benefit makes this GameFi ecosystem not a zero-sum game, it generates additional positive benefits within the game framework and makes people more willing to be part of the whole ecosystem. Stepn is just one example. There are many elements that can be added to the gamefi ecosystem beyond health, such as social attributes, merchandise, platform-based advertising revenue, etc. But what we must realise is that without a fundamental change in the economic model of P2E's infinite inflation, all attempts will only slow down the project's death spiral. From this perspective, building a new economic system that can be self-circulating becomes the key to gamefi's breakthrough. Esport, as a complete ecosystem, can meet the needs of its audience for competition, education, social interaction and entertainment. For people who like the esport culture, video games are just not a mode of entertainment but are a preferred choice of interaction. In terms of the economic model, esports satisfies the conditions required for a deflationary model, namely the winner-takes-all principle. When a gamefi team set up an e-sports system, their own role changes from that of project operator to that of event organiser, while their revenue changes from a single project fee income to a diversified revenue structure. What should projects be aware of when they want to get into gamefi 2.0? Esport does not apply to all projects Unfortunately, not all commercially available P2E projects can introduce esport; on the contrary, only a very small number of projects are suitable for esport. It is a well-known fact that most gamefi projects are currently not favoured by traditional gamers because the games are not interesting enough and not well made. To meet the requirements of esport, gamefi projects need to be competitive and require a certain level of skill in addition to the two points mentioned above. Even though, esport is not the only way forward for the gamefi project. For some well-known p2e projects, there are many transformative directions that can be taken when they have already accumulated a large amount of user base and high traffic. They can go ahead to build a launchpad or platform, or just add more social functions to the project to support more users’ needs. On another hand, the gamefi teams can also change the revenue model by adding revenue pillars, for example advertising revenue, title revenue, etc. A good tokenomic system is essential In order to support the proper functioning of the gaming system, gamefi teams may refer to the following recommendations when setting up their tokenomic. Regular destruction of tokens is necessary. Whether it is a single-coin model or a two-coin model, the treasury should regularly recycle and destroy tokens in different ways, thus making the supply of tokens inherently deflationary. Increasing the number of scenarios in which tokens are used will prevent the game economy from bubbling. Project owners should consider adding different scenarios when designing their token consumption system so that tokens are well circulated. When valuing in-game NFTs, play records are used as one of the evaluation criteria. This can have several positive effects, as follows. OG players are rewarded for their playing time, and good players can permanently increase the value of their NFT through their superior playing skills Players who buy directly on the secondary market save time and effort The team earn a percentage of royalties Reference: Why play-to-earn gaming is losing steam and where GameFi must go next, RIAZ LALANI https://forkast.news/why-is-p2e-axie-infinity-declining-gamefi/ Animoca Brands: Blockchain Gaming Giant Says Web3 Future Isn’t Far Off, Robert Olsen https://www.forbes.com/sites/robertolsen/2022/09/02/animoca-brands-blockchain-gaming-giant-says-web3-future-isnt-far-off/?sh=6689799d1e09 An Overview of the Gaming Industry and Ways to Invest in It, Prableen Bajpai https://www.nasdaq.com/articles/an-overview-of-the-gaming-industry-and-ways-to-invest-in-it azf.eth https://twitter.com/azfnft/status/1537172472495235080?ref_src=twsrc^tfw|twcamp^tweetembed|twterm^1537172472495235080|twgr^b0faa015f1854d0fe85cd6e91a334e645c03a532|twcon^s1_c10&ref_url=https%3A%2F%2Fabmedia.io%2F20220616-p2e-gaming-sustainable-business Originally posted . here

Gamefi 2.0: The Movement From Play To Earn To Blockchain Esports

We all know Reactjs is a UI library in which we can build user interfaces rapidly. However, Reactjs’ ecosystem is so rich that you can build dynamic interactive web applications just by using some external libraries. In this post, we will discuss 5 Reactjs libraries you can use in your next project. We will discuss its use cases and how it can help you to build your next project without any hiccups. Here is the list of libraries that we are going to discuss: React Flow React Markdown Code Mirror React Draft WYSIWYG React Tippy Let's get started. 1. React Flow Introduction React Flow is a library to build node-based applications and it is built by webkid.io. Use Case Let's say you want to build an application where you want to render node diagrams/flow that could be anything from static to dynamic. Here we can use react flow to render node diagrams/flow and end-user can customize it, can add a new node, can update the existing node, and delete the existing node. Why Use React Flow Over Other Libraries Customizable. Utils-rich. Types support. Fast-rendering. And many more. Live Example Here is a live example of OpenMetadata. 2. React Markdown Introduction React markdown is a markdown parser react component which can take string markdown and render it to the react element. Use Case There could be a variety of use cases for this library. Let's say you are building a blog application and content would be in markdown, so there you can use react markdown to parse the content. We all have used GitHub or a similar application at least once in our developer journey and know that most of the contents are in markdown format. So, we can also build a tool using react markdown to create a GitHub readme profile. Why Use React Markdown Over other Libraries Easy to use. Rich API Support. Custome Component Support. Extendible with plugins system. Live Example Here you can see I have built a using react markdown. You can write markdown and see the output side by side. Also, you can download it as a markdown file. markdown previewer 3. Code Mirror Introduction Code Mirror is a text editor with support for different languages and add-ons to implement the advance and extend the existing functionality. Use Case Let's say you are building an LMS (Learning Management System) and you want to provide an editor to end-users for editing code snippets that could be in any language like python, javascript, etc., or format likes JSON, Avro, SQL, etc. Here you can use code mirror to build an editor that can allow end-users to play with code snippets. Why Use Code Mirror Over other Libraries Support for more than 100 languages Code Folding Support Support for inline and block widget Configurable bindings Used by browsers like Firefox, Chrome, and Safari. Live Example Here you can see the live use case of code mirror by Carbon tool for creating and sharing beautiful images of your source code. 4. React Draft WYSIWYG Introduction React Draft WYSIWYG is a rich text editor built on top of reactjs and .. It’s a ready to use react component to provide rich text support in your application. draftjs Use Case Nowadays most of the communities/publications like , , , and provide support for rich text content and they have built their own rich text editor. But you don't have to reinvent the wheel, you can use React Draft WYSIWYG directly in your application. Hashnode dev.to medium showscase devdojo Why Use React Draft WYSIWYG Over Other Libraries You can configure toolbar options. Support for Mention and Hashtags. Support for emojis. Support for internationalization. Live Example Here you can see the of . live example React Draft WYSIWYG 5. React Tippy Introduction React Tippy is a lightweight tooltip component library for reactjs. It provides a tooltip element or HOC (Higher-order component) and it uses the React DOM to render tooltip. Use Case The use case for React Tippy is simple: if you need to show some content using a tooltip then you should use React Tippy library. Why Use React Tippy Over other Libraries Work's well with reactjs. Enhancement of Tippy.js Support multiple triggers. Customizable. Support for different themes. Support for different sizes. Live Example Here you can find the live example of react tippy And that’s it for this topic. Thank you for reading. Connect With Me | LinkedIn Twitter Also published . here

Discover Why These 5 Libraries are Perfect for Your Next ReactJS Project

What is OpenTelemetry and How It Can Improve Your Backend Quality? 

Wat is OpenTelemetry en hoe dit u agterkantkwaliteit kan verbeter? 

OpenTelemetry ምንድን ነው እና የእርስዎን የኋላ ጥራት እንዴት ማሻሻል ይችላል? 

ما هو OpenTelemetry وكيف يمكنه تحسين جودة الواجهة الخلفية لديك؟

¿Kunas OpenTelemetry ukat Kunjamas Backend ukan calidad ukar askinchaspa? 

OpenTelemetry nədir və O, Backend Keyfiyyətinizi Necə Təkmilləşdirə bilər? 

Што такое OpenTelemetry і як ён можа палепшыць якасць бэкэнда? 

Какво е OpenTelemetry и как може да подобри качеството на вашия бекенд? 

OpenTelemetry কি এবং কিভাবে এটি আপনার ব্যাকএন্ড গুণমান উন্নত করতে পারে? 

Šta je OpenTelemetry i kako može poboljšati vaš backend kvalitet? 

Què és OpenTelemetry i com pot millorar la qualitat del vostre backend? 

Co je OpenTelemetry a jak může zlepšit kvalitu vašeho backendu? 

Hvad er OpenTelemetry, og hvordan det kan forbedre din backend-kvalitet? 

Was ist OpenTelemetry und wie kann es Ihre Backend-Qualität verbessern?

Τι είναι το OpenTelemetry και πώς μπορεί να βελτιώσει την ποιότητα του Backend σας; 

¿Qué es OpenTelemetry y cómo puede mejorar la calidad de su backend? 

Zer da OpenTelemetry eta nola hobetu dezake zure backend kalitatea? 

OpenTelemetry چیست و چگونه می تواند کیفیت Backend شما را بهبود بخشد؟ 

Mikä OpenTelemetry on ja miten se voi parantaa taustasi laatua? 

Qu'est-ce qu'OpenTelemetry et comment peut-il améliorer la qualité de votre backend ? 

Que é OpenTelemetry e como pode mellorar a calidade do teu backend? 

מהי OpenTelemetry וכיצד היא יכולה לשפר את איכות הקצה האחורי שלך? 

ओपनटेलीमेट्री क्या है और यह आपकी बैकएंड गुणवत्ता को कैसे सुधार सकती है?

Što je OpenTelemetry i kako može poboljšati vašu pozadinsku kvalitetu? 

Ki sa ki OpenTelemetry ak ki jan li ka amelyore kalite backend ou a? 

Mi az az OpenTelemetry, és hogyan javíthatja a háttérrendszer minőségét? 

Ի՞նչ է OpenTelemetry-ն և ինչպես այն կարող է բարելավել ձեր Backend-ի որակը: 

Apa itu OpenTelemetry dan Bagaimana Itu Dapat Meningkatkan Kualitas Backend Anda?

Che cos'è OpenTelemetry e come può migliorare la qualità del tuo backend?

OpenTelemetry とは何ですか? また、バックエンドの品質をどのように向上できますか?

რა არის OpenTelemetry და როგორ შეუძლია გააუმჯობესოს თქვენი Backend-ის ხარისხი? 

OpenTelemetry дегеніміз не және ол сіздің сервер сапасын қалай жақсарта алады? 

តើអ្វីជា OpenTelemetry និងរបៀបដែលវាអាចធ្វើអោយគុណភាព Backend របស់អ្នកប្រសើរឡើង? 

OpenTelemetry란 무엇이며 어떻게 백엔드 품질을 향상시킬 수 있습니까? 

OpenTelemetry деген эмне жана ал сиздин Backend сапатын кантип жакшыртат? 

OpenTelemetry ezali nini mpe Ndenge nini ekoki kobongisa qualité ya Backend na yo? 

OpenTelemetry ແມ່ນຫຍັງ ແລະມັນສາມາດປັບປຸງຄຸນນະພາບ Backend ຂອງເຈົ້າໄດ້ແນວໃດ? 

Kas yra „OpenTelemetry“ ir kaip ji gali pagerinti jūsų fono kokybę? 

Kas ir OpenTelemetry un kā tas var uzlabot jūsu aizmugursistēmas kvalitāti? 

Inona no atao hoe OpenTelemetry ary ahoana no hanatsarany ny kalitaon'ny backend anao? 

Што е OpenTelemetry и како може да го подобри квалитетот на вашиот заднина? 

OpenTelemetry гэж юу вэ, энэ нь таны арын хэсгийн чанарыг хэрхэн сайжруулах вэ? 

Apakah OpenTelemetry dan Bagaimana Ia Boleh Meningkatkan Kualiti Bahagian Belakang Anda? 

OpenTelemetry के हो र यसले तपाइँको ब्याकएन्ड गुणस्तर कसरी सुधार गर्न सक्छ? 

Wat is OpenTelemetry en hoe kan het de kwaliteit van uw backend verbeteren?

OpenTelemetry ke eng le gore e ka kaonafatša bjang Boleng bja gago bja Backend? 

Czym jest OpenTelemetry i jak może poprawić jakość Twojego zaplecza?

OpenTelemetry څه شی دی او دا څنګه کولی شي ستاسو د شالید کیفیت ښه کړي؟ 

O que é OpenTelemetry e como ele pode melhorar a qualidade do back-end? 

¿Imataq OpenTelemetry chaymanta Imaynatataq Backend calidadniyki allinchayta atin? 

Ce este OpenTelemetry și cum vă poate îmbunătăți calitatea backend-ului? 

Что такое OpenTelemetry и как она может улучшить качество вашей серверной части? 

OpenTelemetry niki kandi nigute ishobora kuzamura ireme ryinyuma yawe? 

OpenTelemetry යනු කුමක්ද සහ එය ඔබගේ පසුපෙළ ගුණාත්මකභාවය වැඩි දියුණු කරන්නේ කෙසේද? 

Čo je OpenTelemetry a ako môže zlepšiť kvalitu vášho backendu? 

Chii chinonzi OpenTelemetry uye Sei Inogona Kuvandudza Yako Kumashure Hunhu? 

Waa maxay OpenTelemetry iyo sidee ayay u hagaajin kartaa tayadaada dambe? 

Çfarë është OpenTelemetry dhe si mund të përmirësojë cilësinë tuaj të backend? 

Шта је ОпенТелеметри и како може да побољша ваш позадински квалитет? 

Vad är OpenTelemetry och hur det kan förbättra din backend-kvalitet? 

OpenTelemetry ni nini na Jinsi Inaweza Kuboresha Ubora wako wa Nyuma? 

OpenTelemetry என்றால் என்ன, அது உங்கள் பின்தள தரத்தை எவ்வாறு மேம்படுத்தலாம்? 

OpenTelemetry чист ва он чӣ гуна метавонад сифати пуштибонии шуморо беҳтар кунад? 

OpenTelemetry คืออะไร และสามารถปรับปรุงคุณภาพแบ็กเอนด์ของคุณได้อย่างไร

OpenTelemetry እንታይ እዩ ብኸመይ ንጽሬት ባክኤንድካ ከማሓይሽ ይኽእል? 

“OpenTelemetry” näme we arkaňyzyň hilini nädip ýokarlandyryp biler? 

Ano ang OpenTelemetry at Paano Ito Mapapabuti ang Kalidad ng Iyong Backend? 

OpenTelemetry Nedir ve Arka Uç Kalitenizi Nasıl Artırabilir? 

Xana OpenTelemetry I Yini Naswona Yi Nga Antswisa Njhani Khwalithi Ya Wena Ya Backend? 

Що таке OpenTelemetry і як він може покращити якість серверної частини? 

OpenTelemetry کیا ہے اور یہ آپ کے بیک اینڈ کوالٹی کو کیسے بہتر بنا سکتا ہے؟ 

OpenTelemetry nima va u sizning orqa tomoningiz sifatini qanday yaxshilashi mumkin? 

OpenTelemetry là gì và nó có thể cải thiện chất lượng phụ trợ của bạn như thế nào? 

Yintoni i-OpenTelemetry kwaye inokuphucula njani umgangatho wakho wokubuyela umva? 

什么是 OpenTelemetry 以及它如何提高您的后端质量？

Iyini i-OpenTelemetry futhi Ingayithuthukisa Kanjani Ikhwalithi Yakho Yasemuva? 

is a distributed unit of infrastructure. It can provide information about FreeBSD and Linux operating systems. It also supports metrics transmission to Clickhouse, InfluxDB, and Prometheus. Alligator The main interface of alligator is . It connects with spreading Prometheus as a monitoring system in many infrastructures. However, the Prometheus ecosystem has some problems. OpenMetrics First of all - the specialized and simplified agents. Every service on the host needs a specialized exporter. This is the reason for having a lot of exporters per host (like a node_exporter, redis_exporter, sentinel_exporter, and so on). Secondly, simplified agents. A lot of agents can be used for different tasks (like an SNMP, blackbox, etc.), and it inflates infrastructure for such daemons to collect information. Sometimes, a pull model has problems with multi-worker applications and cron jobs - nothing can give metrics from hosts when Prometheus makes a request, or it is connected with problems related to synchronizing results between workers. It forces us to use statsd/pushgateway. Then the question comes up: how to remove outdated metrics? How to make it a high-availability service? The approach of the alligator is different - it is a smart agent for the Prometheus. Alligator consists of a system exporter and an application exporter. It also provides a push model for pushgateway, statsd, and graphite emulation protocol. Metrics Export Alligator supports such sets of metrics: FreeBSD and GNU/Linux system metrics. Filesystem metrics (file’s checksum, sizes, and access level). Certificate metrics (PEM and P12). Some popular applications (ElasticSearch, uwsgi, Druid, RabbitMQ, and others). Ephemeral metrics. It supports pushgateway, statsd, and graphite protocol, and it can collect metrics from file on FS. It also has a cluster mode for centralized installation of pushgateway. Simple JSON to metrics parser. Simple HTTP/TCP checker. Use Cases A part of supported technologies for collecting metrics are reviewed in the . The rest I am describing here. official documentation Application Server Metrics First of all, alligator can collect application server metrics. For instance, alligator can gather metrics from uwsgi instances with this configuration: aggregate {
    uwsgi tcp://localhost:1717;
} and rerun the uwsgi server with the : expose stats port --stats 127.0.0.1:1717 Another example, fastcgi-application: aggregate {
    php-fpm fastcgi://127.0.0.1:9000/stats?json&full;
} and write this option in : php-fpm config pm.status_path = /stats After this, you will have common metrics from the application server. Application Metrics I recommend using synchronization structures to collect metrics in the application internally. Alternatively, you can make only 1 worker per instance and use such an abstraction as orchestration (Kubernetes, opennebula, and others). And next, use alligator as a push server for metrics. Alligator has many collectors to collect it: pushgateway prom-aggregation-gateway statsd graphite file These approaches ensure that metrics will be collected from all workers of the application Applications have to send metrics to alligator. To remove influence on one metric by different workers, you can use two different approaches: mark metric with the worker ID from prometheus_client import Counter, push_to_gateway, CollectorRegistry
    registry = CollectorRegistry()
    c = Counter('my_requests_total', 'HTTP Failures', ['method', 'worker_id'], registry=registry)
    c.labels(method='get', worker_id=uwsgi.worker_id()).inc()
    c.labels(method='post', worker_id=uwsgi.worker_id()).inc()
    push_to_gateway('localhost:1111', job='myapp', registry=registry) use instead of metric type if you can. File collector cannot calculate increment, but others can increment if the counter is a type of metric. You can only send increment value to alligator, and alligator will calculate the sum. Counter Gauge More information about the difference between metric types can be found or on the . here official page What Metrics Do I Recommend to Measure in Applications? Firstly, it is any outgoing requests to external systems. In worker applications, it often is the reason to overflow workers or cause of low speed of query processing. My recommended list of metrics: count of requests to external services grouped by request types request time from external services (avg, or if possible, by quantiles/buckets of time) grouped by request types status of requests grouped by request type (success or errors) It would be nice if you could group not only by request type but more by network layer of requests: DNS resolve count, time, and status connect count, time, and status write/read count, time, and status It is also important to measure internal application metrics: ingoing requests grouped by types processing ingoing request time avg (quantile or buckets if possible) by type status of ingoing request by type cache status (hit/miss if it exists) I guess you can make another list of metrics, but these metrics are the most important to reveal problems and give main observability about application health. If your application is working in podman or docker environment, you can expose the container statistics. Alligator mechanics are similar to Cadvisor's when collecting container statistics. To enable, you have to use option cadvisor in the system context: Container checks. system {
    cadvisor;
} . This is a good way to monitor applications or external sources for reliability. Let’s check the resources on two ports every minute: Blackbox checks aggregate_period 60;
aggregate {
    blackbox http://127.0.0.1:80/check;
    blackbox https://127.0.0.1:443/check;
} Alligator produces network and application response metrics for this resource (response code, response time, and certificate expiration date). If the application has a page with JSON, alligator can also convert JSON to metrics: aggregate {
    jsonparse https://jsonplaceholder.typicode.com/todos;
} The next level of blackbox metrics - checking the process, service, and listening port: system {
	services uwsgi.service;
	process uwsgi;
}


query {
	expr 'count by (src_port, process) (socket_stat{process="uwsgi", src_port="80"})';
	make socket_match;
	datasource internal;
} And in the end: launching external programs and proactive actions (like watchdog): aggregate {
	process 'exec:///usr/local/bin/mycheckscript.sh';
} If the script can produce a metric that can determine the exact application downtime, this metric can be used for proactively restart: query {
    expr 'myapplication_status < 1';
    make app_is_down;
    datasource internal;
    action app_is_down;
}
action {
    name app_is_down;
    type shell;
    expr 'exec://systemctl restart uwsgi;
    datasource internal;
} System Metrics Any application metrics are also important. But without system metrics of servers, you won’t be able to clearly understand what happens in your system. System metrics are a set of metrics that help us gain insight into the health of our systems. This is the basis for making decisions if something goes wrong. An alligator with an empty configuration doesn’t collect any metrics at all. Any system metrics are classified into the following categories: base - CPU, memory, time, logical resources, and other common metrics about the server disk - disk usage and I/O network - interface statistics, network counters, sockets statistics smart - collecting S.M.A.R.T. statistics cpuavg - analog for load average when alligator make by CPU-only load firewall - counters from the system firewall process - check process (uptime, ulimits usage, status, CPU, memory, and disk consumption) packages - metrics with a list of installed packets in the system services - systemd services stats This is configured in a block called “system” in alligator.conf: system {
    base;
    disk;
    network;
    process;
    cpuavg period=5;
} What about cron jobs? How can it be connected to the Prometheus via alligator? There are several ways to do this. The simple way is to output metrics to a file: aggregate {
    prometheus_metrics file:///var/lib/stats/metrics.txt state=save;
} The parameter called ‘state’ enables autosaving at a point of the file when alligator is stopped. It makes us sure that we can stop the alligator and run again, and it rereads only that part of the file that has not been read yet. If we want to reread a file from the start (every 10 sec), we have to use . The third way - is to read only realtime data. To enable this mode, we have to remove any state from the configuration file. state=begin The other way is to use statsd or extended pushgateway protocol (called prom-aggregation-gateway): It only needs to have this section in the configuration: entrypoint {
	tcp 1111;
    metric_aggregation count;
} For instance, we have a code: #!/bin/sh
TASKS=0
do_something() {
        echo 0
        TASKS=$1
}

res=`do_something 1`
echo "amount_of_done_tasks {method=\"do_something\", status=\"done\"} $TASKS
task_exit_code {method=\"do_something\"} $res" | curl --data-binary @- http://localhost:1111/ This script will make two metrics in alligator: amount_of_done_tasks {status="done", method="do_something"} 0.000000
task_exit_code {method="do_something"} 0.000000 Sometimes, we might get a situation when the script is not working, but the metric exists (because alligator doesn’t know that the application doesn't work). To solve this problem, alligator has TTL mechanics. You can specify the lifetime in seconds for a request: curl -H "X-Expire-Time: 30" --data "living_thirty_seconds_metric 4" http://localhost:1111/ Or in the configuration file: entrypoint {
    ttl 30;
    tcp 1111;
    metric_aggregation count;
} The next way is using statsd protocol (udp or tcp). TCP server for statsd is enabled by default. If you want to use udp, specify the UDP port in config: entrypoint {
        ttl 30;
        tcp 1111;
        udp 1111;
} And your job script might have a section like this: from statsd import StatsClient, TCPStatsClient
from time import sleep

statsd = StatsClient(host='localhost',
                     port=1111,
                     prefix=None,
                     ipv6=False)

statsd.incr('baz')
statsd.incr('baz') Graphite protocol is also supported by alligator, and it works like graphite_exporter. Internet Web Server If I have an internet web server, what options do I have? First of all, https is the most popular protocol for web servers today. It forces us to check the certificate expiration time: To check directories with certificates: x509
{
	name nginx;
	path /etc/nginx/ssl/;
	match .crt;
}
x509
{
	name letsencrypt;
	path /etc/letsencrypt/live;
	match .pem;
} Secondly, to check the nginx listening ports: query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="80"})';
	make socket_match;
	datasource internal;
}
query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="443"})';
	make socket_match;
	datasource internal;
} To check nginx -t to exclude the situation, when nginx after server reboot cannot be started: aggregate {
	process 'exec:///sbin/nginx -t';
} To check system, web-server process, and firewall counters system {
    base;
    disk;
    network;
    cpuavg period=5;
    services nginx.service;
    process nginx;
    firewall ipset=on;
} We also make alligator port accessible only on local interfaces: entrypoint {
    handler prometheus;
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    tcp 1111;
} or you can use the notation tcp in the tcp directive. <localIP>:<port> Statsd Metric Mapping Simple statsd doesn’t work with labels. To create a label from a statsd metric name, you must use mapping with metric name templates to change new names and create new labels. Due to this, I recommend using an extended format of Statsd, like DogStatsd (alligator also supports this extension), which supports labels in statsd protocol. But, some programs have already made pure statsd protocol without labels. One such example is Airflow. You can catch metrics from Airflow like this: airflow.ti.finish.upload_job.load_data.success:1|c
airflow.ti.finish.upload_job.other_data.success:1|c where the name is the following template: airflow.ti.finish.<job name>.<process name>:<increment by>|type The mapping for such metric name can be: entrypoint {
    tcp 1111;
    udp 1111;
    mapping {
        template   airflow.ti.finish.*.*.*;
        label dag_id "$1;
        label task_id "$2";
        label state "$3";
        name airflow_ti_finish;
        match glob;
    }
} This mapping will create such a metric: airflow_ti_finish {state="success", dag_id="upload_job", task_id="load_data"} 1.000000
airflow_ti_finish {state="success", dag_id="upload_job", task_id="other_data"} 1.000000 Statsd or Pushgateway Cluster Firstly, you can only make some amount of servers statsd or pushgateway and simply proxying to them. The first problem with using this: if you want to use gauges, you will have problems with calculating. If there is a distributed application sending a metric with the same name and labels, the gauge metric will also be replaced by the application. The first way to fix this - is to specify different labels for each application (such as hostname and/or worker_id/process_id). But that is not all. You also have a problem with the same gauge metric: this metric can be a balancer on different machines. Here is what to do with a metric that was sent at different times, but now they are interfering with each other: 10:12 http_request_time_avg {appid="43"} 12
10:14 http_request_time_avg {appid="43"} 22
10:55 http_request_time_avg {appid="43"} 122
11:58 http_request_time_avg {appid="43"} 1232 You can use nginx upstreams with a balancing method called hash: upstream alligators {
    server srv1.example.com:1112;
    server srv2.example.com:1112;
    server srv3.example.com:1112;
    server srv4.example.com:1112;
    server srv5.example.com:1112;
    hash $remote_addr consistent;
} Then one backend will always push all metrics to one server. On the other hand - cluster mode in alligator. Let's set it up! Firstly, we allocated 3 ports: for getting metrics 1111 for replication works 1112 for getting metrics from other applications 80 Imagine that our server is listening to the internet. Then, we need to secure our push metric ports. We can use the directive “return” with arg “empty” to remove any answers outside. We also know that internet browsers don't send POST queries to other domains if CORS doesn't allow this. Thirdly, make an authentication for POST requests. entrypoint {
   allow 127.0.0.1;
   allow 10.0.0.0/8;
   tcp 1111;
}
entrypoint {
   allow 10.0.0.0/8;
   tcp 1112;
   cluster replication;
   instance srv1.example.com:1112;
}
entrypoint {
   return empty;
   allow 0.0.0.0/0;
   ttl 300;
   tcp 80;
   metric_aggregation count;
   auth basic user:setme;
   auth_header Authorization;
   header access-control-allow-headers 'Authorization';
   header access-control-allow-methods POST;
   header access-control-allow-origin *;
} That's good. We annotate that we have a cluster called “replication” and the current instance name. Since this moment, alligator allows to answer collected oplog to anyone from network 10.0.0.0/8 on port 1112. How to command an alligator to join other instances of cluster? For this, we will use the “cluster” directive. We can specify oplog size (the limit of in-memory stored metrics that will wait to sync to other instances), and distribute metrics by its metric name (sharding key in our case, but you can use any other label). The replication factor hides replicated metrics while another primary server is running in the cluster. If the primary server goes down, hidden metrics will be exposed and made available to Prometheus from other instances. cluster {
   name replication;
   size 10000;
   sharding_key __name__;
   replica_factor 2;
   type oplog;
   servers srv1.example.com:1112 srv2.example.com:1112 srv3.example.com:1112 srv4.example.com:1112 srv5.example.com:1112;
} Check the External Sites Status Blackbox mechanics for collecting information about network services have already been mentioned. Nevertheless, that is not all that alligator can be used for. Alligator also has a puppeteer parser for loading external pages with a lot of site resources (which can load a page DOM). puppeteer {
    https://site1.example.com;
    https://site2.example.com;
    https://example.com;
} They might measure domain lookup name in metric: puppeteer_domainLookupEnd - puppeteer_domainLookupStart Connect time: puppeteer_connectEnd - puppeteer_connectStart Time to load DOM: puppeteer_DomContentLoaded Status of loading: puppeteer_eventSourceResponseStatus And even errors in the browser console: puppeteer_eventConsole This is not all that a puppeteer collector can do. However, this engine needs nodejs to be installed on a machine with puppeteer dependencies. Install puppeteer dependency: cd /var/lib/alligator/
npm i puppeteer@14.3.0
npm i ps-node-promise-es6 Conclusion Monitoring systems is a really big challenge despite the amount of tools to solve it. You can choose between a monitoring system or tools in one system of monitoring (such as VictoriaMetrics, Thanos, some amount of exporters, and so on). I guess that the approaches outlined in the article can be applicable to any other tools of monitoring. Alligator is a tool that can simplify it. The main goal of any monitoring is to make your system observable. You must have all measured information from your systems to ensure that it always operates correctly.

Alligator Is a Prometheus Monitoring Agent: Everything You Need to Know About It

Stories, News & Mentions about OPENMETA

Categories

Trending Topics

Stories, News & Mentions about OPENMETA

Openmeta’s Stories on HackerNoon

Categories

Trending Topics