RNT Associates has offices in Mumbai and Singapore and has invested in startups such as ikure, cult.fit and Greaves Electric Mobility according to Tracxn data. An email sent to Tata did not elicit a reply till the time of going to press. Agencies Mehli ...

Mehli Mistry joins board of Ratan Tata's investment firm

JSW Ventures on Monday said it has sold its shareholding through Fund I in e-commerce firm Purplle to Manipal Group. The company will continue its investment in Purplle through the second fund, JSW Ventures said in a statement.

JSW Ventures fully exits Purplle, sells stake to Manipal Group’s Ranjan Pai

Pixxel, Agnikul Cosmos, Satsure, Satyukt Analytics and PierSight – raised venture funds in the last five months

62% jump in funding to Space tech startups in 2023

A majority share -- $45.2 million -- of the total funding went to early-stage startups, making up 57% of the total weekly funding. Late-stage startups saw $30 million being pumped in as investments, while seed-stage companies received a mere $4.

ETtech Deals Digest: Startups raise $80 million funding in yet another dry week

The drone technology developed by the company caters to different industries and operations such as survey, surveillance, agriculture, AI-based inspection, and defense.

Drone startup UrbanMatrix raises Rs 6 crore funding led by Inflection Point Ventures

With venture debt funds shying away from subscribing to equity warrants at high valuations, most start-ups are closing out pure-debt deals at high coupon rates. “Start-ups are taking on debt at higher than the usual 14-15.5%. Interest rates on these can go as high as 18-20%,”said an investor.

Venture debt deals largely loan-driven

Legacy footwear brand Inc.5 Shoes raised its maiden venture funding of $10 million (about Rs 83 crore) in a round led by Mumbai-based venture firm Carpediem Capital, with participation from Param Capital and P3 Venture Fund.

Footwear brand Inc.5 raises maiden institutional funding of $10 million

Exponent Energy had last raised $13 million in August 2022 in a Series A led by Lightspeed that valued the Bengaluru-based startup at around $45 million.

EV company Exponent eyes $25 million at $100-120 million valuation

While Thailand’s healthcare sector has been witnessing a rise in HealthTech startups, with approximately 270 ventures currently operating in the field according to Tracxn data, the agricultural sector has lagged behind. Surprisingly, there are only 53 ...

Thailand’s Ambitious Startup Project Aims to Boost Innovation in Agriculture and Healthcare

Indian startups secured $146 million in funding across 22 rounds this week, down 70% from last years $494 million across 46 rounds, according to market research firm Tracxn.

ETtech Deals Digest: Early-stage deals in focus as startup funding remains muted at $146 million this week

Tracxn

tracxn

India’s Space Tech sector is set for remarkable growth, owing to significant government investment. The recent Union Budget has allocated INR 1000 crore to support space technology startups, aimed at boosting the sector’s development and attracting further investment. This initiative, along with the establishment of the Indian National Space Promotion and Authorization Centre (IN-SPACe) in 2022, is expected to drive substantial innovation and growth. As per Tracxn’s Space Tech Geo Report 2024 India’s space tech sector witnessed a record-breaking funding of $126 million in the year 2023, a 7% increase from the $118 million raised in 2022 and a 235% increase from $37.6 million in 2021. In 2024, the year-to-date funding stands at $10.8 million. Despite a global slowdown in funding, the Indian space tech sector has been experiencing an upward trend, driven by substantial government support and significant innovations. India has emerged as a formidable player in the global space technology arena, with more than 100 space tech startups, the majority of which were founded in the past five years. Government support has been a critical accelerant for this growth. The Union Budget’s funding of 1000 Crore is expected to boost the industry significantly, attracting more entrepreneurship and investment in the coming months. Currently, India boasts 55 active space assets, including communication, meteorological, and earth observation satellites. The Indian Space Tech sector’s funding landscape is predominantly driven by early-stage investments. In 2023, early-stage rounds attracted $120 million of the total $126 million raised, reflecting a notable growth from $114 million in 2022, which represents a 5% increase. In 2024, early-stage funding has reached $8.5 million to date. Seed-stage funding has also seen a significant rise, growing from $4.3 million in 2022 to $5.3 million in 2023, marking a 24% increase. However, despite this growth in early-stage and seed-stage funding, the nascent ecosystem for private sector participation in Indian Space Tech startups has not yet experienced any late-stage funding. Skyroot Aerospace leads as the highest-funded active space tech startup in India, with an overall funding of $99.8 million, followed by Pixxel at $71.7 million and Agnikul at $61.5 million. No other startup in this segment has raised funds above $50 million. The top-funded business models in this space include Small Satellite Launch Vehicles, Satellite Imaging Services, and Satellite Communication Services. Notably, the Small Satellite Launch Vehicle segment has attracted $168 million to date, with Skyroot raising the highest funding in this category. Despite the vibrant funding landscape, no acquisitions have been observed in 2024 YTD. The only acquisition to date is the 2022 purchase of Prakshep, a satellite imagery provider for the agricultural industry, by Arya. Additionally, the sector has yet to witness the emergence of unicorns. MTAR and Ananth Technologies are the only public companies in this space. We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology
Neha Singh, Co-Founder of Tracxn Neha Singh, Co-Founder of Tracxn, saidg, “India’s Space Tech sector is growing rapidly, thanks to strong government support and a dynamic startup scene. The significant funding and strategic investments we’re seeing now are setting the stage for India to become a major player in the global space industry. Looking ahead, we expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology.” The privatization of the space sector has catalyzed a multifold increase in activity. The International Astronautical Federation awarded India the prestigious World Space Award for its successful launch of Chandrayaan 3, which made history in 2023 as the first mission to achieve a soft landing on the moon’s south pole. This landmark achievement has set a global standard for India’s potential and cost-effective engineering in the space tech industry. Bangalore leads the list of top-funded cities in India’s space tech sector, followed by Hyderabad and Chennai. Startups based in Bangalore have contributed to more than 55% of the total funding in this space, and it is also the city with the highest number of space tech startups. Among the top investors in the sector are Speciale Invest, Anicut Capital, and GrowX Ventures. Anicut Capital, LetsVenture, and Rosmerta Technologies have been the leading seed-stage investors over the last two years, while Ban Labs, Anicut Capital, and Artha Venture Fund have emerged as the top early-stage investors during the same period. The evolution of the space industry highlights a global shift from government control to private-sector collaboration. The United States, a pioneer in privatizing its space industry with the 1984 Commercial Space Launch Act, has significantly benefited from private company contributions through NASA’s Artemis program. Federal funding for NASA has consistently increased, with $24.8 billion allocated in the 2024 budget. In terms of geographic funding, the US remains a leader in space tech startup funding, although it has seen a decline from $5.5 billion in 2021 to $660 million in 2024. Europe, on the other hand, has seen increased funding in its space tech sector, with $407 million raised in 2024 to date, following a record $512 million in 2023. The UK, while experiencing fluctuations in funding, shows signs of recovery with $135 million raised in 2024 to date. Similarly, the UK has invested €50 million in its Positioning, Navigation, and Timing (PNT) sector since 2017, employing a hybrid model that leverages private technologies for space missions. In recent years, the European Space Agency (ESA) has shifted to a competition-based model, inviting more private players to develop and launch space missions. This strategic shift, driven by a series of delays and failures in rocket launches, aims to attract private investment and boost entrepreneurship in the region. Despite the existence of the ESA, countries like Germany, Italy, and France maintain their space research agencies, working in collaboration with the ESA. The ESA’s budget for 2024 is $8.3 billion, a 10% increase from the previous year, with significant contributions from Germany, France, and Italy. A major portion of this budget is allocated to Earth Observation operations (30%), followed by space transportation and navigation. India’s Space Tech sector is also growing rapidly, fueled by government support and a dynamic startup ecosystem. Significant funding and strategic investments are positioning India to become a major player in the global space industry, with expectations of attracting more investment and sparking further innovation. Navanwita Bora Sachdev, Editor, The Tech Panda India’s Space Tech sector is set for remarkable growth, owing to significant government investment. Space Tech The recent Union Budget has allocated INR 1000 crore to support space technology startups, aimed at boosting the sector’s development and attracting further investment. This initiative, along with the establishment of the Indian National Space Promotion and Authorization Centre ( IN-SPACe ) in 2022, is expected to drive substantial innovation and growth. allocated IN-SPACe As per Tracxn’s Space Tech Geo Report 2024 India’s space tech sector witnessed a record-breaking funding of $126 million in the year 2023, a 7% increase from the $118 million raised in 2022 and a 235% increase from $37.6 million in 2021. In 2024, the year-to-date funding stands at $10.8 million. Despite a global slowdown in funding, the Indian space tech sector has been experiencing an upward trend, driven by substantial government support and significant innovations. Tracxn’s Space Tech Geo Report 2024 India has emerged as a formidable player in the global space technology arena, with more than 100 space tech startups, the majority of which were founded in the past five years. Government support has been a critical accelerant for this growth. The Union Budget’s funding of 1000 Crore is expected to boost the industry significantly, attracting more entrepreneurship and investment in the coming months. Currently, India boasts 55 active space assets, including communication, meteorological, and earth observation satellites. The Indian Space Tech sector’s funding landscape is predominantly driven by early-stage investments. In 2023, early-stage rounds attracted $120 million of the total $126 million raised, reflecting a notable growth from $114 million in 2022, which represents a 5% increase. In 2024, early-stage funding has reached $8.5 million to date. Seed-stage funding has also seen a significant rise, growing from $4.3 million in 2022 to $5.3 million in 2023, marking a 24% increase. However, despite this growth in early-stage and seed-stage funding, the nascent ecosystem for private sector participation in Indian Space Tech startups has not yet experienced any late-stage funding. Skyroot Aerospace leads as the highest-funded active space tech startup in India, with an overall funding of $99.8 million, followed by Pixxel at $71.7 million and Agnikul at $61.5 million. No other startup in this segment has raised funds above $50 million. The top-funded business models in this space include Small Satellite Launch Vehicles, Satellite Imaging Services, and Satellite Communication Services. Notably, the Small Satellite Launch Vehicle segment has attracted $168 million to date, with Skyroot raising the highest funding in this category. Despite the vibrant funding landscape, no acquisitions have been observed in 2024 YTD. The only acquisition to date is the 2022 purchase of Prakshep, a satellite imagery provider for the agricultural industry, by Arya. Additionally, the sector has yet to witness the emergence of unicorns. MTAR and Ananth Technologies are the only public companies in this space. We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology Neha Singh, Co-Founder of Tracxn We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology We expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology Neha Singh, Co-Founder of Tracxn Neha Singh, Co-Founder of Tracxn, saidg, “India’s Space Tech sector is growing rapidly, thanks to strong government support and a dynamic startup scene. The significant funding and strategic investments we’re seeing now are setting the stage for India to become a major player in the global space industry. Looking ahead, we expect the sector to attract even more investment, sparking more innovation. The combination of solid early-stage funding and supportive government policies will be key to driving this growth and establishing India as a leading center for space exploration and technology.” The privatization of the space sector has catalyzed a multifold increase in activity. The International Astronautical Federation awarded India the prestigious World Space Award for its successful launch of Chandrayaan 3 , which made history in 2023 as the first mission to achieve a soft landing on the moon’s south pole. This landmark achievement has set a global standard for India’s potential and cost-effective engineering in the space tech industry. awarded Chandrayaan 3 Bangalore leads the list of top-funded cities in India’s space tech sector, followed by Hyderabad and Chennai. Startups based in Bangalore have contributed to more than 55% of the total funding in this space, and it is also the city with the highest number of space tech startups. Among the top investors in the sector are Speciale Invest, Anicut Capital, and GrowX Ventures. Anicut Capital, LetsVenture, and Rosmerta Technologies have been the leading seed-stage investors over the last two years, while Ban Labs, Anicut Capital, and Artha Venture Fund have emerged as the top early-stage investors during the same period. The evolution of the space industry highlights a global shift from government control to private-sector collaboration. The United States, a pioneer in privatizing its space industry with the 1984 Commercial Space Launch Act , has significantly benefited from private company contributions through NASA’s Artemis program . Federal funding for NASA has consistently increased , with $24.8 billion allocated in the 2024 budget. In terms of geographic funding, the US remains a leader in space tech startup funding, although it has seen a decline from $5.5 billion in 2021 to $660 million in 2024. 1984 Commercial Space Launch Act NASA’s Artemis program consistently increased Europe, on the other hand, has seen increased funding in its space tech sector, with $407 million raised in 2024 to date, following a record $512 million in 2023. The UK, while experiencing fluctuations in funding, shows signs of recovery with $135 million raised in 2024 to date. Similarly, the UK has invested €50 million in its Positioning, Navigation, and Timing (PNT) sector since 2017, employing a hybrid model that leverages private technologies for space missions. invested In recent years, the European Space Agency (ESA) has shifted to a competition-based model, inviting more private players to develop and launch space missions. This strategic shift, driven by a series of delays and failures in rocket launches, aims to attract private investment and boost entrepreneurship in the region. Despite the existence of the ESA, countries like Germany, Italy, and France maintain their space research agencies, working in collaboration with the ESA. The ESA’s budget for 2024 is $8.3 billion, a 10% increase from the previous year, with significant contributions from Germany, France, and Italy. A major portion of this budget is allocated to Earth Observation operations (30%), followed by space transportation and navigation. shifted with significant contributions India’s Space Tech sector is also growing rapidly, fueled by government support and a dynamic startup ecosystem. Significant funding and strategic investments are positioning India to become a major player in the global space industry, with expectations of attracting more investment and sparking further innovation. Navanwita Bora Sachdev, Editor, The Tech Panda Navanwita Bora Sachdev, Editor, The Tech Panda Navanwita Bora Sachdev , Editor, The Tech Panda Navanwita Bora Sachdev

India's Space Tech Sector Soars with Unprecedented Funding

2023 might have been India’s lowest-funded year in last 5 years as the funding winter continues, but the decade shows good progress. A tech-savvy young population, Internet penetration, and an expanding middle class have led to an explosion of startups in India in the past decade On National Startup Day, that India witnessed an unprecedented surge in the creation and funding of startups in the last ten years. The country has solidified its position as a major global center for innovation and entrepreneurship with a rise in the number of tech companies in India to 122k to date, with a peak in 2020, witnessing more than 16K new additions. The funding landscape has also experienced a monumental shift, with 2020 and 2021 being the highest investment years over the past decade. Tracxn reported Despite the challenges faced in the funding landscape in 2023, investors, including venture capitalists, angel investors, and private equity firms, have shown remarkable confidence in Indian startups. The funding scenario, both from Indian and global investors, has been encouraging, with a total funding of $8.4 billion in India in 2023, demonstrating resilience amid market uncertainties. The startup ecosystem is ready to lead in bringing more innovation, creating jobs, and boosting economic prosperity. We are on the verge of a new era of success, supported by technology, business-friendly regulations, and the strong dedication of our entrepreneurs Neha Singh, co-founder of Tracxn Neha Singh, co-founder of Tracxn, said, “Over the past ten years, India’s startup sector has grown a lot, becoming a global hub for new ideas and economic growth. Entrepreneurs have shown great determination and creativity, contributing significantly to the country’s economic development. Looking ahead, we are optimistic about the future. The startup ecosystem is ready to lead in bringing more innovation, creating jobs, and boosting economic prosperity. We are on the verge of a new era of success, supported by technology, business-friendly regulations, and the strong dedication of our entrepreneurs.” Key Factors to Growth The upsurge in the number of startups in the country can be attributed to several key factors. These include a tech-savvy young population resulting from widespread mobile internet access, the expansion of the middle class, and a favorable regulatory landscape. These elements collectively have fostered a conducive environment for entrepreneurial growth and innovation in the startup ecosystem. Top Performing Sectors Top-funded sectors from 2014 to 2023 include retail, enterprise applications, FinTech, transportation & logistics tech, food and agriculture tech, auto tech, travel & hospitality tech, and EdTech. Factors like increased internet penetration, digitization, and government initiatives have fueled the growth of these sectors. Startups have played a pivotal role in driving Research and Development (R&D), especially in knowledge-intensive sectors. India’s investment in Deep Tech (R&D oriented) has been increasing steadily over the years with total funding of US$6.73B, and in a decade, it received US$6.23 billion, reflecting the country’s steady growth and commitment to science and technology, particularly in Deep Tech, funding has witnessed a substantial increase, reaching $753.7 million in 2023. This trend emphasizes the importance of technology-driven innovation in shaping India’s future. India’s online commerce has grown tremendously, with sectors like Retail Tech receiving substantial funding. The funding trends in retail tech showcase the sector’s resilience, with US$2.3 billion invested in 2023, indicating sustained growth. New Emerging Sectors The addition of new sectors such as DeepTech, SpaceTech, AI, and EVs has broadened the startup landscape. Noteworthy sectors like DeepTech have seen a rise in funding, reaching $2.1 billion in 2022, underlining the market’s confidence in innovative technologies. The EV industry, which attracted investor interest and is one of the booming sectors, has seen $4.8 billion in funding in the past ten years. Even Space Tech has been in demand after its transformation which is credited to privatisation. India ranks seventh in terms of funding within the International SpaceTech landscape to date. The funding reached $114.9 million in 2022 and $122.2 million in 2023 and will continue to grow. Job Creation One of the most significant achievements of the Indian startup ecosystem has been job creation, with approximately 10 lakh jobs generated by startups till 2023, as per Economic Survey 2023. Furthermore, startups focusing on continued learning and professional upskilling have attracted significant investments, reflecting the industry’s commitment to employee empowerment. Top Favorable Cities for Startups City-wise, Bengaluru remains the startup hub, with a total funding of $70.4 billion in a decade, followed by Delhi-NCR and Mumbai. The government’s focus on promoting entrepreneurship is evident, with increased funding across metros and non-metros alike. Several successful startups have emerged in non-metropolitan areas, driving innovation and transforming their regions, such as Jaipur, which has notably secured its position among the top cities in FinTech funding, garnering an impressive $214 million in 2023. Unicorns, Soonicorns, & Minicorns In a significant achievement for the Indian startup ecosystem, it marked a major milestone by welcoming the 100th Indian startup into the unicorn club in 2022. As of now, a total of 111 unicorns have been created in India, with 355 promising ones poised to join the ranks soon. Similarly, the rise of Soonicorns began in 2019, and the year 2022 witnessed an unprecedented surge with the creation of 96 Soonicorns. Additionally, the Minicorn trend, which also boomed in 2019, experienced substantial growth, reaching its pinnacle in 2022 with the emergence of 264 Minicorns. This showcases the dynamic and flourishing nature of the Indian startup ecosystem, with both Soonicorns and Minicorns contributing to the nation’s entrepreneurial landscape. Women-Led Startups Over the past decade, entrepreneurial advancements have transformed India significantly, with women playing a significant role in driving this progress. The addition of women-led startups has also witnessed growth, with total funding reaching $1.1 billion in 2023 from $0.456B in 2014. This trend is exemplified by the fact that twenty out of the total unicorns created to date are women-led startups, highlighting women’s impactful and growing role in shaping the nation’s startup ecosystem. Future Outlook India’s economic growth is intricately tied to the success of its startup ecosystem. India’s economic outlook remains optimistic, with a projected annual growth rate of 6.3% (as per IMF) in 2023–24. The government’s ongoing support, as seen in the 2023 budget with initiatives like tax concessions and the Drone Shakti Program, reinforces the commitment to nurturing the tech ecosystem. Investor confidence in the Indian startup ecosystem has seen a substantial increase over the last decade. Despite the recent funding winter, the number of investors has increased from 2.45k in 2014 to 16.3k in 2021.  The long-term prospects remain favorable, with India poised to become the third-largest economy in the world by 2027, according to the IMF. As the global situation stabilizes, a revival of investment activity is expected in the coming years, reaffirming India’s position as a top destination for investors. This article was originally published by on Navanwita Bora Sachdev The Tech Panda.

India’s Decade of Tech Startup Growth

After the successful touchdown of ’s lander in the south of the Moon, all eyes are on India’s space program. Already, ISRO and the Japan Aerospace Exploration Agency (JAXA) launch LUPEX, a successor to Chandrayaan-3, in 2026. Chandrayaan-3 have decided to jointly In the limelight is also startup ecosystem, which has secured its position as a formidable contender on the global stage. According to recent insights by on August 11, 2023, India ranks seventh in terms of funding within the international SpaceTech landscape for the year. India’s SpaceTech Tracxn The Indian SpaceTech sector’s journey has been one of transformation, driven by the sector’s privatization in 2020. Previously dominated by governmental players, the sector has witnessed a significant surge in private sector participation post-privatization. Private entities are now actively involved in crucial aspects of research, manufacturing, and fabrication of rockets and satellites, fostering a vibrant ecosystem of innovation. The funding trajectory of the Indian SpaceTech ecosystem tells a compelling story of growth and resilience. From a modest $35 million in funds raised between 2010 and 2019, the sector experienced a meteoric rise in 2020, securing a remarkable $28 million in funding. This trend continued with exponential growth, reaching $96 million in 2021 and an impressive $112 million in 2022, marking a 17% increase from the previous year. This steady expansion in funding sets the Indian SpaceTech sector apart from other industries affected by funding challenges and macroeconomic fluctuations. Building on its upward trajectory, the Indian SpaceTech sector has already attracted $62 million in funding for the year 2023, reflecting a substantial 60% increase compared to the same period last year. This exceptional growth trend is expected to continue in the upcoming months. India’s emergence as a potent player in SpaceTech is further highlighted by its impressive satellite deployment. With 381 satellites placed in low orbit, India has positioned itself as a key player in this arena. Notably, the successful launch of 36 satellites into orbit by ISRO for UK-based OneWeb illustrates India’s contribution to global satellite ownership. The funding trajectory of the Indian SpaceTech ecosystem tells a compelling story of growth and resilience The Indian private sector has taken the lead in for practical applications, including satellite-based communication services like phone signals, broadband, OTT, and 5G. The government’s Budget 2023 initiatives further support the private enterprises’ role in this sector, from launching satellites and rockets to operating Earth stations with a 0% GST regime. Plans for a Production Linked Incentive (PLI) scheme for satellite manufacturing are also in the pipeline. launching satellites India’s space agency, ISRO, boasts a series of remarkable milestones, including the successful landing of Chandrayaan-3 on the moon’s south pole, paving the way for groundbreaking studies in the region. Also, India achieved a major achievement by becoming the first country to successfully land a spacecraft near the moon’s south pole. Additionally, ISRO’s feat of deploying a record-breaking 104 satellites in sun-synchronous orbits during a single mission in 2017 stands as a testament to India’s prowess. Among the thriving business models within the Indian SpaceTech sector, Small Payload-based Launch Vehicles have garnered significant attention, securing $75.6 million in funding over the last two years. Key players such as Skyroot, AgniKul, and Pixxel are spearheading innovative endeavors that promise to reshape the industry landscape. Skyroot, for instance, has launched India’s first privately built rocket, Vikram-S, into space, with plans to revolutionize satellite launches. Satellite-based Imaging Solutions have also witnessed substantial support, with $84.2 million in funding secured over the past two years. Companies like Pixxel have introduced pioneering technologies for earth observation, while Dhruva Space and Bellatrix Aerospace are contributing to the sector through their unique offerings. The global space industry is undergoing a phase of rapid innovation, spanning 3D-printed rocket components, commercial space tourism, reusable launch vehicles, asteroid mining, and novel propulsion technologies. In this ever-evolving landscape, India stands poised to capitalize on emerging opportunities, underlined by its proven capabilities in manufacturing and satellite/launch vehicle development. The accomplishments of the Indian Space industry, bolstered by government support, are expected to fuel entrepreneurship and investments, ushering in a new era of innovation and growth in the years ahead. This article was originally published by on Navanwita Bora Sachdev The Tech Panda.

India's SpaceTech Startup Ecosystem Spotlighted After Chandrayaan-3 Triumph

চন্দ্রযান-৩ জয়ের পর ভারতের স্পেসটেক স্টার্টআপ ইকোসিস্টেম স্পটলাইট হয়েছে

Indiens SpaceTech-Startup-Ökosystem rückt nach Chandrayaan-3-Triumph ins Rampenlicht

El ecosistema de startups de SpaceTech de la India se destaca después del triunfo de Chandrayaan-3

L'écosystème indien des startups SpaceTech mis en lumière après le triomphe de Chandrayaan-3

चंद्रयान-3 की जीत के बाद भारत का स्पेसटेक स्टार्टअप इकोसिस्टम सुर्खियों में आया

チャンドラヤーン3号の成功を受けて注目されるインドの宇宙技術スタートアップエコシステム

찬드라얀 3호 성공 이후 인도의 SpaceTech 스타트업 생태계가 주목받다

Ecossistema de startups SpaceTech da Índia destacado após o triunfo do Chandrayaan-3

Индийская экосистема стартапов SpaceTech оказалась в центре внимания после триумфа Chandrayaan-3

Hindistan'ın SpaceTech Startup Ekosistemi Chandrayaan-3 Zaferinden Sonra Dikkate Alındı

Hệ sinh thái khởi nghiệp công nghệ vũ trụ của Ấn Độ nổi bật sau chiến thắng Chandrayaan-3

“月船 3 号”胜利后，印度航天科技初创企业生态系统成为众人瞩目的焦点

Unsupervised Investments (I): A Guide to AI Investors

OpenTelemetry Tracing in Spring Boot: Choosing Between Java Agent and Micrometer Tracing

স্প্রিং বুটে ওপেনটেলিমেট্রি ট্রেসিং: এর জন্য জাভা এজেন্ট এবং মাইক্রোমিটারের মধ্যে নির্বাচন করা 

OpenTelemetry Tracing in Spring Boot: Auswahl zwischen Java Agent und Micrometer für

Seguimiento de OpenTelemetry en Spring Boot: elección entre Java Agent y Micrometer para

Traçage OpenTelemetry dans Spring Boot : choisir entre Java Agent et Micrometer pour

स्प्रिंग बूट में ओपनटेलीमेट्री ट्रेसिंग: जावा एजेंट और माइक्रोमीटर के बीच चयन करना

Spring BootでのOpenTelemetryトレース: JavaエージェントとMicrometerの選択

Spring Boot에서 OpenTelemetry 추적: Java Agent와 Micrometer 중에서 선택

Rastreamento OpenTelemetry no Spring Boot: Escolhendo entre Java Agent e Micrômetro para

Трассировка OpenTelemetry в Spring Boot: выбор между Java Agent и Micrometer для

Spring Boot'ta OpenTelemetry İzleme: Java Agent ve Micrometer Arasında Seçim Yapmak

OpenTelemetry Tracing trong Spring Boot: Lựa chọn giữa Java Agent và Micrometer cho

Spring Boot 中的 OpenTelemetry 跟踪：在 Java Agent 和 Micrometer 之间进行选择

With its main idea being to showcase traces across various technology stacks, including asynchronous communication via an MQTT queue, I continue to work on my Opentelemetry demo. This week, I added a couple of components and changed the architecture. Here are some noteworthy learnings; note that some of them might not be entirely connected to OpenTelemetry. Here's an updated diagram. New components appear in violet, and updated components appear in green. I want to be able to add more components. Thus, I decided that instead of directly querying the database, the inventory component would query warehouses, which are supposed to be located in different regions. Each warehouse can be implemented in a different stack, and you can have as many as you want—PRs are welcome. I miss Elixir and .Net at the moment. The contract, which I need to write down, is easy: An endpoint /stocks/${productId}


The ability to query PostgreSQL


Return the stock in the form: I've written about the changes in the inventory to account for the new configuration. Let's talk about the warehouse components. The Go warehouse Let me be blunt: I dislike (hate?) Go for its error-handling approach. However, with close to zero knowledge of the language, I was able to build a basic HTTP API that reads from the database in a couple of hours. I chose Gin Gonic for the web library and Gorm for the ORM. OpenTelemetry provides an integration with a couple of libraries, including Gin and Gorm. On the Dockerfile side, it's also pretty straightforward. I skipped optimizing the mount cache and the final base image, though I might return to it later. All in all, that's the component I developed the fastest. I still dislike the Go language, but I begrudgingly understand that developers who want to get things done use it. The Ruby warehouse While Ruby is not this famous anymore, I still wanted the stack in my architecture. I eschewed Ruby on Rails in favor of the leaner Sinatra framework. I use sequel for database access. The dynamic nature of the language was a bit of a hurdle, which is why it took me more time to develop my service than with Go. I also spent a non-trivial amount of time on auto-instrumentation. For stacks with a runtime, auto-instrumentation allows developers to go on their merry way, oblivious to any OpenTelemetry concern. At runtime, the Ops team adds the necessary configuration for OpenTelemetry. For example, we achieve this with a Java Agent on the JVM. I expected the same "carefree" approach with Ruby, but I couldn't find anything related to the stack. Ruby on Rails has a built-in plugin system, but not Sinatra. I tried to use bash to glue files together, but to no avail. If you're a Ruby expert or have any experience doing this, please let me know how. The GraalVM native warehouse This one is a regular Kotlin application on Spring Boot with a twist: I'm using GraalVM native image to compile ahead-of-time to bytecode. This way, I can use a tiny Docker image as my base, e.g, busybox. It's not as efficient as Go or Rust, but it's a good bet if tied to the JVM. OpenTelemetry did work on the JVM version but didn't when I compiled it to bytecode. Given the compilation time, it took me a couple of days of back-and-forth to make it work. The reason is simple: Spring Boot relies on auto configuration classes to activate or not features. Some auto-configuration classes rely on the presence of classes, others on the presence of beans, others on the opposite, others on existing properties, others on a combination of the above, etc. In my case, the guilty class was OtlpTracingConfigurations.ConnectionDetails. It relies on the management.otlp.tracing.endpoint property: class OtlpTracingConfigurations {

  @Configuration(proxyBeanMethods = false)
  static class ConnectionDetails {

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnProperty(prefix = "management.otlp.tracing", name = "endpoint")
    OtlpTracingConnectionDetails otlpTracingConnectionDetails(OtlpProperties properties) {
      return new PropertiesOtlpTracingConnectionDetails(properties);
    }
} If the property is not present at compile-time, the Spring Framework doesn't create a bean of type OtlpTracingConnectionDetails. Through a chain of missing beans, the final binary doesn't contain OpenTelemetry-related code. The solution is easy: set the property to an empty string in the application.properties file, and override it to its regular value in the Docker Compose file. While auto-configuration is a compelling feature, you must understand how it works. That's the easy part. However, it's much more work to understand the whole chain of auto-configuration activation regarding a feature. Having distanced myself from the JVM, I'm no longer an expert in these chains, much less the OpenTelemetry one. I finally understand why some developers avoid Spring Boot and name it magic. Migrating from JavaScript to TypeScript I used JavaScript in my first draft of a subscriber to the MQTT queue. Soon afterward, I decided to migrate to TypeScript. JavaScript code is valid TypeScript, so a simple copy-paste worked, with the addition of @ts-ignore. However, when I tried to fix the code to "true" TypeScript, I couldn't see any OpenTelemetry trace. As for GraalVM, I went back and forth several times, but this time, I decided to solve it once and for all. I migrated code line by line until I isolated the issue in the following snippet: const userProperties = {}

if (packet.properties && packet.properties['userProperties']) {
    const props = packet.properties['userProperties']
    console.error('Props', props)
    for (const key of Object.keys(props)) {
        userProperties[key] = props[key]                         //1
    }
} The TypeScript compiler complains with the following error message: TS7053: Element implicitly has an any type because expression of type string can't be used to index type {} I earlier tried to fix it with the following: const userProperties = new Map<string, any>() It compiled, but my limited understanding of JavaScript prevented me from realizing that a Map is not the same structure as an object. I understood the issue only when I isolated the exact line that went wrong. I just had to find the correct syntax to declare the type of an object: const userProperties: Record<string, any> = {} Adding a Redis cache So far, my services have used only PostgreSQL as a data store. Datastores don't implement OpenTelemetry by themselves, but the correct instrumentation of an app can show the trace going to the datastore. Here, you can see the trace created by the OpenTelemetry agent on a Kotlin/Spring Boot app that uses the PostgreSQL driver. Here's the one from the Gorm framework, instrumented manually. Both traces display the system, the statement, and a couple of other data. The Redis instrumentation shows the same information under the same structure! Icing on the cake, if you use the Lettuce client, which is the default for Spring, you don't need additional changes. The OpenTelemetry agent already takes care of everything. Another Apache APISIX instance Last but not least, I've added another APISIX instance. Most organizations manage their APIs from behind a single multi-node API Gateway. However, it can be a significant burden depending on how the organization structures the teams. When a team needs to deploy a new or change the routing of an existing non-front-facing API, they want the change ASAP. If the team in charge of the centralized API Gateway doesn't respond to tickets, it slows down the API team and the business value they want to deploy. For this reason, it's a perfectly valid pattern to set up API Gateways that are not front-facing under the responsibility of each API team - or business department. The granularity here depends on what works for each organization. It reduces friction when the infrastructure needs to change but does so at the cost of more diversified work for the API team. It also allows for different API Gateway technology (or technologies) from the front-facing one. In my demo, I assume the team responsible for the inventory component has set up such an instance. It handles all routing to the warehouses. All other components still rely on the primary APISIX instance. Conclusion In this post, I've described several changes I made in my OpenTelemetry tracing demo and the lessons I learned. I want to add additional warehouse components in other stacks. What stack would you be interested in? Would you like to contribute to such a component? To go further: End-to-end tracing with OpenTelemetry
Improving upon my OpenTelemetry Tracing demo
Parsing structured environment variables in Rust The complete source code for this post can be found on GitHub: https://github.com/nfrankel/opentelemetry-tracing?embedable=true Originally published at A Java Geek on June 2nd, 2024 With its main idea being to showcase traces across various technology stacks, including asynchronous communication via an MQTT queue, I continue to work on my Opentelemetry demo . This week, I added a couple of components and changed the architecture. Here are some noteworthy learnings; note that some of them might not be entirely connected to OpenTelemetry. traces asynchronous communication Opentelemetry demo Here's an updated diagram. New components appear in violet, and updated components appear in green. New components appear in violet, and updated components appear in green. I want to be able to add more components. Thus, I decided that instead of directly querying the database, the inventory component would query warehouses, which are supposed to be located in different regions. Each warehouse can be implemented in a different stack, and you can have as many as you want—PRs are welcome. I miss Elixir and .Net at the moment. The contract, which I need to write down, is easy: inventory An endpoint /stocks/${productId} The ability to query PostgreSQL Return the stock in the form: An endpoint /stocks/${productId} An endpoint /stocks/${productId} /stocks/${productId} The ability to query PostgreSQL The ability to query PostgreSQL Return the stock in the form: Return the stock in the form: I've written about the changes in the inventory to account for the new configuration. Let's talk about the warehouse components. The Go warehouse Let me be blunt: I dislike (hate?) Go for its error-handling approach. However, with close to zero knowledge of the language, I was able to build a basic HTTP API that reads from the database in a couple of hours. I chose Gin Gonic for the web library and Gorm for the ORM. OpenTelemetry provides an integration with a couple of libraries , including Gin and Gorm. On the Dockerfile side, it's also pretty straightforward. I skipped optimizing the mount cache and the final base image, though I might return to it later. Gin Gonic Gorm couple of libraries All in all, that's the component I developed the fastest. I still dislike the Go language, but I begrudgingly understand that developers who want to get things done use it. The Ruby warehouse While Ruby is not this famous anymore, I still wanted the stack in my architecture. I eschewed Ruby on Rails in favor of the leaner Sinatra framework. I use sequel for database access. The dynamic nature of the language was a bit of a hurdle, which is why it took me more time to develop my service than with Go. Sinatra sequel I also spent a non-trivial amount of time on auto-instrumentation. For stacks with a runtime, auto-instrumentation allows developers to go on their merry way, oblivious to any OpenTelemetry concern. At runtime, the Ops team adds the necessary configuration for OpenTelemetry. For example, we achieve this with a Java Agent on the JVM. I expected the same "carefree" approach with Ruby, but I couldn't find anything related to the stack. Ruby on Rails has a built-in plugin system, but not Sinatra. I tried to use bash to glue files together, but to no avail. If you're a Ruby expert or have any experience doing this, please let me know how. bash The GraalVM native warehouse This one is a regular Kotlin application on Spring Boot with a twist: I'm using GraalVM native image to compile ahead-of-time to bytecode . This way, I can use a tiny Docker image as my base, e.g , busybox . It's not as efficient as Go or Rust, but it's a good bet if tied to the JVM. GraalVM native image bytecode e.g busybox OpenTelemetry did work on the JVM version but didn't when I compiled it to bytecode . Given the compilation time, it took me a couple of days of back-and-forth to make it work. The reason is simple: Spring Boot relies on auto configuration classes to activate or not features. Some auto-configuration classes rely on the presence of classes, others on the presence of beans, others on the opposite, others on existing properties, others on a combination of the above, etc. bytecode auto configuration In my case, the guilty class was OtlpTracingConfigurations.ConnectionDetails . It relies on the management.otlp.tracing.endpoint property: OtlpTracingConfigurations.ConnectionDetails management.otlp.tracing.endpoint class OtlpTracingConfigurations {

  @Configuration(proxyBeanMethods = false)
  static class ConnectionDetails {

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnProperty(prefix = "management.otlp.tracing", name = "endpoint")
    OtlpTracingConnectionDetails otlpTracingConnectionDetails(OtlpProperties properties) {
      return new PropertiesOtlpTracingConnectionDetails(properties);
    }
} class OtlpTracingConfigurations {

  @Configuration(proxyBeanMethods = false)
  static class ConnectionDetails {

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnProperty(prefix = "management.otlp.tracing", name = "endpoint")
    OtlpTracingConnectionDetails otlpTracingConnectionDetails(OtlpProperties properties) {
      return new PropertiesOtlpTracingConnectionDetails(properties);
    }
} If the property is not present at compile-time , the Spring Framework doesn't create a bean of type OtlpTracingConnectionDetails . Through a chain of missing beans, the final binary doesn't contain OpenTelemetry-related code. The solution is easy: set the property to an empty string in the application.properties file, and override it to its regular value in the Docker Compose file. at compile-time OtlpTracingConnectionDetails application.properties While auto-configuration is a compelling feature, you must understand how it works. That's the easy part. However, it's much more work to understand the whole chain of auto-configuration activation regarding a feature. Having distanced myself from the JVM, I'm no longer an expert in these chains, much less the OpenTelemetry one. I finally understand why some developers avoid Spring Boot and name it magic. Migrating from JavaScript to TypeScript I used JavaScript in my first draft of a subscriber to the MQTT queue. Soon afterward, I decided to migrate to TypeScript. JavaScript code is valid TypeScript, so a simple copy-paste worked, with the addition of @ts-ignore . JavaScript @ts-ignore However, when I tried to fix the code to "true" TypeScript, I couldn't see any OpenTelemetry trace. As for GraalVM, I went back and forth several times, but this time, I decided to solve it once and for all. I migrated code line by line until I isolated the issue in the following snippet: const userProperties = {}

if (packet.properties && packet.properties['userProperties']) {
    const props = packet.properties['userProperties']
    console.error('Props', props)
    for (const key of Object.keys(props)) {
        userProperties[key] = props[key]                         //1
    }
} const userProperties = {}

if (packet.properties && packet.properties['userProperties']) {
    const props = packet.properties['userProperties']
    console.error('Props', props)
    for (const key of Object.keys(props)) {
        userProperties[key] = props[key]                         //1
    }
} The TypeScript compiler complains with the following error message: TS7053: Element implicitly has an any type because expression of type string can't be used to index type {} The TypeScript compiler complains with the following error message: TS7053: Element implicitly has an any type because expression of type string can't be used to index type {} TS7053: Element implicitly has an any type because expression of type string can't be used to index type {} I earlier tried to fix it with the following: const userProperties = new Map<string, any>() const userProperties = new Map<string, any>() It compiled, but my limited understanding of JavaScript prevented me from realizing that a Map is not the same structure as an object. I understood the issue only when I isolated the exact line that went wrong. I just had to find the correct syntax to declare the type of an object: Map const userProperties: Record<string, any> = {} const userProperties: Record<string, any> = {} Adding a Redis cache So far, my services have used only PostgreSQL as a data store. Datastores don't implement OpenTelemetry by themselves, but the correct instrumentation of an app can show the trace going to the datastore. Here, you can see the trace created by the OpenTelemetry agent on a Kotlin/Spring Boot app that uses the PostgreSQL driver. Here's the one from the Gorm framework, instrumented manually. Both traces display the system, the statement, and a couple of other data. The Redis instrumentation shows the same information under the same structure! Icing on the cake, if you use the Lettuce client, which is the default for Spring, you don't need additional changes. The OpenTelemetry agent already takes care of everything. Another Apache APISIX instance Last but not least, I've added another APISIX instance. Most organizations manage their APIs from behind a single multi-node API Gateway. However, it can be a significant burden depending on how the organization structures the teams. When a team needs to deploy a new or change the routing of an existing non-front-facing API, they want the change ASAP. If the team in charge of the centralized API Gateway doesn't respond to tickets, it slows down the API team and the business value they want to deploy. For this reason, it's a perfectly valid pattern to set up API Gateways that are not front-facing under the responsibility of each API team - or business department. The granularity here depends on what works for each organization. It reduces friction when the infrastructure needs to change but does so at the cost of more diversified work for the API team. It also allows for different API Gateway technology (or technologies) from the front-facing one. In my demo, I assume the team responsible for the inventory component has set up such an instance. It handles all routing to the warehouses. All other components still rely on the primary APISIX instance. inventory Conclusion In this post, I've described several changes I made in my OpenTelemetry tracing demo and the lessons I learned. I want to add additional warehouse components in other stacks. What stack would you be interested in? Would you like to contribute to such a component? To go further: To go further: End-to-end tracing with OpenTelemetry Improving upon my OpenTelemetry Tracing demo Parsing structured environment variables in Rust End-to-end tracing with OpenTelemetry End-to-end tracing with OpenTelemetry Improving upon my OpenTelemetry Tracing demo Improving upon my OpenTelemetry Tracing demo Parsing structured environment variables in Rust Parsing structured environment variables in Rust The complete source code for this post can be found on GitHub: https://github.com/nfrankel/opentelemetry-tracing?embedable=true https://github.com/nfrankel/opentelemetry-tracing?embedable=true Originally published at A Java Geek on June 2nd, 2024 Originally published at A Java Geek on June 2nd, 2024 A Java Geek

Optimizing OpenTelemetry Tracing with Multi-Stack Warehouse Components

Last year, I wrote a on Open Telemetry Tracing to understand more about the subject. I also created a demo around it, which featured the following components: post The Apache APISIX API Gateway A Kotlin/Spring Boot service A Python/Flask service And a Rust/Axum service I've recently improved the demo to deepen my understanding and want to share my learning. Using a regular database In the initial demo, I didn't bother with a regular database. Instead: The Kotlin service used the embedded Java H2 database The service used the embedded SQLite Python The Rust service used hard-coded data in a hash map I replaced all of them with a regular PostgreSQL database, with a dedicated schema for each. The OpenTelemetry agent added a new span when connecting to the database on the JVM and in Python. For the JVM, it's automatic when one uses the Java agent. One needs to install the relevant package in Python - see next section. OpenTelemetry integrations in Python libraries Python requires you to explicitly add the package that instruments a specific library for OpenTelemetry. For example, the demo uses Flask; hence, we should add the Flask integration package. However, it can become a pretty tedious process. Yet, once you've installed , you can "sniff" installed packages and install the relevant integration. opentelemetry-distro pip install opentelemetry-distro

opentelemetry-bootstrap -a install For the demo, it installs the following: opentelemetry_instrumentation-0.41b0.dist-info
opentelemetry_instrumentation_aws_lambda-0.41b0.dist-info
opentelemetry_instrumentation_dbapi-0.41b0.dist-info
opentelemetry_instrumentation_flask-0.41b0.dist-info
opentelemetry_instrumentation_grpc-0.41b0.dist-info
opentelemetry_instrumentation_jinja2-0.41b0.dist-info
opentelemetry_instrumentation_logging-0.41b0.dist-info
opentelemetry_instrumentation_requests-0.41b0.dist-info
opentelemetry_instrumentation_sqlalchemy-0.41b0.dist-info
opentelemetry_instrumentation_sqlite3-0.41b0.dist-info
opentelemetry_instrumentation_urllib-0.41b0.dist-info
opentelemetry_instrumentation_urllib3-0.41b0.dist-info
opentelemetry_instrumentation_wsgi-0.41b0.dist-info The above setup adds a new automated trace for connections. Gunicorn on Flask Every time I started the Flask service, it showed a warning in red that it shouldn't be used in production. While it's unrelated to OpenTelemetry, and though nobody complained, I was not too fond of it. For this reason, I added a "real" HTTP server. I chose for no other reason than because my knowledge of the Python ecosystem is still shallow. Gunicorn The server is a runtime concern. We only need to change the slightly: Dockerfile RUN pip install gunicorn

ENTRYPOINT ["opentelemetry-instrument", "gunicorn", "-b", "0.0.0.0", "-w", "4", "app:app"] The option refers to binding; you can attach to a specific IP. Since I'm running Docker, I don't know the IP, so I bind to any. -b The option specifies the number of workers -w Finally, the argument sets the module and the application, separated by a colon app:app Gunicorn usage doesn't impact OpenTelemetry integrations. Heredocs for the win You may benefit from this if you write a lot of . Dockerfile Every Docker layer has a storage cost. Hence, inside a , one tends to avoid unnecessary layers. For example, the two following snippets yield the same results. Dockerfile RUN pip install pip-tools 
RUN pip-compile
RUN pip install -r requirements.txt
RUN pip install gunicorn
RUN opentelemetry-bootstrap -a install

RUN pip install pip-tools \
  && pip-compile \
  && pip install -r requirements.txt \
  && pip install gunicorn \
  && opentelemetry-bootstrap -a install The first snippet creates five layers, while the second is only one; however, the first is more readable than the second. With , we can access a more readable syntax that creates a single layer: heredocs RUN <<EOF

  pip install pip-tools 
  pip-compile
  pip install -r requirements.txt
  pip install gunicorn
  opentelemetry-bootstrap -a install

EOF Heredocs are a great way to have more readable and more optimized Dockerfiles. Try them! Explicit API call on the JVM In the initial demo, I showed two approaches: The first uses auto-instrumentation, which requires no additional action The second uses manual instrumentation with Spring annotations I wanted to demo an explicit call with the API in the improved version. The use-case is analytics and uses a message queue: I get the trace data from the HTTP call and create a message with such data so the subscriber can use it as a parent. First, we need to add the OpenTelemetry API dependency to the project. We inherit the version from the Spring Boot Starter parent POM: <dependency>
	<groupId>io.opentelemetry</groupId>
	<artifactId>opentelemetry-api</artifactId>
</dependency> At this point, we can access the API. OpenTelemetry offers a static method to get an instance: val otel = GlobalOpenTelemetry.get() At runtime, the agent will work its magic to return the instance. Here's a simplified class diagram focused on tracing: In turn, the flow goes something like this: val otel = GlobalOpenTelemetry.get()                                   //1
val tracer = otel.tracerBuilder("ch.frankel.catalog").build()          //2
val span = tracer.spanBuilder("AnalyticsFilter.filter")                //3
                 .setParent(Context.current())                         //4
                 .startSpan()                                          //5
// Do something here
span.end()                                                             //6 Get the underlying OpenTelemetry Get the tracer builder and "build" the tracer Get the span builder Add the span to the whole chain Start the span End the span; after this step, send the data to the OpenTelemetry endpoint configured. Adding a message queue When I did the talk based on the post, attendees frequently asked whether OpenTelemetry would work with messages such as MQ or Kafka. While I thought it was the case in theory, I wanted to make sure of it: I added a message queue in the demo under the pretense of analytics. The Kotlin service will publish a message to an MQTT topic on each request. A NodeJS service will subscribe to the topic. Attaching OpenTelemetry data to the message So far, OpenTelemetry automatically reads the context to find out the trace ID and the parent span ID. Whatever the approach, auto-instrumentation or manual, annotations-based or explicit, the library takes care of it. I didn't find any existing similar automation for messaging; we need to code our way in. The gist of OpenTelemetry is the HTTP header. We need to read it and send it along with the message. traceparent First, let's add MQTT API to the project. <dependency>
	<groupId>org.eclipse.paho</groupId>
	<artifactId>org.eclipse.paho.mqttv5.client</artifactId>
	<version>1.2.5</version>
</dependency> Interestingly enough, the API doesn't allow access to the directly. However, we can reconstruct it via the class. traceparent SpanContext I'm using MQTT v5 for my message broker. Note that the v5 allows for metadata attached to the message; when using v3, the message itself needs to wrap them. val spanContext = span.spanContext                                                //1
val message = MqttMessage().apply {

  properties = MqttProperties().apply {
    val traceparent = "00-${spanContext.traceId}-${spanContext.spanId}-${spanContext.traceFlags}" //2
    userProperties = listOf(UserProperty("traceparent", traceparent))             //3
  }
  qos = options.qos
  isRetained = options.retained

  val hostAddress = req.remoteAddress().map { it.address.hostAddress }.getOrNull()
  payload = Json.encodeToString(Payload(req.path(), hostAddress)).toByteArray()   //4
}
val client = MqttClient(mqtt.serverUri, mqtt.clientId)                            //5
client.publish(mqtt.options, message)                                             //6 Get the span context Construct the from the span context, according to the W3C Trace Context specification traceparent Set the message metadata Set the message body Create the client Publish the message Getting OpenTelemetry data from the message The subscriber is a new component based on NodeJS. First, we configure the app to use the OpenTelemetry trace exporter: const sdk = new NodeSDK({
  resource: new Resource({[SemanticResourceAttributes.SERVICE_NAME]: 'analytics'}),
  traceExporter: new OTLPTraceExporter({
    url: `${collectorUri}/v1/traces`
  })
})

sdk.start() The next step is to read the metadata, recreate the context from the , and create a span. traceparent client.on('message', (aTopic, payload, packet) => {
  if (aTopic === topic) {

    console.log('Received new message')

    const data = JSON.parse(payload.toString())

    const userProperties = {}
    if (packet.properties['userProperties']) {                                  //1
      const props = packet.properties['userProperties']
      for (const key of Object.keys(props)) {
        userProperties[key] = props[key]
      }
    }

    const activeContext = propagation.extract(context.active(), userProperties) //2
    const tracer = trace.getTracer('analytics')
    const span = tracer.startSpan(                                              //3
      'Read message',
      {attributes: {path: data['path'], clientIp: data['clientIp']}},
      activeContext,
    )
    span.end()                                                                  //4
  }
}) Read the metadata Recreate the context from the traceparent Create the span End the span For the record, I tried to migrate to TypeScript, but when I did, I didn't receive the message. Help or hints are very welcome! Apache APISIX for messaging Though it's not common knowledge, Apache APISIX can proxy HTTP calls as well as UDP and TCP messages. It only offers a few plugins at the moment, but it will add more in the future. An OpenTelemetry one will surely be part of it. In the meantime, let's prepare for it. The first step is to configure Apache APISIX to allow both HTTP and TCP: apisix:
  proxy_mode: http&stream                                                       #1
  stream_proxy:
    tcp:
      - addr: 9100                                                              #2
        tls: false Configure APISIX for both modes Set the TCP port The next step is to configure TCP routing: upstreams:
  - id: 4
    nodes:
      "mosquitto:1883": 1                                                       #1


stream_routes:                                                                  #2
  - id: 1
    upstream_id: 4
    plugins:
      mqtt-proxy:                                                               #3
        protocol_name: MQTT
        protocol_level: 5                                                       #4 Define the MQTT queue as the upstream Define the "streaming" route. APISIX defines everything that's not HTTP as streaming Use the MQTT proxy. Note APISIX offers a Kafka-based one Address the MQTT version. For version above 3, it should be 5 Finally, we can replace the MQTT URLs in the Docker Compose file with APISIX URLs. Conclusion I've described several items I added to improve my OpenTelemetry demo in this post. While most are indeed related to OpenTelemetry, some of them aren't. I may add another component in another different stack, a front-end. The complete source code for this post can be found on . GitHub Also published . here

OpenTelemetry: Improving My Tracing Demo

Distributed tracing is being promoted as one of the pillars of observability, offering enhanced visibility into cross-service interactions. It gives potentially an opportunity to identify the service interaction problems, providing a precise understanding of where problems may arise. Despite its advertised and widely promoted potential, there are more chances to fail to adopt distributed tracing than to get any value out of it. Let’s review the main myths, misconceptions, and failures associated with adopting distributed tracing. Teams Implement Just Another Logging Frequently, the initial attempts at implementing distributed tracing within organizations fail because they start with a very common approach: instrument only a few microservices. This strategy is a common starting point for demonstrating the successful PoC. However, the typical result of these experiments is that teams use tracing in the boundaries of their service or a few services. So, instrumenting just one service out of dozens or even hundreds leads to an unexpected result - teams end up with what is essentially another form of logging. Think about it … in addition to the next familiar logline: {   "ip": "127.0.0.1",   "user": "-",   "date": "06/Dec/2023:12:30:45 +0000",   "method": "GET",   "url": "/example-page",   "http_version": "HTTP/1.1",   "status_code": 200,   "response_size": 1234 } The application starts sending the following spans to the distributed tracing backend: { "traceId": "8a4da9f548267ea8", "spanId": "bc8d7a6c12345a", "parentSpanId": "a2a7b5c6123456", "name": "HTTP GET /example-page", "kind": "SERVER", "timestamp": 1638760245000, "duration": 1500, "attributes": { "http.method": "GET", "http.status_code": 200, "http.url": "/example-page", "ip": "127.0.0.1", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" }, "status": { "code": "OK", "message": "HTTP 200" } } In other words, that is just structured logging. That is not really the outcome that engineers expect. Many teams stop experimenting with distributed tracing at this stage, as teams do not really need another form of logging albeit with more visualization – the value of which is often questionable. How to Avoid It? Distributed tracing extends beyond the individual service or deployment scope. Successful tracing requires instrumenting the entire request flow passing through dozens of services, making tens or even hundreds of persistence layer calls, and uncontrolled third-party integrations. Nobody Told Us That the Cost of Tracing Grows Unproportionally. Teams are already accustomed to paying for metrics and logging … and sometimes, a lot. Believe me, distributed tracing costs can be as high or even higher as there are costs associated with ingestion, indexing, network usage, and, in some cases, user access. Let’s think about why the tracing cost is that high. The answer is actually obvious: it is common for engineers to assume that the more extensively they instrument with distributed tracing, the more value they'll derive. Undoubtedly, there is value in creating spans, and for each database call, outgoing request, or other resource-intensive operations. That gives incredible transparency on how the system is operating and where are the potential bottlenecks. However, well, it takes some time for teams to acknowledge that this is not exactly how tracing is used as, in practice, a significant portion of data collected remains unused, yet the costs persist. How to Avoid It? Be mindful of how, what, and when to instrument. Obviously, excluding routine health-check requests, numerous database calls, and noisy flows is a good starting point. But even excluding all the noise is not enough: you need to start sampling the data. And sampling means that only a fraction of traces will be stored and by a fraction; I do not mean 50% … but rather 10-15% or even smaller for high throughput applications. Vendors Sold Us an Idea That Tracing Reduces MTTR and MTTD, But It Doesn’t Because of the Limitations of Trace Sampling. You can often see similar text on the vendors’ papers: If a customer reports that a feature in an application is slow or broken, the support team can review distributed traces to determine if this is a backend issue. Engineers can then analyze the traces generated by the affected service to quickly troubleshoot the problem. Well, what often goes unsaid is that distributed tracing might not be the silver bullet for all types of failures. Let’s already agree that we cannot avoid sampling the traces, and only a minority of data reaches the vendor. What would that mean? It means that tracing becomes less effective for: searching for a specific customer problems identifying rare issues that are difficult to pinpoint address scoped outages such as an outage of specific architecture cells or performance degradation of the specific software version How to Avoid It? First, understand why your organization needs distributed tracing and what problem it solves beyond the industry hype and everyone trying. Ask yourself if you can find use cases when storing only 5% of all the traces is an acceptable compromise. Automatic Instrumentation Is Not Always Automatic Automatic code instrumentation is a very useful and time-saving approach to instrumenting your code for distributed tracing. In some cases (e.g., Java), it requires the addition of dependency and a few configuration lines. However, the first impression can be deceptive: While there are instrumentations available for tools like Kafka and other async communication platforms, automatic instrumentation can be less effective for entirely asynchronous systems that rely heavily on background jobs and batched requests. Async complexity. This is because automatic tools may struggle to accurately capture and correlate the spans in such environments. Automated instrumentation can introduce performance overhead. That could become an issue, especially for applications with strong performance requirements. So, never forget about performance tests before enabling them in the production environment. Performance Issues. Most of the software is legacy to some extent, and automated instrumentation doesn't guarantee support for all libraries, versions, or use cases. Legacy systems. Enterprises often develop in-house frameworks, libraries, and packages avoiding widely off-the-shelf solutions. It introduces challenges when it comes to instrumentation as in most cases automated instrumentation won’t be helpful for those use cases. Bespoke frameworks. The Biggest Failure of Distributed Tracing: A Commitment and Culture Again, distributed tracing operates on the level of the entire request meaning that every individual service should be instrumented with distributed tracing. Instrumenting every service requires significant engineering effort and careful planning. Moreover, there is no guarantee that the project will be successful and that all the effort will translate to an improved MTTR and MTTD. As for the culture, teams already have dashboards and monitors for all the downstream, dependencies, incoming, and outgoing requests. Engineers teams have yearly improved logging that actually, in common, is good enough to troubleshoot most of the issues. Introducing distributed tracing as yet another tool to improve MTTR and MTTD can be met with resistance from teams accustomed to their existing monitoring processes. Conclusion A distributed project is an extremely huge and complicated IT project that should be executed on an organizational level. Numerous pitfalls and considerations come into play, and even if you're skeptical based on my experience, it's crucial to bear these points in mind.

Distributed Tracing: What They Did Not Tell Us About It

Hey everyone, the year 2023 has emerged as a pivotal moment, hailed as the epoch of the AI revolution. Throughout this year, we have witnessed great advancements in the realm of AI technology. Notably, the advent of ChatGPT by OpenAI has indelibly reshaped our interactions with artificial intelligence, sparking a cascade of innovation as numerous companies delve into the creation of remarkable AI technologies. However, to truly grasp the significance of these recent feats, we must embark on a voyage through the chronicles of AI's evolution. Join me as we delve into the captivating saga of AI's Odyssey, navigating the currents that have propelled us from the past, through the dynamic present, and into the eagerly anticipated future. Wait, what does an AI mean? According to the Wikipedia: is the ability of machines to perform tasks that are typically associated with human intelligence, such as learning and problem-solving. Artificial intelligence (AI) To put it simply, imagine computers that – that's Artificial Intelligence (AI). think like humans, learn from experience and make decisions The evolution of AI has been nothing short of a captivating odyssey, reshaping the landscape of technology and human interaction. From its beginnings as a concept to its current state as a driving force behind innovation, AI’s journey has been marked by breakthroughs, challenges and a vision of a future where machines rival human intelligence. AI has been a fascinating and elusive goal for many researchers, philosophers, and visionaries since ancient times. The Past of AI: From Myth to Reality The idea of creating artificial beings that can think and act like humans is not new. In fact, it can be traced back to ancient myths and legends, as the Greek story of Pygmalion and Galatea, where a sculptor falls in love with his own creation, or the Jewish legend of the Golem, where a clay figure is brought to life by a rabbi. These stories reflect the human fascination and curiosity with the possibility of creating intelligent life. such The term “artificial intelligence” was by John McCarthy in 1956 at a conference at Dartmouth College, where he invited a group of researchers to discuss “ ”. This marked the beginning of the modern era of AI research, which was initially optimistic and ambitious. Early AI pioneers, such as Alan Turing, Marvin Minsky, Claude Shannon, and Herbert Simon, made significant contributions to the fields of logic, computation, cognitive science, and psychology, and developed some of the first AI programs that could play chess, solve mathematical problems, prove theorems, and understand natural language. coined the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it However, AI research soon faced many difficulties and limitations, such as the complexity and ambiguity of natural language, the uncertainty and incompleteness of real-world knowledge, the scalability and efficiency of algorithms, and the ethical and social implications of intelligent machines. These challenges led to several periods of reduced funding and interest in AI research, as “ ”. Despite these setbacks, AI research continued to advance and diversify into various subfields and applications, such as expert systems, neural networks, computer vision, robotics, natural language processing, machine learning, data mining, speech recognition, and more. known AI winters The Present of AI: From Narrow to General The current state of AI is characterized by rapid progress and widespread adoption in various domains and industries. Thanks to the availability of large amounts of data, the development of powerful computing hardware and software platforms, the improvement of algorithms and techniques, and the integration of multiple disciplines and modalities, AI has achieved remarkable feats and breakthroughs in recent years. We are seeing some AI applications from voice assistants to fraud detection, recommendations to autonomous vehicles and more. But the one that revolutionized the usage of AI is the release of ChatGPT. It is an LLM Chatbot that uses NLP to create humanlike conversations. Because of its versatility, intelligence and engagement in a conversational manner, it became the fastest application to reach 100 million active users in just two months. Following its initial release, there was an unprecedented surge in the AI trend, captivating the attention of individuals across the globe. The graph above vividly illustrates the Google Trends data, showcasing the remarkable trajectory of interest in the term 'AI' over the last five years. In response to this fervent demand, industry giants leaped into action. Microsoft swiftly began incorporating , revolutionizing the way users interacted with information. Soon thereafter, Google made its own resounding entry into the AI arena with the of Bard. Then we have PaLM 2, GPT-4, Meta’s Code Llama and the recently released LLM SeamlessM4T and more to say. AI into its search engine introduction called We have AI in various domains from coding, arts, music and more. Now it’s all about the prompting. The better your prompt is, the better the result. There’s even a job title “Prompt Engineer” who develop, test and refine AI prompts. called The banner image that you see at the top of this article is actually created by using Generative AI with just prompts. Some famous AI image generators are DALL-E 2, Midjourney, and Runway Gen-2. Many industries have adopted AI in their workflows and have seen positive results, including significant cost savings. One of the most significant aspects of this current AI trend is that AI is now available to everyone. This has revolutionized innovation. All you need is an idea, and with the help of AI, you can build anything. One of my favourite AI tools is Replit Ghostwriter, which acts as a coding partner, and you can build anything you want. These are just a few examples of the many ways that AI is being used today. As the technology continues to develop, we can expect to see even more innovative and impactful applications emerge. The advancement of AI is raising a number of . These concerns include the potential for AI to be used to create autonomous weapons, to discriminate against people, and to invade people's privacy. It is important to address these concerns as AI continues to develop. ethical and legal concerns Charting the Course Ahead: Envisioning AI's Future As AI's capabilities continue to expand, we stand at the threshold of a future that holds both immense potential and significant challenges. The prospect of Artificial General Intelligence (AGI), where machines possess human-like cognitive abilities, tantalizes researchers and fuels discussions about the ethical, societal, and existential implications of such an achievement. achieving Ethical considerations become paramount as AI increasingly influences decision-making in critical domains. Ensuring transparency, fairness, and accountability in AI systems becomes essential to prevent biases and discriminatory outcomes. as AI penetrates areas like autonomous weapons, AI-generated content, and personalized medicine. Striking a balance between innovation and regulation will be vital AI's impact on the job market is another facet of its evolving landscape. While automation may reshape certain industries, it also paves the way for the creation of new roles and opportunities. Nurturing a workforce equipped with the skills to collaborate with AI systems becomes crucial to harness the full potential of this technology. From the conducted by the World Economic Forum: survey Nearly half (49%) of companies expect adopting AI to create jobs, well ahead of the 23% of respondents who expect it to displace jobs. One of the major issues that I want to share is the use of . to develop new and more sophisticated attacks. For example, AI can be used to create malware that is more difficult to detect and remove. AI can also be used to automate tasks such as scanning for vulnerable systems and launching attacks. AI in Cybercrimes Cybercriminals use AI The journey of AI, from its inception to its current prowess and its potential to shape our future, is a testament to human ingenuity and perseverance. The odyssey of AI is marked by its transformative influence across industries, disciplines, and societies. As we move into the future, we need to be thoughtful and make sure we use AI in good ways that help people. In conclusion, AI's odyssey is a testament to human curiosity and ambition. With each milestone achieved and challenges faced, we inch closer to a world where AI augments our abilities, enhances our experiences, and prompts us to reflect on what it truly means to create intelligence, artificial or otherwise. As we stand at this juncture, the true potential of AI lies not just in its technical achievements, but in how responsibly we wield its capabilities to forge a future that . benefits all of humanity Thanks for the read! You can find me on X @ . dotAadarsh

AI's Odyssey: Tracing the Past, Present, and Potential Future of Artificial Intelligence

Distributed Tracing is a divisive topic. Once the , the technology was expected to revolutionize observability. doyen of every KubeCon Fast forward 5 years, the hype has subsided somewhat, there's a , and adoption is moderate. Meanwhile, there continues to be steady activity around expanding and standardizing the technology - Open Telemetry (based on OpenTracing) is the after Kubernetes. So what is the deal with Distributed Tracing? Should one implement it right away or wait and watch? lot more talk about the pain 2nd largest CNCF project In this article, let's explore Distributed Tracing in depth What is special about Distributed Tracing and why do we need it? What are the problems with distributed tracing today? What are upcoming developments and how do they address existing challenges? Introduction - How Distributed Tracing Works For the uninitiated, Distributed Tracing is a technology that allows us to track a single request as it traverses several different components/microservices of a distributed environment. Each network call made in the request's path is captured and represented as a span. To enable this, distributed tracing tools insert a unique trace context (trace ID) into each request's header and implement mechanisms to ensure that the trace context is propagated throughout the request path. How a distributed trace represents a request path Why we need Distributed Tracing in the first place Distributed tracing is unique because it focuses on a as the unit for observability. In a monitoring/ metrics platform, a (e.g., a service, host) is the fundamental unit that is being observed. One can ask these platforms questions about the behavior of this unit as a whole, over time. For example, what is this service's health/ throughput /error rate in a specific timeframe? request component With logs, the fundamental unit being observed is an - e.g., whenever an event occurs during code execution, print some information. These "events" are subjectively defined by developers while writing code. The challenge with logs is that they are all disjointed, with each component printing its own form of log messages in isolation, with no easy way to connect them together to make sense. event In contrast, with distributed tracing what is being observed is a single request as it traverses several components. This allows us to ask questions about the distributed system as a whole and understand what occurred where in a complex, interconnected system. The basic case for distributed tracing lies in the argument that this orientation around requests is the . And as a result, it is also the most intuitive for how we'd like to examine and troubleshoot distributed architectures. closest to the end user's experience The evolution of Distributed Tracing Distributed Tracing has risen in importance due to the widespread adoption of distributed software architectures in the last decade. The modern microservices-based architecture is an evolution from the late 90s internet growth story, when it became common to use systems. request-response "With the late 90s and explosive growth of the internet, came the huge proliferation of , such as two-tier websites, with a web server frontend and a database backend... Requests were a new dimension for reasoning about systems, orthogonal to any one machine or process in aggregate." request-response systems - Distributed Tracing in Practice, O'Reilly Media In these microservices architectures, every single request ends up hitting many (10s or even 100s of microservices), making several network calls in between. Refer below for Uber's microservices architecture, which has 3000+ services. Uber's microservices architecture from 2018. Source: https://www.uber.com/en-IN/blog/microservice-architecture/ In such complex systems, distributed tracing becomes critical for any form of troubleshooting. As a result, Distributed Tracing was pioneered by large companies that were early adopters using large, complex, distributed environments. Google's Dapper paper released in 2010 was the beginning of distributed tracing In the next few years, two more companies open-sourced their own distributed tracing systems (Twitter open-sourced Zipkin in 2012 and Uber open-sourced Jaeger in 2017). Zipkin and Jaeger continue to be among the most popular distributed tracing tools even today Since 2016, there has been a significant effort to standardize distributed tracing across components through the OpenTracing project. OpenTracing eventually became in 2019. OpenTelemetry is widely popular and has thousands of contributors globally OpenTelemetry Now Distributed Tracing is widely regarded as the third "pillar" of observability alongside metrics and logs. Most major monitoring and observability players provide distributed tracing tools as part of their products. State of Distributed Tracing: Theory vs Reality However, despite the promise, excitement, and community effort, the adoption of distributed tracing today is around ~25%. It is not uncommon to find companies on microservices architectures who are making do with logs and metrics, even though they clearly need distributed tracing. At the same time, Mean-Time-To-Resolve production errors are going up in the world today. to resolve production issues today. 73% of companies report it takes over an hour Ask any developer what the most painful moments in their life are and they'll talk about time spent debugging a Sev-1 error in production with what seemed like a few hundred people breathing down their neck. Seems then, that any company that cares about its MTTR (which is nearly every company) should be using distributed tracing, and the adoption should have skyrocketed in this environment. But the actual numbers do not support that - so what gives? Challenges with Distributed Tracing today There are several problems with distributed tracing today that companies have to overcome to get value - all of which don't get discussed as widely in the mainstream narrative. 1. Implementation is hard! To implement distributed tracing in a service today, we need to make a code change and a release. While making code changes is a common-enough ask for observability, the challenge specifically with distributed tracing is this - e needs to be instrumented to get a distributed trace, or the trace breaks. very service or component One cannot just get started with a single service - as one can with monitoring or logging - and realize value. Distributed tracing requires instrumentation across a to generate usable traces. collective set of services This requires coordination across several teams and service owners to make changes in their services. So it becomes an organizational problem- imagine getting 100s of teams to instrument their services over several months before you can realize value. This is the biggest challenge with distributed tracing today. 2. Need for complex sampling decisions Next, the volume of trace data generated by Distributed Tracing can be overwhelming. Imagine hundreds of services each emitting a small amount of trace data for . This is going to be millions of requests per second, and makes distributed tracing expensive both in terms of storage and network bandwidth. every single request While logging also does the same thing (and emits per request, which is then managed by massive log aggregation tools), the difference is that most companies today logging. Introducing one more data type which is going to be almost as voluminous as logging is a daunting task and will likely double the spend. more data already have To handle this problem of cost, all distributed tracing systems today use and record only a subset of traces. The common sampling rates in practice today are between 0.1% to 2%. The rationale is that even 1% of samples are sufficient to give a decent aggregate picture of where the performance bottlenecks are. sampling Most platforms today let customers choose their sampling strategy and make their own cost-visibility trade-offs. However, this decision process adds to the already complex overhead of instrumenting and managing a distributed tracing system. 3. But sampling meaningfully diminishes the value Let's assume a company goes through the effort of instrumenting every service/ component and then deciding the sampling strategy to ensure they don't break the bank. What now - should we expect MTTR to drop dramatically? No, because developers can't use distributed tracing to actually troubleshoot issues, because of sampling. Imagine a developer's experience - "I can't find the issue is there. I generated the error, but I cannot find the corresponding trace". I know So what happens? Developers stop trusting the quality of distributed tracing data and revert to their regular methods for debugging/ troubleshooting (i.e., using logs) 4. Developer usage is low frequency Given these constraints, today Distributed Tracing is primarily sold as a way to troubleshoot . performance problems Remember that a basic distributed trace really just tells us who called who and how long each span took. Distributed traces don't tell us that caused the error/ high latency. For that, developers still have to look at the log message and/ or reproduce the issue locally to debug. what happened within the service In a typical company, performance issues are likely <10% of the total. So in reality, distributed tracing is only useful for this small segment of issues. The average developer who ships and owns a service is using a distributed tracing tool maybe 2-3 times a year. Impact of all these challenges In summary: Distributed tracing is hard to implement Distributed Tracing needs extensive sampling to control costs But sampling reduces the value considerably As a result, developers only use tracing for the odd one-off performance use case All this makes the RoI case for distributed tracing quite fuzzy. In a typical hype cycle, what we can say is that we are now past the stage of inflated expectations and disillusionment is beginning to settle in. If we think in terms of end-state though, if the future of computing systems is distributed, then distributed tracing is naturally the most fundamental vector for observability. In that world, any company with a distributed architecture uses tracing as the primary mechanism for troubleshooting anything occurring in production - true "observability" - vs the passive monitoring of systems we have today. Before we can get to that end-state though, we will need several improvements over the status quo. The good news is that much of this is already underway. Let's look at each of them. So what can we expect to see in the future? Future of distributed tracing Instant instrumentation with no code changes Imagine dropping in an agent and being able to cover an entire distributed system (all services, components) in one go without code changes. This looks realistically possible in the next 2-3 years. OpenTelemetry's already enable this for some programming languages (however fall short in compiled languages such as Go). In parallel, are evolving to . Between the two, we can safely expect the instrumentation problem to be solved in a few years. auto-instrumentation libraries technologies like eBPF enable system-wide instrumentation with no code change Sampling gives way to AI-based selection of requests-of-interest In an LLM world, random sampling begins to look like a relic from the dark ages. Ideally, we should be able to look at 100% of traces, identify anything that looks anomalous, and store that for future examination. No more random sampling. If we think about it, we don't really care about the ~95% "happy requests". We only care about the ~5% of anomalous traces - errors, exceptions, high latency, or some form of soft errors. So we just need a way to look at 100% and pick out the interesting 5%. There are mechanisms like that aim to do this today. In tail-based sampling, the system waits until all the spans in a request have been completed, and then based on the full trace, decides whether it has to be retained. tail-based sampling The main challenge with tail-based sampling is that you have to store all the spans of a trace until the whole request is completed and then decide whether to keep/ discard the trace. This means we store every single request, with all the spans, for a certain period (until the request completes) - this requires a separate data architecture with components for load-balancing, storage & processing which is highly complex and expensive. OpenTelemetry has a , however, it is not yet mature and has several (due to the problem mentioned above). Meanwhile, several companies including are working on using AI to make anomaly detection efficient and scalable. tail-based sampling collector scalability challenges ZeroK.ai With the fast pace of development in this space, we can reasonably expect this problem to also be solved in the next 3-5 years. The emergence of "rich" distributed traces that enable all debugging A true leap into the next generation of tracing will be when tracing evolves from the realm of "performance issues only" to "all issues". That is when the true power of distributed tracing is unleashed. For this to be possible, each trace needs to have rich context. Imagine a scenario where each span in each trace has: Request & response payloads (with PII masking) Stack traces for any exceptions Logs Kubernetes events Pod states And anything else that occurred along that span All in one integrated, seamless flow. And imagine if the trace one is looking for is super easy to find - there are no sampling-related data gaps, issues are deduped & grouped, and can be filtered across several dimensions. This then, is all a developer needs to debug any software issue. And potentially, all an AI model needs to diagnose and point a developer to what's going wrong. In this world, the trace becomes the primary axis for observability, . That is what the end-state of distributed tracing could look like - while it's not here yet, it is visible from where we are today. replacing logging The main barrier to making this possible is the explosion in data volume that storing all this context data will cause. We will require deep innovation in data processing and storage architectures to make this possible. It is still early days yet and we have to wait and see what happens here. Summary In summary, Distributed Tracing is a necessary and intuitive view required for being able to observe distributed application architectures in production. The first generation of distributed tracing, while promising, has been beset by several challenges which have made it difficult for companies to get value from tracing, which has stymied adoption somewhat. However, several exciting developments are occurring in the space which are expected to make tracing easier, simpler, and more powerful than what we have today, making observability more seamless in the future.

Stories, News & Mentions about TRACXN

Tracxn’s Stories on HackerNoon

Tracxn's News & Mentions