While the project has drawn $1 trillion in its first decade, according to estimates from think tank Green Finance & Development Center, the momentum has tapered off in recent years. China’s ...

Xi’s $1 Trillion ‘Project of the Century’ Gets a Reality Check

Amazon said two prototype satellites launched on Friday on a test mission of the company’s Project Kuiper broadband internet constellation are stable in orbit and already collecting important data.

Project Kuiper

Commissions do not affect our editors' opinions or evaluations. Project portfolio management (PPM) is a critical aspect of project management that helps companies best determine which resources to ...

Best Project Portfolio Management Software (2023)

The Sims 5 is very much happening, but Project Rene is breaking new ground for EA's beloved franchise. The recent news that Sims 5 will "coexist" with other Sims games means that while it will ...

The Sims 5 Project Rene: Everything we know about the future of The Sims

2. They possess extraordinary organizational skills Top-notch project managers are highly organized individuals. “But it’s not just about making a list and sticking with it,” says Hema ...

20 traits of highly effective project managers

Module categories: Mail 76,052 sites report using this module Created by Allie Micka on 16 November 2005, updated 23 March 2023 Drupal 10 is here! This module supports Drupal 10 and is being tested against Drupal 10 on a weekly basis. See https://www ...

Mime Mail

Using some Roblox Project Slayers codes is a great idea when you are struggling with the game and want to get a headstart. This way, you will earn free art spins, breathing resets, race resets ...

Roblox Project Slayers Codes (October 2023)

Get the most recent info and news about Kloud Project on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Kloud Project

In 2021, hosted the first-ever awards!! HackerNoon Startup of the Year The goal of this campaign was to give recognition to the hardworking startups who pulled through the pandemic -- growing and thriving (despite all of the challenges). More than 44,000 startups were nominated all across the globe! They were selected from over 4,553 cities, with a total number of * cough * “4 billion” votes. Insane, right? To see who is a winner in each respective city, visit and pick a city. The name at the top of the list is the winner. (Compromised awards are under review. Cities with tie votes will be updated to have two winners!) startups.hackernoon.com The will announce the winners per region in their respective cities. Startups Winners Announcement series Here are the winners in the : Asia region Top Startup Cities in East Asia Seoul The Startup of the Year in is with 377 votes! Seoul dable Central is the Startup of the Year! It came out ahead with 121 votes. UpNow Central Sheung Wan came out as the Startup of the Year in with 154 votes. Votee Sheung Wan Tokyo The Startup of the Year in is with 41 votes! Tokyo ACES Wan Chai The Startup of the Year in is with 54 votes! Wan Chai Zetl Osaka, Japan In , is the winner! Osaka Nanophoton Taiwan came out as the Startup of the Year in ! Tico Taiwan Dongcheng, China In , is declared the Startup of the Year! Dongcheng Waiju Taipei came out as the Startup of the Year in ! Tico Taipei Akasaka In , is the winner! Akasaka WAKUWAKU Chuo In , is declared the Startup of the Year! Chuo GO-NET Ginza came out as the Startup of the Year in ! Netyear Ginza Kanazawa In , is the winner! Kanazawa Granzella Minami-aoyama In , is declared the Startup of the Year! Minami-aoyama Tokyo6 Entertainment Shibuya came out as the Startup of the Year in ! Hupro Japan Shibuya Toranomon In , is the winner! Toranomon Small Worlds Toshima In , is declared the Startup of the Year! Toshima Altplus Guangzhou, China came out as the Startup of the Year in ! Leyaoyao Guangzhou Hong Kong, Hong Kong In , is declared the Startup of the Year! Hong Kong Episode Six Kunming In , is the winner! Kunming Green A Sapporo In , is declared the Startup of the Year! Sapporo Polar Star Space Shirokanedai came out as the Startup of the Year in ! Nupp1 Shirokanedai Wuhan In , is the winner! Wuhan CSICEE Baoan, China In , is declared the Startup of the Year! Baoan Jumper Baoshan, China In , is declared the Startup of the Year! Baoshan Renren Games Chengdu, China came out as the Startup of the Year in ! Ju You Xi Chengdu Fujisawa, Japan In , is the winner! Fujisawa Scohia Pharma Fukuoka, Japan In , is declared the Startup of the Year! Fukuoka Kyulux Haidian, China came out as the Startup of the Year in ! 51vv Haidian Hongkou, China In , is the winner! Hongkou Kidstone Jiading, China In , is declared the Startup of the Year! Jiading IMRobotic Kawasaki, Japan came out as the Startup of the Year in ! QD Laser, Inc. Kawasaki Kyoto, Japan In , is declared the Startup of the Year! Kyoto Hacarus Minato, Japan In , is the winner! Minato Visionary Fun Nagoya, Japan came out as the Startup of the Year in ! Prodrone Nagoya Ningbo, China The winner in is . Ningbo OlymTech Corporation Aomori In , is declared the Startup of the Year! Aomori Forte Japan Bandai-cho In , is the winner. Bandai-cho Delta-Fly Pharm Changchun In , the Startup of the Year is . Changchun Kaidi Animation Fuchu In , is declared the Startup of the Year! Fuchu TMS Fukuoka-shi The winner in is . Fukuoka-shi SkyDisc Fukushima In , the Startup of the Year is . Fukushima Matsuri Technologies Gotanda In , is the winner. Gotanda Basset Hachioji In , is declared the Startup of the Year! Hachioji Adriakaim Harbin The winner in is J . Harbin oining Technology Iwaki In , is the winner. Iwaki Tao Engineering Izumo In , the Startup of the Year is . Izumo PuREC Kagoshima In , is declared the Startup of the Year! Kagoshima Farmers Support Kamakura In , the Startup of the Year is . Kamakura KAYAC Kami-meguro The winner in is . Kami-meguro Buzzreach Kanagawa In , is the winner. Kanagawa LoiLo Kashiwa In , the Startup of the Year is . Kashiwa Intelligent Surfaces Kita In , is declared the Startup of the Year! Kita Ax Robotix Kobe The winner in is . Kobe Coconomi Koriyama In , the Startup of the Year is . Koriyama Tocoton Kumamoto In , is the winner. Kumamoto Bloom Technology Kunitachi In , is declared the Startup of the Year! Kunitachi Trys Liaocheng The winner in is . Liaocheng Jinzheng Animation Minami In , is declared the Startup of the Year! Minami ATLAB Miyazaki In , is the winner. Miyazaki Caster Nagano In , is declared the Startup of the Year! Nagano Precision Forestry Nagaoka The winner in is . Nagaoka Plant Form Nihombashi-hamacho In , the Startup of the Year is . Nihombashi-hamacho Shizen Energy Niigata In , is declared the Startup of the Year! Niigata UniBio Japan Nishi-shinagawa In , is the winner. Nishi-shinagawa Play Heart Okayama In , is the winner! Okayama Arch Farm Okinawa In , is declared the Startup of the Year! Okinawa BlogNinja Otemachi In , the Startup of the Year is . Otemachi beBit, Inc Qingdao Shi In , is the winner! Qingdao Shi Manyoubang Quanzhou In , is declared the Startup of the Year! Quanzhou KungFu Animation Group Roppongi The winner in is . Roppongi Holo-X Saitama In , the Startup of the Year is . Saitama GameTector Sendai In , is the winner! Sendai Sendai Smart Machines Setagaya In , is declared the Startup of the Year! Setagaya Skyestate Shenzhen In , the Startup of the Year is . Shenzhen Aiator Shinjuku In , is declared the Startup of the Year! Shinjuku Scalar Shizuoka In , is the winner! Shizuoka PiPhotonics Sumida The winner in is . Sumida Holotch Tachikawa In , the Startup of the Year is . Tachikawa PlusOne Takamatsu In , is declared the Startup of the Year! Takamatsu Ken Robotec Tokushima In , is the winner! Tokushima DennoKotsu Toyonaka In , the Startup of the Year is . Toyonaka FISH BIOTECH Tsukuba In , is declared the Startup of the Year! Tsukuba WARPSPACE Wakayama In , the Startup of the Year is . Wakayama 4D Sensor Xiangzhou In , is the winner! Xiangzhou ALLiN Edutech Group Xianyang The winner in is . Xianyang TONLY Xiaoshan In , is declared the Startup of the Year! Xiaoshan GONGHUI Yamaguchi In , the Startup of the Year is . Yamaguchi Archis Yangpu The winner in is . Yangpu Tmxmall Yau Tsim Mong In , is the winner! Yau Tsim Mong Airnugget Yeoksamdong In , the Startup of the Year is . Yeoksamdong WizSchool Yichang In , is declared the Startup of the Year! Yichang Hubei Kento Electronic Yokohama In , is the winner! Yokohama Next Innovation Yubei In , the Startup of the Year is . Yubei Ophiux Network Yueqing The winner in is . Yueqing Yueqing Easythink Media Zhuhai In , is the winner! Zhuhai Aijuba Hongo In , the Startup of the Year is . Hongo Smile Robotics, Inc Admiralty In , is declared the Startup of the Year! Admiralty BUNCH Southern In , the Startup of the Year is . Southern Retykle Yongdungpo In , is the winner! Yongdungpo Push News Taichung In , is declared the Startup of the Year! Taichung Wondercise Cyberport, Hong Kong In , the Startup of the Year is . Cyberport Animoca Brands Hefei, China In , is the winner! Hefei Tsinglink New Territories, Hong Kong The winner in is . New Territories, Hong Kong Zeek Dezhou In , the Startup of the Year is . Dezhou Dezhou Baji Kitakyushu In , is declared the Startup of the Year! Kitakyushu Reif Seongnam In , is the winner! Seongnam EOFlow Yanshi In , the Startup of the Year is . Yanshi CASI Vision Top Startup Cities in South Asia Hyderabad, India The Startup of the Year in is with 962 votes Hyderabad Quixy Lahore, Pakistan is the Startup of the Year! It came out ahead with 186 votes. SmartWindows Lahore Bangalore, India When it comes to , is the Startup of the Year with 210 votes! Bangalore vBridge Hub Pune, India couldn’t be beaten for Startup of the Year in , it won with 129 votes! InfraCloud Pune Jaipur, India The Startup of the Year in is with 166 votes Jaipur Emizen Tech Lucknow, India is the Startup of the Year! It came out ahead with 279 votes. Keeros Superfoods Lucknow Noida, India came out as the Startup of the Year in with 156 votes! Bagisto Noida Mumbai, India When it comes to , is the Startup of the Year with 96 votes! Mumbai Sequretek Sylhet couldn’t be beaten for Startup of the Year in , it won with 122 votes! DataXpie Sylhet Mylapore, India In , is the winner! Mylapore Yalimart Goa, India In , the Startup of the Year is Goa Filter Coffee Delhi, India In , is declared the Startup of the Year! Delhi EkAnek Networks Karachi, Pakistan The winner in is Karachi Bykea Ahmedabad, India In , is the winner Ahmedabad MageComp Dhaka, Bangladesh The winner in is Dhaka SOLshare Islamabad, Pakistan In Islamabad, is declared the Startup of the Year! TAG Gurgaon, India In , the Startup of the Year is Gurgaon ixigo Bengaluru, India In , is the winner! Bengaluru Prakshep Private Limited Haryana, India couldn’t be beaten for Startup of the Year in ! Footprints Childcare Haryana Kolkata, India In , is the winner! Kolkata Wow! Momo New Delhi, India In , is declared the Startup of the Year! New Delhi Rusk Media Indore, India The winner in is Indore Chattar Patar Kormangala In , is the winner Kormangala Sanctum Technologies Ghaziabad came out as the Startup of the Year in Craftfurnish.com Ghaziabad! Mohali In , the Startup of the Year is Mohali DataKund Mysore The Startup of the Year in is Mysore Excelsoft Karnal came out as the Startup of the Year in ! Dunar Karnal Kolhapur In , is the winner! Kolhapur Zanvar Group Patiya In , the Startup of the Year is Patiya Kloud Project Kathmandu In , is declared the Startup of the Year! Kathmandu Vairav Technology Bangalore City, India In is the winner! Bangalore City, PagarBook Ernakulam The winner in is Ernakulam Epixel Solutions Colombo, Sri Lanka couldn’t be beaten for Startup in the Year in MAs Foods Colombo Andheri, India When it comes to , is the Startup of the Year! Andheri DoorMint Kanpur, India came out as the Startup of the Year in ! ServiceWalaa Kanpur Faisalabad In , is the winner! Faisalabad DigiKhata Nasik The Startup of the Year in is Nasik Sahyadri Farms Patna came out as the Startup of the Year in ! Asvin Tech Patna Chennai, India In , is the winner! Chennai Agnikul Cosmos Faridabad, India In , is declared the Startup of the Year! Faridabad UClean Panjim, India The winner in is Panjim TempoGO Ranchi, India In , is the winner Ranchi Osam Dairy Surat, India In , is declared the Startup of the Year! Surat Wishbook Thane, India In , is the winner! Thane Delight Foods Trivandrum, India came out as the Startup of the Year in ! GenRobotic Trivandrum Vijayawada, India In , the Startup of the Year is Vijayawada Eruvaka Technologies Visakhapatnam, India In , Ltd is declared the Startup of the Year! Visakhapatnam Greenline Labs Pvt Ahmadabad In , is the winner! Ahmadabad KiwiQA Services Bandra came out as the Startup of the Year in ! Car Club Bandra Bombay In , is the winner! Bombay Locobuddy Mobile Technology (Sportobuddy) Cochin In , the Startup of the Year is Cochin Tuttifrutti Interactive Gandhinagar In , is declared the Startup of the Year! Gandhinagar Kroop AI Guwahati In , the Startup of the Year is Guwahati GNRC Hospitals Jamshedpur In , is declared the Startup of the Year! Jamshedpur Intelliber Kabul The winner in is Kabul Miracle Shop (Vocinno Technology) Karur In , is the winner Karur Karur Vysya Bank Kochi The winner in is Kochi BestDoc Kozhikode In , is declared the Startup of the Year! Kozhikode OfficeKit HR Ludhiana In , the Startup of the Year is Ludhiana Trendy Beat Network Madhapur In , is the winner! Madhapur Head Digital Works Mangalore When it comes to , is the Startup of the Year Mangalore Hangyo Nagpur came out as the Startup of the Year in ! UPrudence Agro Nagpur Rawalpindi The Startup of the Year in is Rawalpindi Pro Options Marketing Rishikesh came out as the Startup of the Year in ! Descatuk Rishikesh Secunderabad couldn’t be beaten for Startup of the Year in Prakriti Biosystems Engineering Secunderabad Tirupur In , is the winner! Tirupur Tubeknit Fashions Udupi The winner in is Udupi Krishitantra Top Startup Cities in Southeast Asia Singapore, Singapore The Startup of the Year in is with 6556 votes! Singapore Cake DeFi Hanoi, Vietnam is the Startup of the Year! It came out ahead with 683 votes. DeFi For You Hanoi Kuala Lumpur, Malaysia came out as the Startup of the Year in with 109 votes! Trambellir Kuala Lumpur Pasig, Philippines When it comes to , . is the Startup of the Year with 110 votes! Pasig Quickreach Inc Bangkok, Thailand couldn’t be beaten for Startup in the Year in , it won with 60 votes! Perception Codes Bangkok Johor Bahru In , the Startup of the Year is . Johor Bahru AGR1 Bandung, Indonesia In , is declared the Startup of the Year! Bandung Prosa Portugal The winner in is . Portugal SCEMAI Tangerang, Indonesia In , is the winner Tangerang Yummy Corp Jakarta, Indonesia In , is the winner! Jakarta Tokocrypto Dki Jakarta, Indonesia In , is declared the Startup of the Year! Dki Jakarta Limakilo Bangkok Metropolis In , is the winner! Bangkok Meticuly Medan The winner in is ! Medan DIKA Manila, Philippines In , is declared the Startup of the Year! Manila PawnHero Phnom Penh, Cambodia In , the Startup of the Year is Phnom Penh vKirirom Bali In , is the winner! Bali Radius App Penang When it comes to , is the Startup of the Year Penang EasyParcel Surabaya came out as the Startup of the Year in ! Super Surabaya Setiabudi, Indonesia The Startup of the Year in is Setiabudi Chilibeli Cebu City came out as the Startup of the Year in MiCab Cebu City! Jawa couldn’t be beaten for Startup in the Year in DAUR Jawa Ho Chi Minh City, Vietnam In , is the winner! Ho Chi Minh City Pharmacity Selangor, Malaysia is the Startup of the Year! BabyDash Selangor Jakarta Pusat, Indonesia In , is the winner! Jakarta Pusat StickEarn Semarang When it comes to , is the Startup of the Year! Semarang Food Bear Petaling Jaya, Malaysia The winner in is . Petaling Jaya Deemples Kebayoran Baru, Indonesia couldn’t be beaten for Startup in the Year in ! Yogrt Kebayoran Baru Kuningan The winner in is . Kuningan Prinzio Bukit Merah came out as the Startup of the Year in ! DC Alliance Bukit Merah Huai Khwang In , the Startup of the Year is . Huai Khwang AppMan Co.,Ltd Kuta In , is declared the Startup of the Year! Kuta Gringgo Mandaluyong came out ahead as the Startup of the Year! Work Inspire Mandaluyong Midview City In , is the winner! Midview City BioMers Phuket The Startup of the Year in is ! Phuket Sapahn Yangon, Myanmar The winner in is . Yangon Wave Money Denpasar When it comes to , is the Startup of the Year! Denpasar Revasi Nakhon Ratchasima, Thailand The winner in is . Nakhon Ratchasima Siam Lottery co., Ltd Taguig, Philippines In , is the winner! Taguig PayMongo Bogor came out as the Startup of the Year in ! TIRANYX Bogor Senai, Malaysia won as the Startup of the Year. Nutrition Technologies Senai Bekasi Kota couldn’t be beaten for Startup in the Year in ! 8villages Bekasi Kota Top Startup Cities in Middle East Tel Aviv, Israel The Startup of the Year in is with 1,065 votes! Tel Aviv hyro Beirut, Lebanon is the Startup of the Year! It came out ahead with 1,276 votes. Hovi Digital Lab Beirut Riyadh, Saudi Arabia With 350 votes, came out as the Startup of the Year in ! Flicron Riyadh Tel Aviv-yafo, Israel When it comes to , is the Startup of the Year with 167 votes! Tel Aviv-yafo Cymbio Haifa, Israel couldn’t be beaten for Startup in the Year in , it won with 183 votes! Pico Haifa Amman, Jordan In , with 141 votes, is the winner! Amman Edexy Dubai, United Arab Emirates In , the Startup of the Year is with 74 votes. Dubai Ziina Doha, Qatar In , is declared the Startup of the Year! Doha SkipCash Tehran, Iran The winner in is . BeyondBoard AR Tehran Jerusalem, Israel In , is the winner! Jerusalem SolidBlock Rishon Le Zion The Startup of the Year in is ! Rishon Le Zion Passparto Ness Ziona, Israel is the Startup of the Year! Soapy Ness Ziona Kuwait, Kuwait came out as the Startup of the Year in ! Brapper Shop Kuwait Manama, Bahrain couldn’t be beaten for Startup in the Year in ! Doctori Manama Lod, Israel is the winner in ! Phinergy Lod Agudat Israel, Israel When it comes to is the Startup of the Year with [vote ] votes! Agudat Israel, Aqaurius Engines Beersheba, Israel In , the Startup of the Year is . Beersheba Visual Estate Ltd Giv'atayim, Israel In , is declared the Startup of the Year! Giv'atayim WSC Sports Modi'in, Israel The winner in is . Modi'in Loginno Nazareth, Israel In , is the winner! Nazareth AR Guide In Medical Ramat Gan, Israel The Startup of the Year in is . Ramat Gan Datos Health Baghdad When it comes to , is the Startup of the Year! Baghdad Mishwar Khobar In , the Startup of the Year is . Khobar Garage Misgav couldn’t be beaten for Startup in the Year in ! Eden Shield Misgav Sharjah came out as the Startup of the Year in ! Bookends Sharjah Affula In , the Startup of the Year is . Affula eMotion Technologies Bene Beraq, Israel In , is the winner Bene Beraq Reflectiz Caesarea, Israel In , the Startup of the Year is . Caesarea CardiacSense Dubayy In , is declared the Startup of the Year! Dubayy North Ladder Muscat couldn’t be beaten for Startup in the Year in ! iLab Marine Muscat Tirat Carmel In , the Startup of the Year is . Tirat Carmel PassCall Advanced Technologies Hod Hasharon, Israel The winner in is . Hod Hasharon Idomoo Holon, Israel came out ahead as the Startup of the Year! OptimalPlus Holon Jeddah, Saudi Arabia When it comes to , is the Startup of the Year! Jeddah Arabian Combustion Technologies Ramat Hasharon, Israel is the Startup of the Year! Lumen Ramat Hasharon Al Khobar came out as the Startup of the Year in ! Swyft Al Khobar Bet Shemesh In , is the winner! Bet Shemesh Degania Medical Binyamina When it comes to , is the Startup of the Year! Binyamina OncoHost Makkah In , the Startup of the Year is Makkah Scimagine Or Akiva Last but not least, couldn’t be beaten for Startup in the Year in the city of ! WMT Or Akiva More Information: If you’re a winner, please answer ! If you’re a reader, participant, just a curious individual, consider answering one of our interviews . this interview here Read the FAQs about or . Startups here here To see all of the winners you may visit , pick any city and view the name that appears at the top. this site If you’re interested in partnering with us, we invite you to look through some of these or reach out to our business department. unique sponsorship options Perhaps creating your very own tickles your fancy or maybe you have a whole new idea for how we can partner! writing contest Thank you to all of our sponsors! We truly appreciate your contribution to the tech world and thus the future! The Startups Awards are sponsored by: , , , , , , , , , , Blockster Brex FreeCodeCamp LaunchDarkly Coil Udacity Udemy Vercel Asia Pacific Transgender Network Bybit Everscale Network The post was written by Kushal Tamrakar and Valerie Ejegi, edited by Hang Ngo. If you have any questions or concerns or would like to report a discrepancy please email ellen@hackernoon.com.

Meet the Winners of Startups in Asia

On the seventh day, amidst the darkness, God granted ProjectDiscovery, aiding brave souls in their perilous pen-testing journey. In the wake of ProjectDiscovery's ascent, tools like Nuclei, Katana, Subfinder, Naabu, and the latest addition, cvemap, emerged to fortify the arsenal of the brave. CVEmap and Nuclei missing feature And most of the were disappointed by the missing feature to “map” results directly to existing nuclei templates. penetration testers cvemap This absolutely necessary feature does not exist today, and I hope it will change - I will create an issue on to address this problem. GitHub However, I decided to fix this problem in this guide myself because a solution might not come at all. Fortunately, the existing nuclei templates use the same naming convention as the CVE one, and nuclei support an argument where you can enter the ID of the desired templates directly into the input.  CVEmap’s flag  nicely informs that the template exists in the Nuclei, but unfortunately, it doesn't tell how and where to find it. -template By combining this information, I came up with an ugly but short solution. Solution A bit of allows us to create the following script. bash glue This script scans targets from a file, with Nuclei templates filtered based on criteria from cvemap. Command expansion ( ) creates the resulting list of template IDs, separated by a comma, which Nuclei accepts as valid template selection. targets.txt $(…) nuclei -l "targets.txt" \
       -id "$(cvemap -template -list-id -silent \
        -age '<365' -q '"remote code execution" is_remote:true' -vendor cisco |
        tr -s '\n' ',')" \
       -o "output.txt" Precisely, in this case, the command scans CVEs of the Cisco products that provide RCE to the attackers and are less than one year old. Isn't that great!? Feel free to change the query on line 3 for your purpose. The other lines are mandatory. Postface Why this feature doesn't exist officially, I don't know, but if I had to bet, it's because they want to use the synergy of these tools in their paid cloud platform, and they don't want to give all the features for free but at the same time they want to be opensource. So they created a flag with the flag, which informs you that the template exists, but which one it is, and so on, you must look up yourself. cvemap -template Resources https://docs.projectdiscovery.io/tools/cvemap/running https://cloud.projectdiscovery.io/?ref=api_key https://github.com/projectdiscovery/cvemap https://github.com/projectdiscovery/nuclei

The Pen-Tester's Arsenal: ProjectDiscovery's CVEmap to Nuclei Template Mapping

Guardians at the Gateway: Unmasking the Constraints of Security Groups in IaaS Environments As the demand for dynamic and scalable computing resources continues to surge, the Cloud allows key components and benefits for teams. In deployments, security groups are essential elements for managing instances security. Security groups allow you to specify the traffic that is alowed or blocked at the virtual instances. They act as a virtual firewall by filtering network traffic according to network rules that the network administrator can define. DevOps IaaS The guardians at the Gateway The cloud allows developers to have scalable environments. With IaaS infrastructures, OPS teams make deployment easier, setting up CI/CD pipelines and benefits from innovations such as infrastructure as code and new security services. These environments are designed to be API native and facilitate the transition to serverless in the short term. Nowadays, OPS and security teams divide IaaS networks into subnets with different virtual machines. OPS or SecOPS teams configure the Security Groups by creating rules based on ports, IP addresses and protocols. Then each virtual instance on the infrastructure can be associated with the security group. On the other hand, the security group can also be associated to one more virtual instances. It’s a security mechanism that allows more flexibility as they can be added, modified or even deleted based on the evolving needs. Security groups help ensure that different resources remain isolated. In the event of a security breach, they restrict the lateral movement of attackers. Limits of Security groups Security groups are not typically considered a limitation but a fundamental component of cloud security. They are designed by cloud providers to control incoming and outgoing traffic to cloud instances. They are part of the security measures to secure data and applications hosted in the cloud. Security Groups are not the solution to all Cloud security issues. They as they are not an observability tool. Certain industries or regulations require high logging level and in-depth security audits. Only Firewall can help meet the compliance requirements. can’t log and monitor security events As network filtering tools they such as intrusion detection, traffic inspection, WAF protection, DDoS attack prevention or traffic monitoring that improve the overall security. don’t integrate advanced security features However, it's fundamental to note that how security rules are implemented can create limitations or security issues. For example, overly restrictive rules could prevent legitimate traffic from reaching cloud resources and leading to connectivity issues. In large cloud infrastructures, managing rules can become complex. There are other limitations to keep in mind: Security groups don't operate at layer 7 of the OSI model. The application layer protection require a Web Application Firewall (WAF). Limited protection against advanced threats that may require IDS or IPS. Best practices Here are some best practices when configuring security groups effectively in a cloud environment: Follow the Principlen of Least Privilege: Grant only the necessary permissions in your security groups. Explicit Whitelisting: Define rules based on specific IP, other security groups IDs rather than broad ranges. Use descriptive names: Give descriptive names that simplifies troubleshooting and auditing. Audit rules: Regularly review the security groups rules to ensure they are up to date. Remove rules to minimize the attack surface. Automate security group management with infrastructure as a Code with tools like Terraform. Document the security groups configuration, rules, forbidden protocols or ports. Test and validate rules to ensure they are working as expected and instances communicate without issues. As the cloud ressources grow and scale, managing the security groups manually can be difficult. Using infrastructure as a Code and automation tools can help while they don't introduce their own complexity. Conclusion As a fundamental part of any IaaS infrastructure, Security groups controls network traffic to virtual instances of the cloud. They can help secure virtual machines at the network considerations. However only Firewalls can enforce cross-cutting security policies across different areas of the cloud infrastructure and create true . DMZ Security groups are not the only security feature. Virtual instances should be hardened, monitored and audited. Security groups play an important role in a mico-segmentation startegy. The granularity provided help ensure the principle of least privilege, ensuring each virtual instances only communicates with what's necessary. They can automatically adapt to changes in the infrastructure with real time enforcement. This approach is well suited for immediate protection against unauthorized access attempts. In summary, security groups are not inherently a limitation but their proper management is essential to ensure effective security and avoiding undesirable limitations. Implementing correctly security groups reduce the surface attack of virtual instances and protect data. They contribute significantly to a robust micro segmentation strategy and security posture. It's a security feature offered by several major cloud service providers (with different terminologies such as network security groups, security lists or Firewall rules).

The limits of Security Groups in IaaS Infrastructures

Image segmentation plays a crucial role in accurately identifying and delineating objects of interest within an image. In autonomous driving, computer vision algorithms are applied to solve the task of road surface segmentation. This task is challenging as you cannot rely on one type of images only — both cameras and LiDARs have their strengths and flaws. For example, LiDARs provide accurate depth information, but they typically generate sparse point clouds, so they cannot precisely segment objects within the scene. And they may produce distorted clouds when they face transparent or reflective surfaces. Cameras don’t capture the depth, but they provide complete information about the shape, texture, and color of objects. This leads us to a simple idea that an effective fusion of point clouds and images in training 2D road segmentation may harness the advantages of each data domain. The problem is such fusion requires labor-intensive annotation of both datasets. So, can we make data annotation more efficient to enjoy the benefits of a multi-sensor setup for road segmentation? I currently work at . This company provides cargo transportation service and manufactures its own autonomous electric vehicles. As a deep learning engineer, I specialise in the development of 3D detection systems for the autopilot of self-driving vehicles. So, at Evocargo, we decided to find a way to improve the efficiency of road surface segmentation that would keep prediction quality high and reduce annotation costs. After some time of researching and experimenting, my colleagues and I created . Thus, lidar points projected onto the image and used in further training as ground truth masks provide comparable quality of image segmentation and allow training of models without standard annotated 2D masks. Evocargo an approach that effectively leverages lidar annotations to train image segmentation models directly on RGB images In this post, I’ll describe our approach step-by-step and show some test results. If you want to dive deeper in our work, other methods research and our test results, then refer to our article in the IEEE Access journal. This article is supported by the published with the method implementation, processed datasets, and code base for future research. If you find our work useful for your research, please consider giving it a star ⭐ and citing the paper. ‘Lidar Annotation Is All You Need’ GitHub repository Training segmentation models in 4 steps The overall pipeline of our approach consists of four main parts: point cloud road annotation, data preparation, masked loss, and the segmentation model itself. Firstly, we get the data with road annotations in the point cloud domain. After that, we project points using homogeneous transformations and camera parameters. Then, using the projected points we get the road ground truth mask for loss calculation with added random noise. Images from the camera are processed by the segmentation model. Predictions and masks from the previous step are utilised by the Masked loss, which allows training the model using sparse ground truth data. Finally, after model training, we get an image with a segmented road. The training procedure as well as the Masked loss allow mixing projected ground truth with traditional 2D masks, which makes the approach flexible in terms of data. Now let’s look closely at each of the parts. 1 Point cloud data annotation To use lidar data during training, we need semantic segmentation annotations for point clouds. This could be done either manually using an open source point cloud annotation tool, such as , or using some algorithmic approaches. My colleague has described one such approach to road surface detection in his step-by-step guide . An algorithmic approach especially for road annotation could allow getting without manual annotation at all, but it needs fine tuning for specific data. At Evocargo we use both approaches. For simple areas, the road is annotated algorithmically, and for complex sections - manually. Semantic Segmentation Editor How to annotate 100 lidar scans in 1 hour 2 Data preparation An obtained point cloud is projected on the image plane using homogeneous transformations to get an image segmentation mask with the class label we require, in our case it is a road. For such projection, we use synchronised camera and lidar frames, accompanied by camera parameters, and a transformation matrix from lidar to camera frame. To project homogeneous point in lidar frame coordinates to point on an image plane, we use the equation: x = (x, y, z, 1)ᵀ y = (u, v, 1)ᵀ After transformation, we get points on the image as a mask of pixels, both for road class and for all other points from the lidar scan. We need other lidar scan points, because the lidar points are mostly located at the bottom of the image and the top of the image has no points at all potentially leading to inaccurate predictions in that area. To eliminate this effect, we add random points in the upper half of the mask (negative class) to balance the distribution of the points where we will calculate the loss. 3 Masked loss A crucial component of our method is the application of a masked loss function during model training. This approach eliminates the inherent sparsity in lidar-derived ground-truth masks. Unlike conventional loss functions, which consider the entire image masks to calculate errors, masked loss focuses solely on the regions where lidar points are present.  This targeted loss calculation ensures that the model's learning is concentrated on relevant regions, leveraging the information provided by lidar to enhance road segmentation accuracy. In other words, we force the model to learn road segmentation by measuring the error of predictions on a grid of points. It could be compared to looking at the image using shutter glasses (shutter shades). Masked loss for each image can be formulated this way: Masked loss is performed by applying a binary mask to the training images. This mask outlines the areas where the lidar points are located and projected onto the image plane. During the training phase, the loss is calculated only for those pixels that are under the mask, effectively ignoring large unannotated parts of the image. This method not only improves the efficiency of the training process, but also mitigates the problems represented by the sparse nature of lidar data. 4 Model training The final step involves training the segmentation model on the created dataset. The training process can be suitable for any segmentation model, and in our research we used PSPNet. At this stage, everything depends on the quality of your data, its quantity and available computing power. Promising test results We tested our approach on various datasets including open-source ones, such as Perception Waymo Open Dataset and KITTI-360 dataset. Each time we conducted a set of three experiments: using only 2D road ground truth, only projected points as ground truth, and a mix of these types of ground truth. And the road segmentation (% of IoU) results look promising: Experiment Trained on KITTI-360 dataset Trained on Waymo dataset 2D only (baseline) 92.3 96.1 Projected 3D only 89.6 94.7 mix 2D + projected 3D 92.7 96.3 These numbers mean that if you only have lidar segmentation and do not want to spend additional resources on 2D image annotations, it’s okay. The drop in quality compared to training only on 2D image masks may be insignificant. If you have resources to annotate data from both sensors, then simply combine these two types of annotations during the training process and get an increase in metrics. All-in-all that we observed during the research are: the benefits of the approach high-quality performance of neural networks in image segmentation tasks, fewer resources required to annotate data from several types of sensors, flexibility to adapt to different image types and segmentation tasks. Authors of the approach

Lidar Annotation is All You Need

লিডার টীকাই আপনার প্রয়োজন

Lidar-Annotation ist alles, was Sie brauchen

La anotación Lidar es todo lo que necesita

L'annotation Lidar est tout ce dont vous avez besoin

लिडार एनोटेशन ही आपकी ज़रूरत है

必要なのはLiDARアノテーションだけ

Lidar 주석만 있으면 됩니다

A anotação Lidar é tudo que você precisa

Лидарная аннотация — это все, что вам нужно

Lidar Açıklaması İhtiyacınız Olan Tek Şey

Chú thích Lidar là tất cả những gì bạn cần

您所需要的只是激光雷达注释

DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon.


Our product (firewall) will continue working; clients just won't be able to access their accounts.


We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data.
Make secondary backups of critical data in the cloud.
Backup your Kubernetes master.
Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations.
Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything The fire will be put out soon. The fire will be put out soon. The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything We have backups; we'll recover everything We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't store backups in the same data center as your data. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). jobs@wallarm.com

How We Migrated to the Cloud... of Ashes (Wallarm OVH Recovery in 2021)

This tutorial shows you how to deploy and automate a Docker container using Pulumi in Google Cloud Platform's Cloud Run with minimum permissions. Here's the GitHub Repository for reference: https://github.com/superjose/deploy-to-cloud-run-go Get the CLIs: . Create a Google Cloud Account Install Google Cloud CLI Install Pulumi CLI - We need it to build the Docker container. Install Docker Bootstrap the project: Create a directory: pulumi mkdir pulumi Run to initialize a pulumi project with Go (It can be your language of choice pulumi new go Navigate to the directory. pulumi cd pulumi Navigate to and create a new project. cloud.google.com Take note of the (Usually the project's name). project-id Generate the necessary permissions using gcloud CLI Login with the Google Auth CLI: gcloud auth login (Optional) Set the project in google cloud CLI (Can be changed anytime). This saves you from passing into every command. --project [PROJECT-ID] gcloud If your machine has multiple GCP projects, skip this step and pass the flag into every command. --project gcloud Create a service account (The account that Pulumi will connect to): gcloud iam service-accounts create pulumi-gcp --description="Pulumi GCP" Download the credentials for the service accounts and store them locally (Remember to replace with your GCP Project Id): [PROJECT-ID] gcloud iam service-accounts keys create ~/keys/gcp/pulumi-service-account-key-file.json --iam-account=pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Set Pulumi's gcp credentials config path:
(This will connect the service account with Pulumi) pulumi config set gcp:credentials ~/keys/gcp/pulumi-service-account-key-file.json Set the GCP Project by doing: pulumi config set gcp:project [PROJECT-ID] Create a file (Inside the dir) and add the required permissions in : roles.gcp.yml pulumi includedPermissions # https://cloud.google.com/iam/docs/creating-custom-roles#creating
# Yaml to define the Pulumi GCP Roles that need to be created with gcloud CLI
title: Pulumi GCP Roles
description: |
  This policy ensures that all GCP roles are created using Pulumi.
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
  - serviceusage.services.list
  - serviceusage.services.enable
  - serviceusage.services.disable
  - serviceusage.services.get
  - serviceusage.services.use
    # Permissions for GCR
  - storage.objects.create
  - storage.objects.delete # Optional: only include if you need to delete images
  - storage.objects.get
  # Permissions for Google Artifact Registry
  - artifactregistry.repositories.create
  - artifactregistry.repositories.delete
  - artifactregistry.repositories.get
  - artifactregistry.repositories.list
  - artifactregistry.repositories.update
  - artifactregistry.repositories.downloadArtifacts
  - artifactregistry.repositories.uploadArtifacts
  - artifactregistry.repositories.deleteArtifacts
  # Permissions
  # Permissions
  - run.services.create
  - run.services.get
  - run.services.list
  - run.services.update
  - run.services.delete
  - run.services.getIamPolicy
  - run.services.setIamPolicy
  - iam.serviceAccounts.actAs
# NOTE: This should be removed the first time you're creating a role.
# This etag is to update the current active role (As GCP lets you manage multiple roles)
# I'm commenting it out so I can always replace the role
# etag: BwYS74Xx5y4= Create the with the file above:
(We assume we're running this code from the directory) pulumi_admin_role pulumi gcloud iam roles create pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' In case you need to make edits, change the file and use: gcloud iam roles update pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' We're also adding the role (I haven't found a better way) (Otherwise we'd get 403 errors when refreshing and updating in Pulumi)<sup>1</sup> serviceAccountAdmin gcloud projects add-iam-policy-binding [PROJECT-ID] --role roles/iam.serviceAccountAdmin   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com (The following will add the new role to the service account) gcloud projects add-iam-policy-binding [PROJECT-ID] --role projects/[PROJECT-ID]/roles/pulumi_admin_role   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Our in the directory:
(Check the code for comments!) main.go pulumi We enable the required services (Artifact Registry, and Cloud Run). Artifact Registry is used to host the docker container image. Cloud Runner will launch the Docker image from Artifact Registry. We build the docker image locally (We specify the platform in case you're using an ARM chip like M1, M2, Snapdragon SQ, X Elite, etc.) We create a chain of "DependsOn" to notify Pulumi:
5.1 Services need to be enabled first
5.2 We create the Artifact Repository
5.3 We build the docker image and push it to Artifact Registry.
5.4 We pull the Docker Image from Artifact Registry and run it.
5.5 We add IAM permissions so it can be accessed from anywhere. package main

import (
	"errors"
	"log"
	"os"

	"github.com/joho/godotenv"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/artifactregistry"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/cloudrun"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/projects"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

// This is the name you created in Google Cloud Platform (GCP).
const gcpProjectId = "deploy-to-cloud-run-go"

// The Docker Image Name
const dockerImageName = "my-app-docker"
const artifactRegistryServiceName = "artifact-registry-api"
const artifactRegistryRepoName = "my-app-artifact-repo"
const artifactRegistryRepoLocation = "us-east1"
const cloudRunAdminServiceName = "cloud-run-admin-service"
const cloudRunServiceName = "cloud-run-service"

// For more info: https://cloud.google.com/run/docs/locations
const cloudRunLocation = "us-east1"

// The tag for the Docker image
const imageTag = "latest"

// This is a url like: us-east1-docker.pkg.dev
// It is used to push the Docker image to Google Container Registry
// For more info: https://cloud.google.com/container-registry/docs/pushing-and-pulling
// The format is: <region>-docker.pkg.dev
var dockerGCPServer = cloudRunLocation + "-docker.pkg.dev"

// The full path to the Docker image
// It is used to deploy the Docker image to Google Cloud Run
// The format is: <region>-docker.pkg.dev/<project-id>/<repo-name>/<image-name>:<tag>
// For more info: https://cloud.google.com/run/docs/deploying
// Example: us-east1-docker.pkg.dev/deploy-to-cloud-run-go/my-app--artifact-repo/my-app-docker:latest
var dockerImageWithPath = dockerGCPServer + "/" + gcpProjectId + "/" + artifactRegistryRepoName + "/" + dockerImageName + ":" + imageTag

func main() {

	// Load the .env file
	err := godotenv.Load()

	if err != nil {
		log.Fatal("Error loading .env file")
	}

	pulumi.Run(func(ctx *pulumi.Context) error {

		enabledServices, serviceResultErr := enableServices(ctx)

		if serviceResultErr != nil {
			return serviceResultErr
		}

		artifactRegistryRepo, createArtifactErr := createArtifactRegistryNewRepository(ctx, &enabledServices)
		if createArtifactErr != nil {
			return createArtifactErr
		}

		dockerImage, buildAndPushErr := buildAndPushToContainerRegistry(ctx, &enabledServices, artifactRegistryRepo)

		if buildAndPushErr != nil {
			return buildAndPushErr
		}

		deployContainerErr := deployContainerToCloudRun(ctx, &enabledServices, dockerImage)

		if deployContainerErr != nil {
			return deployContainerErr
		}

		return nil
	})
}

type EnabledServices struct {
	CloudRunService         *projects.Service `pulumi:"cloudRunService"`
	ArtifactRegistryService *projects.Service `pulumi:"artifactRegistryService"`
}

func enableServices(ctx *pulumi.Context) (EnabledServices, error) {

	cloudRunService, cloudRunAdminErr := projects.NewService(ctx, cloudRunAdminServiceName, &projects.ServiceArgs{
		Service: pulumi.String("run.googleapis.com"),
		Project: pulumi.String(gcpProjectId),
	})

	if cloudRunAdminErr != nil {
		return EnabledServices{}, cloudRunAdminErr
	}

	artifactRegistryService, err := projects.NewService(ctx, artifactRegistryServiceName, &projects.ServiceArgs{
		Service: pulumi.String("artifactregistry.googleapis.com"),
	})

	if err != nil {
		return EnabledServices{}, err
	}
	return EnabledServices{
		CloudRunService:         cloudRunService,
		ArtifactRegistryService: artifactRegistryService,
	}, nil
}

func createArtifactRegistryNewRepository(ctx *pulumi.Context, enabledServices *EnabledServices) (*artifactregistry.Repository, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	dependingResources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
	}

	repo, err := artifactregistry.NewRepository(ctx, artifactRegistryRepoName, &artifactregistry.RepositoryArgs{
		Location:     pulumi.String(artifactRegistryRepoLocation),
		RepositoryId: pulumi.String(artifactRegistryRepoName),
		Format:       pulumi.String("DOCKER"),
		Description:  pulumi.String("The repository that will hold social-log Docker images."),
	}, pulumi.DependsOn(dependingResources))

	if err != nil {
		return nil, err
	}

	return repo, nil
}

func buildAndPushToContainerRegistry(ctx *pulumi.Context, enabledServices *EnabledServices, artifactRegistryRepo *artifactregistry.Repository) (*docker.Image, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	if artifactRegistryRepo == nil {
		return nil, errors.New("artifactRegistryRepo cannot be nil")
	}

	// Lookup GOOGLE_CREDENTIALS environment variable which should hold the path to the JSON key file
	jsonKeyPath, present := os.LookupEnv("GOOGLE_CREDENTIALS_FILE_PATH")
	if !present {
		return nil, errors.New("GOOGLE_CREDENTIALS_FILE_PATH environment variable is not set")
	}

	// Read the JSON key file
	jsonKey, err := os.ReadFile(jsonKeyPath)
	if err != nil {
		return nil, err
	}

	dependingSources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
		artifactRegistryRepo,
	}

	// Build and push Docker image to Google Container Registry using the JSON key
	image, err := docker.NewImage(ctx, dockerImageName, &docker.ImageArgs{
		Build: &docker.DockerBuildArgs{
			Context: pulumi.String("../"), // Adjust the context according to your project structure

			ExtraOptions: pulumi.StringArray{
				// This option is needed for devices running on ARM architecture, such as Apple M1/M2/MX CPUs
				pulumi.String("--platform=linux/amd64"),
			},
		},
		ImageName: pulumi.String(dockerImageWithPath),
		Registry: &docker.ImageRegistryArgs{
			Server:   pulumi.String(dockerGCPServer),
			Username: pulumi.String("_json_key"),     // Special username for GCP
			Password: pulumi.String(string(jsonKey)), // Provide the contents of the key file
		},
	}, pulumi.DependsOn(dependingSources))
	if err != nil {
		return nil, err
	}

	return image, nil
}

func deployContainerToCloudRun(ctx *pulumi.Context, enabledServices *EnabledServices, dockerImage *docker.Image) error {

	if enabledServices == nil || enabledServices.CloudRunService == nil {
		return errors.New("enabledServices cannot be nil")
	}

	if dockerImage == nil {
		return errors.New("dockerImage cannot be nil")
	}

	dependingSources := []pulumi.Resource{
		enabledServices.CloudRunService,
		dockerImage,
	}

	appService, err := cloudrun.NewService(ctx, cloudRunServiceName, &cloudrun.ServiceArgs{
		Location: pulumi.String(cloudRunLocation), // Choose the appropriate region for your service
		Template: &cloudrun.ServiceTemplateArgs{
			Spec: &cloudrun.ServiceTemplateSpecArgs{
				Containers: cloudrun.ServiceTemplateSpecContainerArray{
					&cloudrun.ServiceTemplateSpecContainerArgs{
						Image: dockerImage.ImageName,
						Resources: &cloudrun.ServiceTemplateSpecContainerResourcesArgs{
							Limits: pulumi.StringMap{
								"memory": pulumi.String("256Mi"), // Adjust the memory limit as needed
							},
						},
					},
				},
			},
		},
		Traffics: cloudrun.ServiceTrafficArray{
			&cloudrun.ServiceTrafficArgs{
				Percent:        pulumi.Int(100),
				LatestRevision: pulumi.Bool(true),
			},
		},
	}, pulumi.DependsOn(dependingSources))

	if err != nil {
		return err
	}

	_, iamErr := cloudrun.NewIamMember(ctx, "invoker", &cloudrun.IamMemberArgs{
		Service:  appService.Name,
		Location: appService.Location,
		Role:     pulumi.String("roles/run.invoker"),
		Member:   pulumi.String("allUsers"),
	})

	if iamErr != nil {
		return iamErr
	}

	ctx.Export("containerUrl", appService.Statuses.Index(pulumi.Int(0)).Url().ToOutput(ctx.Context()))
	return nil
} Update the in your Dockerfile:
There's a , and in which you need to export your environment variable to ENV known issue here HOME /root # Set the ENV HOME before your ENTRYPOINT.
ENV HOME=/root

# This is specific to your project. 
ENTRYPOINT ["/whatever-is-your-entrypoint"] Create a .env file in the pulumi directory: Set the * to the one you saved on full path 11) GOOGLE_CREDENTIALS_FILE_PATH="/Users/myusername/keys/gcp/pulumi-service-account-key-file.json" Run pulumi up You should be up and running! Extras The go.mod module social-log-go

go 1.21

toolchain go1.22.0

require (
	github.com/pulumi/pulumi-gcp/sdk/v7 v7.11.2
	github.com/pulumi/pulumi/sdk/v3 v3.108.1
)

require (
	dario.cat/mergo v1.0.0 // indirect
	github.com/Masterminds/semver v1.5.0 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
	github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
	github.com/agext/levenshtein v1.2.3 // indirect
	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
	github.com/atotto/clipboard v0.1.4 // indirect
	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
	github.com/blang/semver v3.5.1+incompatible // indirect
	github.com/charmbracelet/bubbles v0.16.1 // indirect
	github.com/charmbracelet/bubbletea v0.24.2 // indirect
	github.com/charmbracelet/lipgloss v0.7.1 // indirect
	github.com/cheggaaa/pb v1.0.29 // indirect
	github.com/cloudflare/circl v1.3.7 // indirect
	github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
	github.com/djherbis/times v1.5.0 // indirect
	github.com/emirpasic/gods v1.18.1 // indirect
	github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
	github.com/go-git/go-billy/v5 v5.5.0 // indirect
	github.com/go-git/go-git/v5 v5.11.0 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/glog v1.1.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-multierror v1.1.1 // indirect
	github.com/hashicorp/hcl/v2 v2.17.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
	github.com/joho/godotenv v1.5.1 // indirect
	github.com/kevinburke/ssh_config v1.2.0 // indirect
	github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
	github.com/mattn/go-isatty v0.0.19 // indirect
	github.com/mattn/go-localereader v0.0.1 // indirect
	github.com/mattn/go-runewidth v0.0.15 // indirect
	github.com/mitchellh/go-ps v1.0.0 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
	github.com/muesli/cancelreader v0.2.2 // indirect
	github.com/muesli/reflow v0.3.0 // indirect
	github.com/muesli/termenv v0.15.2 // indirect
	github.com/opentracing/basictracer-go v1.1.0 // indirect
	github.com/opentracing/opentracing-go v1.2.0 // indirect
	github.com/pgavlin/fx v0.1.6 // indirect
	github.com/pjbgf/sha1cd v0.3.0 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/pkg/term v1.1.0 // indirect
	github.com/pulumi/appdash v0.0.0-20231130102222-75f619a67231 // indirect
	github.com/pulumi/esc v0.6.2 // indirect
	github.com/pulumi/pulumi-docker/sdk/v3 v3.6.1 // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/rogpeppe/go-internal v1.11.0 // indirect
	github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
	github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/skeema/knownhosts v1.2.1 // indirect
	github.com/spf13/cobra v1.7.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/texttheater/golang-levenshtein v1.0.1 // indirect
	github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
	github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
	github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
	github.com/xanzy/ssh-agent v0.3.3 // indirect
	github.com/zclconf/go-cty v1.13.2 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	golang.org/x/crypto v0.17.0 // indirect
	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
	golang.org/x/mod v0.14.0 // indirect
	golang.org/x/net v0.19.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.15.0 // indirect
	golang.org/x/term v0.15.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	golang.org/x/tools v0.15.0 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
	google.golang.org/grpc v1.57.1 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/warnings.v0 v0.1.2 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	lukechampine.com/frand v1.4.2 // indirect
) If you include this Run , and this will fetch all the packages for you go.mod go tidy Footnotes I fought against permissions for 5 days. The predefined GCP role brought in the additional permissions needed. 1* serviceAccountAdmin

Automate and Deploy a Docker Container to Google Cloud Run from Scratch Using Pulumi and Go

Hello everyone! Reading emails in automated tests is a trick that you might need to know how to do in a lot of the projects. Here’s why and how to deal with it… Why Is This Important? Well, if the system sends emails - you might want to verify that they are being sent and the content is right, including clickable links. So these are the usual scenarios when you need this: Sign up. A lot of platforms send a confirmation email once you create an account, and you need to click a confirmation link from the email. Invoices. If your app does any payments - they usually send a cheque or invoice via email, so you might want to see if their contents are correct too. Results. Like medical test results. In my experience - some hospitals send an email with test results once they are done (or at least a link to a personal page where you can view the report). … Of course, this is not the complete list - these are just the use cases I’ve met the most in my career. Failing to verify these scenarios might lead to users not getting emails or getting the wrong information in emails. In one of the projects, it helped us find early an expired token. SendGrind What Are the Complications of Reading Emails in Tests? Reading emails is a tricky thing to do, but not an impossible one. Early email clients were using to fetch emails from the server, but there’s a small problem - it deletes the emails from the server once they are downloaded. POP3 Later, protocol came in - it allows you to simply read emails from the server. IMAP The problem is - or when you connect for the first time (like when you restart an automated test). some services ban these protocols make you confirm your identity On the other hand, some services like provide a to manipulate your emails, but there’s another problem - the documentation and its libraries are not really that user-friendly (especially for those who don’t work with email testing much). Gmail REST API This is where comes in - an library that allows you to read emails and even open them in your browser if you use browser automation tools like . gmail-getter npm Playwright Reading Gmail Emails With Gmail-Getter First thing first - you need to install the package in your project: npm install gmail-getter Create a Google Cloud Console Project, and Obtain Credentials To log into you need to obtain credentials: a , a , and a . These credentials let you get an Access token, which is required in further requests (like getting a list of emails or a single email, etc.). Gmail API, Client ID Client Secret Refresh token Steps to go: Create a project in . Google Cloud Console Create in the section ( ) and download it. OAuth credentials API & Services preferably, select app there if you need it for automated tests Desktop Enable the . Gmail API Obtain a Refresh token. To get a Refresh token - simply execute a command in a project root: npx gmail-getter get-refresh-token (or anywhere else if you've got the package installed globally) get-refresh-token You must put the file in a place where you will execute the command. credentials.json ⚠️ credentials file name is case-sensitive ⚠️ Obtain Access Token Once the configuration is done, you must get an Access token in your automated tests. I’d say that the best place for it is a global setup (a function that executes before the test run starts). So, it’ll be something like this: import {getToken} from 'gmail-getter'

export default async function globalSetup() {
  process.env['ACCESS_TOKEN'] = await getToken(
    process.env.CLIENT_ID!,
    process.env.CLIENT_SECRET!,
    process.env.REFRESH_TOKEN!
  )
} The example above uses environment variables, for which you will definitely need to install the package as well. dotenv But, if you wanna keep things simple - you can use this snippet wherever you need the token to be accessible from: const accessToken = await getToken(
    '<put-your-client-id-here>',
    '<put-your-client-secret-here>',
    '<put-your-refresh-token-here>'
  ) Get Emails Right now, uses to find the email you need: Gmail’s REST API its own query language const email = await checkInbox(accessToken!, 15000, 1500, 'from:squier7 subject:Test!') The command above returns you an email object, but you still need to get either its contents or a link. Replace to your own query following the syntax from the link above ☝️ from:squier7 subject:Test! You can parse a link using : a regex const link = parseLinkFromHtml(email!, /(https:\/\/)(\S*)(gmail-getter)([\w\/\?\=\-]*)/im) Replace with your own regular expression ☝️ /(https:\/\/)(\S*)(gmail-getter)([\w\/\?\=\-]*)/im OR get the whole HTML content of the email, and render it in your browser: import {getToken, parseHtml} from 'gmail-getter'
import {test} from '@playwright/test'

test('Render an email', async ({page}) => {
  const accessToken = await getToken(
      '<put-your-client-id-here>',
      '<put-your-client-secret-here>',
      '<put-your-refresh-token-here>'
    )

  const email = await checkInbox(accessToken!, 15000, 1500, 'from:squier7 subject:Test!') // get an email
  const html = parseHtml(email!) // parse HTML from the email
  await page.setContent(html) // render the email in the browser
}) Overall I think it’s pretty obvious now that working with emails in automated tests might be tricky! But knowing these tricks makes things much easier. You might still go through some other email services or try , but I guarantee you - it won’t be as easy as it is in this guide, as you’d need to create a helper of your own, like provided here. IMAP gmail-getter You can find the on . too. Playwright example GitHub Cypress example

How to Read Gmail Emails With Playwright

নাট্যকারের সাথে জিমেইল ইমেলগুলি কীভাবে পড়বেন

So lesen Sie Gmail-E-Mails mit Playwright

Cómo leer correos electrónicos de Gmail con Playwright

Comment lire les e-mails Gmail avec Playwright

नाटककार के साथ जीमेल ईमेल कैसे पढ़ें

Playwright で Gmail メールを読む方法

극작가와 함께 Gmail 이메일을 읽는 방법

Como ler e-mails do Gmail com o Playwright

Как читать электронные письма Gmail с помощью драматурга

Oyun Yazarı ile Gmail E-postaları Nasıl Okunur?

Cách đọc email Gmail với nhà viết kịch

如何使用 Playwright 阅读 Gmail 电子邮件

As a seasoned PHP developer, I utilise Google Cloud Spanner when I need a scalable database solution for web applications. I typically reach for Spanner when global distribution, strong consistency, and high availability are imperative. Throughout my career in PHP development, I’ve worked with several database services. Among other things, I’ve grown to be quite familiar with Google Spanner and its robust features. I appreciate Spanner’s blend of global scalability and classic SQL traits. The practicality of bridging scalability and structure makes Google Spanner perfect for projects that require a distributed database. In this article, I’ll describe what Google Spanner is, its advantages & disadvantages, in what scenarios it’s better to use vs. other cloud solutions, and how to migrate to it. What Is Google Spanner? Google Cloud Spanner is a globally distributed, horizontally scalable relational database service offered by Google Cloud Platform (GCP). Spanner is designed to provide the best of both traditional relational databases and NoSQL databases. Additional advantages to using Spanner are global scalability and strong consistency. Some key features of Google Cloud Spanner include: Spanner automatically scales your database horizontally across multiple regions and zones. This enables high availability and low-latency access to your data. Horizontal Scalability: By replicating your data across multiple regions, Spanner lets you serve users from different parts of the world with low-latency data access. Global Distribution: Unlike many other NoSQL databases, Spanner offers strong consistency guarantees. Transactions are ACID-compliant, and data is always up-to-date across all replicas. Strong Consistency: Spanner supports a subset of SQL. This makes it easy for developers familiar with relational databases to work with it. SQL Compatibility: Spanner automatically breaks down data into smaller units called "splits" and distributes them across nodes to ensure efficient data management and query execution. Automatic Sharding: Spanner automatically backs up your data and handles replication across zones and regions. This enhances data durability and availability. Automatic Backups and Replication: Changes are replicated to multiple regions before a write is acknowledged. This gives you strong consistency and minimal data loss. Synchronous Replication: Spanner is designed with built-in redundancy and fault tolerance, minimizing downtime and providing high availability. High Availability: MVCC enables concurrent transaction management. This means transactions are executed without blocking each other. Multi-Version Concurrency Control (MVCC): Google Spanner Pros and Cons Based on my long-term experience using Google Spanner, here are some pros and cons I’ve discovered: Pros: Spanner combines horizontal scalability seamlessly with SQL fundamentals. This includes robust transactional capabilities and data integrity assurance. It’s highly suitable for multi-regional or global distributed projects, including websites or applications. Spanner is a great fit for projects involving worldwide transactions and those leveraging Google's infrastructure. Spanner provides higher control over data storage, writing, and retrieval strategies. But this can be a drawback if your usage requirements change as it might require adjustments to the storage architecture. Cons: If your project doesn't require distribution and scalability, alternative solutions, e.g., MySQL, will offer better performance benefits compared to Google Spanner. Spanner provides a limited range of data types, lacks stored procedures, аnd supports only serializable isolation levels. Plus, it does not support advanced SQL statements like UPDATE with JOIN. I feel Spanner is less intuitive with standard SQL techniques for implementing business logic. Google Spanner is still quite raw. For example, a JSON column will not work correctly if it contains float values. Additionally, the gRPC server implementation for PHP is not well-optimized. I’ve seen this impact performance with tables featuring a significant number of columns. Developers should be ready for these and other unexpected difficulties. When Should You Move to Google Spanner? Google Spanner outperforms substantially when dealing with high-traffic endeavours backed by a robust core on the Google Cloud Platform. I see it as a bridge between a sophisticated database logic that craves more than NoSQL but isn't entangled in unnecessary complexity. When I’m working on a project that spans multiple regions, Google Spanner works well. From what I’ve observed, Google Spanner is ideal for the following types of projects: High-traffic projects with substantial clustered data, where the core business logic is already hosted on or intended for the Google Cloud Platform. Projects where the database's business logic is advanced enough to go beyond NoSQL capabilities, yet not overly intricate. When global distribution is a necessity for the project's requirements. Things to Consider Before Migrating to Google Spanner While I won’t go into all the details of migration, you should assess the complexity of your migration before anything else. This means gathering data about your source database. Consider query patterns, the amount of application logic that's dependent on database features (e.g., stored procedures and triggers), hardware requirements, and total cost of ownership (TCO). Here’s the complete Migration Process: Collect all queries to the current database for each endpoint. Analyze whether the data is clusterizable, i.e., whether the data could be broken down into small predefined groups. An example of an ideal case is when users only have access to their own data or their company's data. If partitioning is not possible and you need to select collections of rows using filters, Google Spanner is not a good fit. Google Spanner clusters data based on the initial bytes of the primary key. If the existing primary keys do not result in the desired data clustering, consider adding a technical column at the beginning of the primary key to achieve the desired data grouping during storage. Additionally, if auto-increment was previously used for generating IDs, you will need to use hash functions to transform them into UUIDs. If, within a single endpoint, you need to retrieve data from multiple tables connected by one-to-one or one-to-many relationships, consider using interleaved tables. Utilize LEFT JOIN queries for one-to-one relationships and ARRAY (SELECT AS STRUCT ...) constructions when dealing with one-to-many relationships. If you're using PHP and your tables contain many columns, consider storing data from columns that are not involved in WHERE clauses by converting it to one or several JSON strings. The column type should be STRING. Migrate the schema. Migrate the application. To take advantage of interleaved tables, you will likely need to add an additional abstraction layer to query the framework you are using. Test and tune your performance. Please note that tests on small volumes may not be representative. Migrate the data. Validate the migration. Configure cutover and failover mechanisms. When planning a migration, you’ll need to account for size instances and app preparation. Make sure you have sufficiently provisioned the instance for your workload by performing a load test on the instance with some synthetic representative workload. Next comes bulk migration and validation. You may want to explore the option of using Google Dataflow for data transfer, keeping in mind that Dataflow requires Python or Java, as it does not support PHP. If you are migrating from a non-Google Cloud source, refer to your current database’s documentation regarding how best to perform bulk migration. And finally, you’ll then need to warm up Spanner if you can’t activate your application to users in stages. Google Cloud provides an in-depth overview of Cloud Spanner migration . Other migration tips can be found . here here Final Thoughts About Google Spanner In my experience with Google Spanner, the fusion of global scalability and traditional SQL principles is quite useful. Its knack for juggling multi-region configurations stands out, and it helps make potential complexity more routine. Spanner is more than just a database; it's a strategic tool for projects aiming to wield scalability without sacrificing data integrity. Google Spanner leaves me convinced that it's not just about storing data; it's about skillfully orchestrating performance and reliability.

KLOUD PROJECT

kurrro.com

-- employees

Since n.d.

COMPANY RANKING

KLOUD PROJECT

EVERGREEN INDEX #5549

Representing this company? Click here to claim and customize this page!

Kloud Project WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

KLOUD PROJECT

kurrro.com

-- employees

Since n.d.

COMPANY RANKING

KLOUD PROJECT

EVERGREEN INDEX #5549

RELATED COMPANIESRANK

STORIES ABOUT KLOUD PROJECT

Representing this company? Click here to claim and customize this page!

Kloud Project WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!