Ki jan yo voye .NET Crash Dumps Slack soti nan ECS Fargate Task

Pafwa aplikasyon .NET kraze nan pwodiksyon an, epi pa gen moun konnen poukisa, paske log ak metrik yo ok. Li se byen enkyetid ak fè debugging trè mal. Nan ka sa yo, demann memwa ka senplike debugging ak redwi tan rezoud nan jou a minit. Nan atik sa a eksplike ki jan yo konfigirasyon dumps pou aplikasyon .NET deplase nan ak Lè sa a, voye yo nan ekip la devlopman nan fason ki pi pratik ak an sekirite. AWS ECS Fargate In this article, we will create AWS resources, and I will refer to AWS documentation in particular situations. IAC won’t be in our focus. Nevertheless, if you enjoy Terraform as much as I do, you can use open-source AWS modules for each article section. From my side, I can recommend you take a look at two AWS Terraform modules projects: https://github.com/cloudposse https://github.com/terraform-aws-modules Nan atik sa a, nou pral kreye resous AWS, epi mwen pral refere a dokimantasyon AWS nan situasyon espesyal yo. IAC pa pral nan konsantre nou an. Sepandan, si ou jwi Terraform menm jan mwen jwi, ou ka sèvi ak modil AWS sous louvri pou chak seksyon atik. Soti nan m ', mwen ka rekòmande ou gade nan de pwojè modil AWS Terraform: https://github.com/cloudposse https://github.com/terraform-aws-modules https://github.com/cloudposse https://github.com/terraform-aws-modules Arkitektur Solisyon Li se tan yo gade nan aritèktrik nou an. Mwen pral kòmanse pa presume ekip la dev se pa konsidere retire .NET dumps soti nan depo tankou EBS oswa EFS paske nan konplisite li yo. S3 se anpil pi senp pou devlopè yo jwenn nenpòt ki kalite dosye, epi li pafè satisfè espere nou yo. Anplis de sa, receiving proactive notifications when a new .NET dump is generated would be quite valuable. Pou egzanp, mwen pral sèvi ak Slack, men lòt opsyon gen ladan Teams, Mattermost, WhatsApp, ak sou sa. Pou voye mesaj an notifikasyon nou pral sèvi ak Lambda ak S3 triggers. Apre sa, nou pral kreye yon layer middleware bati sou EFS, DataSync, ak sidecar ECS konteiner / Lambda fonksyon. EFS pral itilize kòm yon depo dosye enteryè pou tout travay ECS nou an, Datasync pral transfere done soti nan EFS nan S3 otomatikman, ak yon container sidecar oswa Lambda pral netwaye done ansyen soti nan EFS. Tanpri revize dyagram la: AWS Lambda deletes old EFS files by the schedule configured in EventBridge. Alternatively, during ECS Task bootstrap phase, sidecar container removes outdated dumps from EFS and quits. janitor During .NET application crash, a new dump is created at EFS filesystem, and only after that the process is terminated. DataSync moves data to S3 after a new file is uploaded to EFS. When an S3 hook detects a newly uploaded file, AWS Lambda is triggered. AWS Lambda uses IAM to obtain the necessary secrets from AWS Secret Manager. AWS Lambda sends a message to Slack via API. Etap-pa-etap aplikasyon Kreye ECS Fargat travay Nan seksyon sa a nou bezwen kreye yon ECS Fargate Task lè l sèvi avèk yon egzanp aplikasyon .NET. Kondisyon Anvan nou kontinye, gen kèk etap ki nesesè yo dwe konplete: Setup ECS cluster via AWS Console, or Terraform. An official AWS guide: Creating an Amazon ECS cluster for Fargate workloads Create an IAM execution role for ECS task. To do it, you can follow . In the scope of this article I will use name for IAM execution role. this AWS guide kvendingoldo-dotnet-crash-dump-demo Kreye yon Amazon ECS Cluster pou Fargate travay chaj Gide nan AWS Minimòm sa a pou fonksyon egzekisyon pral ase: Trust policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } kòm byen ke minimòm : permissions policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*" } ] } Kreye definisyon nan travay Yon fwa tout prerequisites yo pare, li se tan yo kreye yon minimòm Fargat travay ak yon echantiyon .NET app. , epi sèvi ak definisyon an nan travay sa a dosye json: Guid ofisyèl pou AWS Guid ofisyèl pou AWS { "containerDefinitions": [ { "cpu": 0, "essential": true, "image": "mcr.microsoft.com/dotnet/samples:aspnetapp", "mountPoints": [], "name": "app", "portMappings": [ { "containerPort": 8000, "hostPort": 8000, "protocol": "tcp" } ], "systemControls": [], "volumesFrom": [] } ], "cpu": "256", "executionRoleArn": "kvendingoldo-dotnet-crash-dump-demo", "family": "kvendingoldo-dotnet-crash-dump-demo", "memory": "512", "networkMode": "awsvpc", "placementConstraints": [], "requiresCompatibilities": ["FARGATE"], "volumes": [], "tags": [] } Konfigirasyon nan .NET dumps By default, .NET apps do not generate any dumps. To configure it, we must set the following environment variables: # Forces the runtime to generate a stack dump on unhandled exceptions. COMPlus_StackDumpOnUnhandledException=1 # Enable mini dump generation on crash COMPlus_DbgEnableMiniDump=1 # Choose dump type: # 1 = Mini, # 2 = Full (use carefully) # 4 = Triage (includes stack, threads, and some heap info — a good balance for debugging). COMPlus_DbgMiniDumpType=2 # Target path for dump file (EFS is mounted here) COMPlus_DbgMiniDumpName=/dumps/dump-%e-%p-%t.dmp Variables sa yo ka ajoute dirèkteman nan Dockerfile a oswa defini kòm varyab anviwònman nan ECS Task Definition json. Nan egzanp nou an, nou pral enjecte yo nan espesifikasyon an nan ECS. Pou fè sa, nou pral ajoute yo nan Kòm yo montre anba a: containerDefinitions[0].environment "environment": [ { "name": "COMPlus_StackDumpOnUnhandledException", "value": "1" }, { "name": "COMPlus_DbgMiniDumpType", "value": "4" }, { "name": "COMPlus_DbgEnableMiniDump", "value": "1" }, { "name": "COMPlus_DbgMiniDumpName", "value": "/dumps/%t-kvendingoldo-dotnet-demo-crash.dmp" } ] Kòm ou ka wè, mwen sèvi ak plizyè posesyon nan COMPlus_DbgMiniDumpName. Dotnet otomatikman ekspande posesyon sa yo nan non dosye a depo: %e - executable name %p - process ID %t - tèminal Li de lyen sa yo pou plis enfòmasyon sou kolekte ak analize .NET crash dumps: Collect .NET Crash Dumps (Microsoft aprann) Debugging pwoblèm memwa .NET Core (sou Linux) ak Dump dotnet Kòm ou ka wè, mwen sèvi ak plizyè posesyon nan COMPlus_DbgMiniDumpName. Dotnet otomatikman ekspande posesyon sa yo nan non dosye a depo: %e - executable name %p - ID pwosesis %t - tèminal Li de lyen sa yo pou plis enfòmasyon sou kolekte ak analize .NET crash dumps: Collect .NET Crash Dumps (Microsoft aprann) Debugging pwoblèm memwa .NET Core (sou Linux) ak Dump dotnet Collect .NET Crash Dumps (Microsoft aprann) Debugging pwoblèm memwa .NET Core (sou Linux) ak Dump dotnet Kreye EFS depo ak monte li nan ECS Fargate Task Kòm mwen te di nan kòmansman an nan atik sa a, anbake yon bwat S3 nan yon travay ECS se byen difisil; olye de sa, nou pral itilize kòm magazen enteryè pou dosye .NET depo, ki ka fasilman monte nan yon seri de travay ECS. Amazon EFS (Elastic File System) Pou kreye depo EFS, swiv AWS a ofisyèl gid: Amazon ECS Tutorial: Sèvi ak Amazon EFS sistèm dosye Pou kreye magazen EFS, swiv gid ofisyèl la nan AWS: Amazon ECS Tutorial: Sèvi ak Amazon EFS sistèm dosye Amazon ECS Tutorial: Sèvi ak Amazon EFS sistèm dosye Pa gen anyen espesyal yo ajoute nan dokiman ofisyèl la. jis asire w ke: EFS ak ECS Cluster yo nan menm VPC . Allow inbound access to NFS ports in the EFS security group to do this. EFS can be accessed by ECS tasks over NFS (port 2049/tcp) Pou monte EFS sistèm dosye nan travay la ECS nou bezwen bay otorizasyon ki nesesè nan Role IAM (atake atansyon nan posesyon yo): kvendingoldo-dotnet-crash-dump-demo { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowEFSAccess", "Effect": "Allow", "Action": [ "elasticfilesystem:ClientMount", "elasticfilesystem:ClientWrite", "elasticfilesystem:ClientRootAccess" ], "Resource": "arn:aws:elasticfilesystem: : :file-system/ " } ] } Kòm yon etap final, definye volim EFS yo ak mount pwen nan definisyon travay ECS ou a (change fileSystemId ak id sistèm dosye reyèl ou apre bootstrapping): fs-xxxxxx "volumes": [ { "name": "dotnet-dumps", "efsVolumeConfiguration": { "fileSystemId": "fs-xxxxxx", "rootDirectory": "/" } } ] "mountPoints": [ { "containerPath": "/dumps", "readOnly": false, "sourceVolume": "dotnet-dumps" } ] Konfigirasyon nan AWS DataSync pou transfere dosye EFS nan S3 Sèvis DataSync se yon zouti estanda AWS pou transfè done ant diferan kalite depo. Nan ka nou an, li pral ede nou move.NET dumps soti nan EFS nan S3. Pou reyalize objektif nou an, nou bezwen: Create an S3 bucket to store our.NET dumps. Further in this article I’ll use S3 bucket name kvendingoldo-dotnet-demo-crash Use to create a bucket. this official doc Create DataSync Use to create DataSync. this official doc Some service parameters I'll be using: Source: EFS Destination: S3 bucket (e.g., ) s3://kvendingoldo-dotnet-demo-crash/ Include path filters like /dumps/* Schedule sync every minute Ofisyèl dok Ofisyèl dok Create slack alerts based on AWS Lambda Kòm te di pi bonè, alèt sou new.NET dumps yo trè itil pou ekip la devlopè. Soti nan aspè nan estrikti, alèt ka bati nan yon diferan fason: Yon fonksyon lambda senp ki voye mesaj yo nan Slack atravè API ak te triggered pa S3 evènman. Mesaj yo pibliye nan yon tèm SNS lè l sèvi avèk konfigirasyon S3 nouvèl evènman, ki Lè sa a, aktive yon fonksyon Lambda voye evènman yo nan Slack. Kòm nou pa espere yon gwo chaj, opsyon an premye se pi bon pou nou. Nan ka a, si ou vle aplike opsyon an dezyèm sèvi ak sa yo de lyen: Modil Terraform pou deplwaye SNS ak Lambda stack Yon gid pou konfigirasyon evènman S3 nan SNS Modil Terraform pou deplwaye SNS ak Lambda stack Yon gid pou konfigirasyon evènman S3 nan SNS Nou itilize Python yo voye mesaj nan Slack. Nan atik sa a nou pral voye sèlman yon lyen nan dosye a S3, men nan kèk ka li nesesè pou voye dosye a tout antye. Slack API te chanje yon tan de sa, epi voye dosye ka yon ti kras konplike. Si ou vle konnen plis, tanpri wè a "Uploading files to Slack with Python" atik la. Nou itilize Python yo voye mesaj nan Slack. Nan atik sa a nou pral voye sèlman yon lyen nan dosye a S3, men nan kèk ka li nesesè pou voye dosye a tout antye. Slack API te chanje yon tan de sa, epi voye dosye ka yon ti kras konplike. Si ou vle konnen plis, tanpri wè " » atik. Upload dosye yo nan Slack ak Python Upload dosye yo nan Slack ak Python Ok, nou pral bati alèt etap pa etap: 1. Create Slack secret Sèvi ak AWS Secret Manager Secret ak yon sèl zòn: Klav sa a ta dwe gen yon lyen nan webhook ou Slack (pou aprann plis sou Slack webhook tcheke ). kvendingoldo-dotnet-crash-dump-demo slack_webhook_url Guid ofisyèl Guid ofisyèl 2. Configure AWS Lambda Nou pa pral ale nan detay sou kreyasyon an nan AWS Lambda, men nou pral soulye kèk pwen kle. Pou plis enfòmasyon fondamantal sou enstalasyon an nan AWS Lambda, wè . Guid ofisyèl Guid ofisyèl 2.1. Asire w ke wòl la Lambda IAM gen lisans pou li soti nan S3: { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::kvendingoldo-dotnet-demo-crash/*" } 2.2: Pou jwenn yon done soti nan AWS Secret manadjè, nou bezwen spesifye varyab anviwònman nan AWS Lambda konfigirasyon: SECRET_NAME=kvendingoldo-dotnet-demo-crash 2.3: Upload kòd Python nan Lambda import json import urllib3 import os import boto3 def get_secret(secret_name): client = boto3.client("secretsmanager") try: response = client.get_secret_value(SecretId=secret_name) if "SecretString" in response: secret = response["SecretString"] try: return json.loads(secret) except json.JSONDecodeError: return secret else: return response["SecretBinary"] except Exception as e: print(f"Error retrieving secret: {e}") return None def lambda_handler(event, context): print("Event received:", json.dumps(event)) secret_name = os.environ.get('SECRET_NAME', '') if secret_name == "": return { 'statusCode': 500, 'body': json.dumps("SECRET_NAME env variable is empty") } secret = get_secret(secret_name) slack_webhook_url = secret["slack_webhook_url"] for record in event['Records']: bucket_name = record['s3']['bucket']['name'] file_name = record['s3']['object']['key'] region = record['awsRegion'] if ".aws" in file_name: print(f"Skipping internal file: {file_name}") continue message = ( f":package: *New .NET dump is uploaded!*\n\n" f":cloud: Bucket: `{bucket_name}`\n" f":floppy_disk: File: `{file_name}`\n" f":link: Link: https://{bucket_name}.s3.{region}.amazonaws.com/{file_name}" ) http = urllib3.PoolManager() slack_resp = http.request( "POST", slack_webhook_url, body=json.dumps({ "text": message }), headers={ "Content-Type": "application/json" } ) if slack_resp.status != 200: raise Exception( f"Slack webhook request failed with status {slack_resp.status}: {slack_resp.data.decode('utf-8')}") return { "statusCode": 200, "body": json.dumps("Message has been sent successfully!") } 2.4: Konfigire S3 Evènman Notifikasyon pou bwat S3 ou. Pou fè sa, ale nan ak seleksyon » ". Konfigire evènman an lè l sèvi avèk opsyon sa yo: bucket -> properties -> Event notifications Kreye nouvèl evènman Isit la nan evènman an: kvendingoldo-dotnet-demo-crash Prefiks nan: dumps / Tip nan evènman: s3:ObjèTe kreye:* Target: Konfigirasyon EFS depo netwaye Konplètman, .NET dumps livrezon lanmè se pare, men ki sa sou demode a fin vye granmoun? EFS pa pèmèt nou retire dosye fin vye granmoun lè l sèvi avèk règleman sik la lavi; nou ka sèlman transfere yo nan Nonfrekans Access kalite depo ki pa ase si nou pa vle peye pou espas inutil. Pou solisyon nan pwoblèm sa a, gen de opsyon: Kreye ECS sidecar container ki pral netwaye dosye EFS fin vye granmoun nan faz la initialization Create Lambda or ECS task that will mount EFS, and clean-up old files by CRON. Tanpri tcheke nan tou de. Opsyon 1: AWS Lambda Li se solisyon a pi bon paske li pa afekte pa sik la lavi nan travay ECS ak lòt faktè. Pou aplike estrateji sa a, ou bezwen kreye yon fonksyon Lambda ak yon depo EFS monte (learn more about mounting a file system to Lambda from ) and the following Python code: Doktè ofisyèl Doktè ofisyèl import os import time import json def lambda_handler(event, context): # Note: you can only mount the filesystem to the /mnt/ directory. directory = '/mnt/dumps' # File pattern to match pattern = 'crash.dmp' # Time in minutes (by default 1d) minutes_old = 1440 # Convert minutes to seconds age_seconds = minutes_old * 60 # Current time now = time.time() for root, dirs, files in os.walk(directory): for file in files: if pattern in file: file_path = os.path.join(root, file) file_mtime = os.path.getmtime(file_path) if now - file_mtime > age_seconds: print(f"Found a file that older than {minutes_old} minutes: {file_path}") try: os.remove(file_path) except Exception as e: print(f"Failed to delete {file_path}: {e}") return { "statusCode": 200, "body": json.dumps("EFS clean-up completed successfully!") } Kòm ou ka wè, sa a se yon kòd senp ki retire dosye yo soti nan depo monte ki se ansyen pase yon jou. Lè Lambda ou a se pare, nou tou bezwen konfigirasyon trigger la CRON yo kouri fonksyon an periòdman. Li ka kreye lè l sèvi avèk__ nan. cloudwatch event rule Se konsa, apre tout etap sa yo, depo EFS ou a pral netwaye otomatikman pa orè CRON ou a. Option 2: ECS sidecar container. To implement this option we have to to add new container to our task definition: { "essential": false, "name": "janitor", "image": "public.ecr.aws/amazonlinux/amazonlinux:2", "command": [ "bash", "-lc", "find /dumps -name '*crash.dmp*' -type f -mmin +10080 -print -delete" ], "mountPoints": [ { "containerPath": "/dumps", "readOnly": false, "sourceVolume": "dotnet-dumps" } ], "linuxParameters": { "initProcessEnabled": true } } The logic behind this task: Inicialize yon nouvo travay ECS ak de containers: app ak janitor Netwaye dosye EFS oblije nan janitor konteynè a ak eksite. Pa gen pwoblèm, travay la pa pral interrupted oswa sispann akòz opsyon ECS "essential": false. Kòm ou ka wè, teknik sa a se byen senp ak depann sou jwenn komando a, ki ou ka Customize. Nan egzanp, li retire dosye ki pi gran pase 10080 minit (7 jou). Natirèlman, estrateji sa a se mwens dezi pase premye a lè fè fas ak travay ECS long-lived, men li ka pi pratik pou travay ECS kout-lived oswa prototip. Tan tès Nan seksyon sa a, nou pa pral fè yon plonje byen nan konstriksyon an aplikasyon .NET. Pou objektif tès, ou ka modifye Li te itilize nan kòmansman an. Pwodiksyon nan aspnetapp Pwodiksyon nan aspnetapp Yon fason ki pi senp yo fè yon kras nan .NET se Metòd sa a se souvan itilize pou simile . Environment.FailFast() Kòmanse Tanpri simile aksidan an: Ajoute Environment.FailFast("kvendingoldo-dotnet-demo-crash .NET egzanp kras"); liy nan dosye a dotnet-docker/samples/aspnetapp/aspnetapp/Program.cs. Kreye yon nouvo imaj docker, epi kreye travay la ECS ankò. ECS Task pral sispann, men premye genere yon .NET crash dump, ki pral disponib sou S3 nan yon kèk segonn. At the final phase, you'll receive a message on your Slack like this: 📦 New .NET dump is uploaded! ☁️ Bucket: kvendingoldo-dotnet-demo-crash 💾 File: 1739104252-kvendingoldo-dotnet-demo-crash.dmp 🔗 Link: https://kvendingoldo-dotnet-demo-crash.s3.us-east-2.amazonaws.com/1739104252-kvendingoldo-dotnet-demo-crash.dmp Potansyèl amelyorasyon Anvan anrjistreman atik la, mwen ta renmen bay kèk kòmantè sou chanjman posib: Li pral yon bon ide yo kreye pre-signed URLs pou objè S3 Konfigirasyon règleman sik lavi pou S3 bwat yo retire bwat fin vye granmoun otomatikman soti nan bwat la Sèvi ak SNS pou voye alèt sou nouvo S3 objè nan plizyè destinasyon Konklisyon Nan anviwònman pwodiksyon, vit vizibilite nan defè se enpòtan. Otomatik livrezon depo redwi MTTR (Mean Time To Resolution) ak amelyore repons aksidan. Kòm ou ka wè, implemantasyon pwosesis sa a se pa tèlman difisil ke ou ta ka espere. Se konsa, nou te itilize anpil sèvis AWS fè travay sa yo, men lè nou gade pi byen, yo tout enpòtan. Mwen espere ke atik sa a te ede ou bati yon chaj livrezon depo pèsonèl ak te fè ekip devlopman ou pi kontan. Senpleman santi ou lib yo modifye metòd la sijere, epi tanpri kontakte m 'nan nenpòt ki lè si ou gen nenpòt kesyon. Happy coding!