Best match

Most recent

As technology advances and morphs, startups continue to emerge, looking to differentiate themselves from the rest against incredible odds — some 90% of all startups fail, according to Forbes.

Companies to Watch in 2021

SD Times

And so we just thought that TF was a little bit too close to Terraform." OpenTofu arrives with the support of numerous companies – including Harness, Gruntwork, Spacelift, env0, Scalr, Digger, Terrateam, Massdriver, and Terramate, among others.

Terraform fork OpenTF renamed and relocated as OpenTofu

theregister

Get the most recent info and news about env0 on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

env0 is the best way to deploy and manage your IaC, including Terraform, Kubernetes, and others. The env0 platform enables users and teams to collaborate and provide self-service cloud deployments, all with advanced policies to meet governance and compliance. With env0, every engineer, from development, operations, and DevOps can deploy infrastructure simply, quickly and safely. Maximum productivity, minimum friction.

env0

In the last several years, Kubernetes has become the “go to” standard for managing and orchestrating containerized workloads. Thanks to it’s vendor agnostic nature, you can easily run Kubernetes almost anywhere, and in fact, all the major cloud vendors offer a managed Kubernetes service (AWS EKS, Google GKE, and Azure AKS). With Kubernetes, one of the key advantages is the ease of managing multiple environments and workloads in a single cluster, by separating the cluster into logical areas using namespaces. This post will dive into how we can manage this by using Terraform to both manage the cluster provisioning as well as manage the namespaces. Multiple Environments in One Cluster When using Kubernetes for a team, you usually want to have an isolated environment for each developer, , or pull request. There are a few ways to achieve that using Kubernetes: one way is to create a full blown cluster for each division, but the way we’re focusing on is using . branch Kubernetes namespaces feature While the namespaces feature is quite powerful it’s not without its complications, especially with a large team of developers: How many namespaces do I have on my cluster? Can I remove them? Does anyone use them? Can I schedule them to automatically shut down during night time and weekend? Can I have policies on who can run what and where? Fortunately, with a good management platform, a lot of this can be alleviated. Why Manage Kubernetes using Terraform? As a cloud-native technology that is deployed at wide scale, it’s quite common to manage the deployment of the infrastructure of Kubernetes clusters using Terraform. And where you have many clusters to manage (Dev, Staging, Production, etc.), Terraform allows you to maintain a consistent configuration for the cluster and underlying infrastructure while creating as many clusters as you like with the same configuration in a reliable and easy way. Terraform’s multi-cloud approach lets you use any cloud provider you wish, including using the native managed services that I’ve mentioned above. Similarly, when managing namespaces there’s a few ways, but using Terraform is probably the best overall. The main advantages of using Terraform are: Use the same to provision the Kubernetes infrastructure and to deploy applications into it.Drift detection — `terraform plan` will always present you the difference between reality at a given time and the config you intend to apply.Full lifecycle management — Terraform doesn’t just initially create resources, but offers a single command for creation, update, and deletion of tracked resources without needing to inspect the API to identify those resources.Synchronous feedback — While asynchronous behaviour is often useful, sometimes it’s counter-productive as the job of identifying operation results (failures or details of created resource) is left to the user. e.g. you don’t have IP/hostname of load balancer until it has finished provisioning, hence you can’t create any DNS record pointing to it. — Terraform understands relationships between resources which may help in scheduling — e.g. if a Persistent Volume Claim claims space from a particular Persistent Volume, Terraform won’t even attempt to create the PVC if creation of the PV has failed. configuration language graph of relationships Putting It Into Action I’ve taken the Terraform code from a simple , popular for its ease, to provision a new EKS cluster in my AWS account. AWS EKS Terraform example To run the actual Terraform, I’m using which allows me to quickly deploy and manage my environments based on Terraform templates. If you’re just getting started with env0, you should first get your organization set up and create a new template (check out the for help there). env0 getting started documentation Now we will run the template which will provision the EKS cluster with env0 and get the outputs: Now that our EKS cluster is set up, we can create a new template for the namespaces and our deployment using the cluster name from the outputs. Set up Namespace + Deployment First we use the following code to authenticate against the Kubernetes cluster that we’ve created and create a new deployment with an ngnix: { { = labels = { = }
    namespace = random_string.namespace_name.result
  }

  spec { = selector { = { = }
    }
    template { { = { = }
      }
      spec { { = name  = port { = }

          resources { { = memory = }
            requests { = memory = }
          }
        }
      }
    }
  }
} resource "kubernetes_deployment" "nginx" metadata name "scalable-nginx-example" App "ScalableNginxExample" replicas 1 match_labels App "ScalableNginxExample" metadata labels App "ScalableNginxExample" container image "nginx:1.7.8" "example" container_port 80 limits cpu "0.5" "512Mi" cpu "250m" "50Mi" provider {
  version = region  = }

data {
  name = }

data {
  name = }

provider {
  host                   = data cluster_ca_certificate = base64decode(data . .data)
  token                  = data load_config_file       = false
} "aws" "~> 2.0" var .region "aws_eks_cluster" "eks_cluster" var .cluster_name "aws_eks_cluster_auth" "eks_cluster_auth" var .cluster_name "kubernetes" .aws_eks_cluster .eks_cluster .endpoint .aws_eks_cluster .eks_cluster .certificate_authority 0 .aws_eks_cluster_auth .eks_cluster_auth .token So now let’s make (and run) a new template using that Terraform code which will create the namespace and run the deployment: Running to get the namespace and the deployment and we can see the resources we’ve created inside the cluster: kubectl You can see that we use the and the data resource to get the authentication data, and to be able to create the namespace and the deployment in the cluster. aws_eks_cluster aws_eks_cluster_auth There is another way to achieve this and import the state file of the Kubernetes cluster you created, but for simplicity we took a simpler approach. You can read more about it . here Take advantage of More Features So now that we have a template we can create as many namespaces and deployments as we want, as well as leverage some of the more advanced features of our management platform. Scheduling First, I want to make sure that those resources aren’t running over night and weekends so let’s set up scheduling for those namespaces: Now, env0 will automatically destroy those namespaces on each workday at midnight and will deploy them at 9am when I start my workday. You can even do that on the entire Kubernetes cluster to save even more money. Dev Team Environments Next, now that we have created an easy way for self service provisioning of environments, we want to set up appropriate permissions for everyone on the team to deploy their own environments. In most cases, we want the EKS cluster to be managed by the DevOps team, while the rest of the dev team should be able to provision their own individual environments. You can achieve this by managing different projects with different permissions for each user. So now I will give our DevOps team the ability to provision the Kubernetes cluster by assigning them a “Deployer” role, but the developers will be assigned as a Planners so they can request changes on the cluster but will need the approval of the DevOps to make the changes. For the ephemeral namespaces Project the developers can have a “Deployer” role so they will be able to create environments whenever they like, but everything will be cost effective and no resource will be left behind. Summary As this is a simple example of provisioning an EKS cluster with relevant deployment, we actually created a balance between a self service way to get resources and governance and a cost efficient way to manage those cloud resources without losing control. In addition you can achieve this with any other managed kubernetes service like AKS and GKE, or even just a self managed cluster. You can also leverage env0’s API and CLI to fully automate a per pull request environment, so each PR will get its own namespace and deployment, and once the PR will be merged to master, it will automatically tear down those resources. You can read more about this concept in a great by Avner Sorek. blog post There is a lot more you can achieve with Kubernetes and Terraform using env0, so go ahead and try it yourself. Previously published at https://www.env0.com/blog/kubernetes-environments-using-namespaces-and-terraform

How Namespace and Terraform Can Manage Kubernetes Environments

In my previous , I went through how to create a maintenance mode page for your application, and how to implement it using Terraform and github pages. But the website is just one part of an Application, and often there’s also a public (or private) API that also needs to have a maintenance mode. Let’s see how we can do that using Terraform on API Gateway. blog post Architecture As previously mentioned, we host all of our infrastructure on AWS, with a clear separation between the Application and the API: The front end is a React application, hosted on S3 with CloudFront as a CDN. The Backend services are mostly Serverless using AWS Lambda with API Gateway that manages our public API. DNS is managed by Route 53. Implementing a Solution Beyond the requirements , we had one new one: When the API is in maintenance it should return “503 Service Unavailable” response This led us led us to the following solution: for our webpage maintenance mode with an informative message. Create a new API Gateway that will be dedicated to the maintenance mode This API will use a mock integration It will return the same response for all endpoints using proxy resource This mocked API will co-exists with our real API. We will use and change the base mapping between the real API and the mocked one. custom domain Based on that, let’s see the Terraform code in action: Mocked API Gateway Looking at the code you are able to see that we are using ANY method together with “{proxy+}” to achieve the ability to cover all resources with a simple configuration, and using the response template to generate the “503 Service Unavailable” response with a clear message to the end user. resource {
  name        = description = } {
  rest_api_id = aws_api_gateway_rest_api.maintenance_mode_api.id
  parent_id   = aws_api_gateway_rest_api.maintenance_mode_api.root_resource_id
  path_part   = } {
  rest_api_id   = aws_api_gateway_rest_api.maintenance_mode_api.id
  resource_id   = aws_api_gateway_resource.maintenance_mode_resource.id
  http_method   = authorization = } {
  rest_api_id          = aws_api_gateway_rest_api.maintenance_mode_api.id
  resource_id          = aws_api_gateway_resource.maintenance_mode_resource.id
  http_method          = aws_api_gateway_method.maintenance_mode_method.http_method = timeout_milliseconds = 29000
  passthrough_behavior = request_templates = { = <<EOF
    { : 503}
EOF
  }
} {
  rest_api_id = aws_api_gateway_rest_api.maintenance_mode_api.id
  resource_id = aws_api_gateway_resource.maintenance_mode_resource.id
  http_method = aws_api_gateway_method.maintenance_mode_method.http_method
  status_code = } {
  rest_api_id = aws_api_gateway_rest_api.maintenance_mode_api.id
  resource_id = aws_api_gateway_resource.maintenance_mode_resource.id
  http_method = aws_api_gateway_method.maintenance_mode_method.http_method
  status_code = aws_api_gateway_method_response.maintenance_mode_response_503.status_code

  # Transforms the backend JSON response XML
  response_templates = { = <<EOF EOF
  }
} {
  depends_on  = [aws_api_gateway_integration.maintenance_mode_integration]
  rest_api_id = aws_api_gateway_rest_api.maintenance_mode_api.id
  stage_name  = } "aws_api_gateway_rest_api" "maintenance_mode_api" "maintenance-mode-API" "This is my Maintenance mode API" resource "aws_api_gateway_resource" "maintenance_mode_resource" "{proxy+}" resource "aws_api_gateway_method" "maintenance_mode_method" "ANY" "NONE" resource "aws_api_gateway_integration" "maintenance_mode_integration" type "MOCK" "WHEN_NO_MATCH" "application/json" "statusCode" resource "aws_api_gateway_method_response" "maintenance_mode_response_503" "503" resource "aws_api_gateway_integration_response" "maintenance_mode_integration_response" to "application/json" #set($inputRoot = $input.path('$')) { "message" : "Sorry for the inconvenience but we are performing some maintenance at the moment. We will be back online shortly!" } resource "aws_api_gateway_deployment" "maintenance_mode_deployment" "prod" Now using a custom domain we can control the maintenance mode, so the base path mapping will either point to the real API and in case of a maintenance mode it will point to the mocked API. resource {
  certificate_arn = var.certificate_arn
  domain_name     = var.domain_name
} {
  api_id      = var.is_maintenance_mode ? aws_api_gateway_rest_api.maintenance_mode_api.id : aws_api_gateway_rest_api.api.id
  stage_name  = var.is_maintenance_mode ? aws_api_gateway_deployment.maintenance_mode_deployment.stage_name : aws_api_gateway_deployment.deployment.stage_name
  domain_name = aws_api_gateway_domain_name.domain_name.domain_name
  base_path   = } "aws_api_gateway_domain_name" "domain_name" resource "aws_api_gateway_base_path_mapping" "base_mapping" "/" Also in AWS, we want to ensure that Route53 is pointing in the custom domain we’ve created. As opposed to the application maintenance mode, here our custom domain switches on and off our maintenance mode and not our DNS record. resource {
  zone_id = name    = type    = alias {
    name                   = aws_api_gateway_domain_name zone_id                = aws_api_gateway_domain_name evaluate_target_health = false
  }
} "aws_route53_record" "api_dns_record" var .route53_zone_id var .route53_suffix_domain_name "A" .domain_name .regional_domain_name .domain_name .regional_zone_id ⚠️ Pay attention that this Terraform code does not create the Route53 hosted zone, nor the SSL certificates — you need to complete those as appropriate for your own setup ⚠️ Deployment Now that our system is all configured, all I have to do is change the Terraform variable of the maintenance mode to be true/false and deploy the environment (in our case via the env0 UI). The complete template source code can be found in this which includes all the Terraform code. We hope you find this useful, or get other ideas for more ways to use Terraform for your deployment workflows. github repo About env0 env0 lets your team manage their own environments in AWS, Azure and Google, governed by your policies and with complete visibility & cost management. You can learn more about env0 and you can also . here try it out yourself Feel free to drop us your thoughts below! Previously published at https://www.env0.com/blog/creating-a-maintenance-mode-for-your-api-gateway-with-terraform

Building Maintenance Mode for your API Gateway Using Terraform

As you may have seen, we just launched our of env0 last month. As part of the run-up to the launch, our dev team had to go through and make sure everything about our infrastructure was ready for ongoing public use: one element being creating a for both our Application and our . public beta maintenance mode public API Why do you need a maintenance mode for something like env0, which is built to be highly available? Well whether it’s human error (as in the case of ), a DDOS attack (like with ), or just a major upgrade (like ), even the most highly available applications from the most experienced providers sometimes need to be able to be taken offline for a short period of time. And when you have to do that, you want to make sure that your users understand what is happening and still have a good experience. AWS S3 github Zapier Here’s how our team put together our maintenance mode using Terraform, AWS, and Github Pages (including all the code at the end!) Architecture We host all of our infrastructure on AWS, with a clear separation between Application and API: React front end application is hosted on S3 with CloudFront as a CDN. The Backend services are mostly Serverless using AWS Lambda with API Gateway that manages our public API. DNS is managed by Route 53. Since we have a separation between the frontend application and our API, we need to have a maintenance mode for each of them, especially as we also have a public API that is used by our customers for integration in their CI/CD pipelines and other tools. Don’t forget this if you have multiple ways in which people access your services! Requirements As with any project, the first step is to lay out requirements and constraints. In this case, we came up with the following list of what the solution should do: Be implemented with Infrastructure as Code (IaC) as all of our stack is based on IaC Switch back and forth as fast as possible between normal and maintenance mode Switch automatically between normal and maintenance mode Be hosted by separate providers, where possible, from the current infrastructure in order to provide maximum redundancy Have an internal backdoor to allow our team to deploy a fix and test it before switching back Not override the current configuration with new deployments Communicate to an active user that we are in maintenance mode Implementing a Solution After investigating, we came to a few conclusions: We wanted to have a simple place to store the maintenance mode html, so we went with github pages. The switch itself will be done through the DNS, pointing to github pages when we are in maintenance mode. We will create a new DNS record and a Cloudfront distribution that will be our backdoor, and it will be always up and running. We will use our current push mechanism to notify existing users that are in the app that we are currently in maintenance mode. Based on that, we implemented the following at each piece of the system — each system contains a description of what we’re doing, along with the link to a gist with the actual Terraform code. Cloudfront Looking at the Cloudfront distribution code, you can see that we are creating 2 distributions, one is for the actual application and the other one is for a backdoor: resource { = }

resource { = = = = = origin { = = aws_s3_bucket.website_bucket.bucket_domain_name
    s3_origin_config { = aws_cloudfront_origin_access_identity.origin_access_identity.cloudfront_access_identity_path
    }
  }

  default_cache_behavior { = [ , ] = [ , ]

    forwarded_values { = = []
      cookies { = }
    } = = // day = // year = = = }

  restrictions {
    geo_restriction { = }
  }

  viewer_certificate { = count. == ? var.acm_certificate_arn : var.acm_certificate_arn_backdoor = = } = list(count. == ? var.dns_name : var.backdoor_dns_name)
} "aws_cloudfront_origin_access_identity" "origin_access_identity" comment " access identity" ${aws_s3_bucket.website_bucket.bucket} "aws_cloudfront_distribution" "website_cdn" count 2 # one for the actual cloudfront and one is for the backdoor enabled true price_class "PriceClass_200" http_version "http2" default_root_object "index.html" origin_id "origin-bucket- " ${aws_s3_bucket.website_bucket.id} domain_name origin_access_identity allowed_methods "GET" "HEAD" cached_methods "GET" "HEAD" query_string false headers forward "none" min_ttl 0 default_ttl 86400 1 max_ttl 31536000 1 target_origin_id "origin-bucket- " ${aws_s3_bucket.website_bucket.id} viewer_protocol_policy "redirect-to-https" compress true restriction_type "none" acm_certificate_arn index 0 ssl_support_method "sni-only" minimum_protocol_version "TLSv1.2_2018" aliases index 0 Route53 Also in AWS, we want to ensure that Route53 is pointing in the right direction. In case we are not in maintenance mode it should point to the Cloudfront distribution, and when in maintenance mode it should point our CNAME to the github page. In either case, we should have the backdoor pointing to the Cloudfront distribution. Additionally, we also set the TTL to be 60 seconds so it won’t take too long to move back and forth between maintenance mode and regular operation: resource {
  zone_id = var.aws_route53_zone_id
  name    = var.dns_name = records = list(var.maintenance_mode_enabled ? : aws_cloudfront_distribution.website_cdn[0].domain_name)
  ttl     = 60
} {
  zone_id = var.aws_route53_zone_id
  name    = var.backdoor_dns_name = alias {
    name                   = aws_cloudfront_distribution.website_cdn[1].domain_name
    zone_id                = aws_cloudfront_distribution.website_cdn[1].hosted_zone_id
    evaluate_target_health = }
} "aws_route53_record" "dns_record" type "CNAME" " .github.io" ${var.github_organization} resource "aws_route53_record" "dns_backdoor_record" type "A" false ⚠️ Pay attention that this Terraform code does not create the Route53 hosted zone, nor the SSL certificates — you need to complete those as appropriate for your own setup ⚠️ Git Repo Next, we need a git repo containing the html files for the new maintenance mode site. So we’ve created a simple Terraform code that will create the repo as well as add all existing files in the folder to the repo, which in our case is the maintenance mode html file: “maintenance_mode_website” resource { = var.github_repo_name = = = = }

resource { = github_repository.maintenance-mode-repo.name = var.github_username = = [github_repository.maintenance-mode-repo]
}

resource { = github_repository.maintenance-mode-repo.name = {
    for key, value fileset(path.root, ): => value
  }
  // There is a limitation here, Terraform file function supports UTF- files,
  // so any images other non text files can't be included the folder. = trimprefix(each.value, ) = file(each.value) = [github_repository_collaborator.users_repos]
} "github_repository" "maintenance-mode-repo" name description "Maintenance mode repo for " ${var.github_repo_name} private true auto_init true default_branch "master" "github_repository_collaborator" "users_repos" repository username permission "admin" depends_on "github_repository_file" "maintenance-mode-files" repository for_each in "maintenance_mode_website/*.*" value 8 or in file "maintenance_mode_website/" content depends_on Github Pages The last part is the trickiest to implement in Terraform, because Github pages configuration is not actually part of the , which means that I can’t really configure it with the Terraform code. However because github offers an API to , it can be done programmatically via the API. In our case, we can use the env0 to trigger those API calls once the deploy is finished: github Terraform provider configure github pages custom flows feature version: 1 deploy: steps: terraformOutput: after: - scripts/set_maintanance_mode.sh ${TF_VAR_github_organization} "$(terraform output github_repo_name)" "$(terraform output website_endpoint)" ${TF_VAR_github_username} ${TF_VAR_github_token} set -e

REPO_ORGANIZATION=$1
REPO_NAME=$2
CNAME=$3
GITHUB_USERNAME=$4
GITHUB_TOKEN=$5

if [[ ${REPO_NAME} != "" &&  CNAME != "" ]]; then

    echo "Going to set environment to maintenance mode, repo name: ${REPO_NAME}, CNAME: ${CNAME}"

    GIT_AUTH="${GITHUB_USERNAME}:${GITHUB_TOKEN}"

    curl --user "${GIT_AUTH}" \
     --request POST \
     --header "Content-Type: application/json" \
     --header "Accept: application/vnd.github.switcheroo-preview+json" \
     --url "https://api.github.com/repos/${REPO_ORGANIZATION}/${REPO_NAME}/pages" \
     --data "{ \"source\": { \"branch\": \"master\" } }"

     curl --user "${GIT_AUTH}" \
     --request PUT \
     --header "Content-Type: application/json" \
     --url "https://api.github.com/repos/${REPO_ORGANIZATION}/${REPO_NAME}/pages" \
     --data "{ \"cname\": \"${CNAME}\", \"source\": \"master\" }"
else
    echo "No need to set environment to maintenance mode"
fi # !/usr/bin/env bash Deployment Now that our system is all configured, all I have to do is change the Terraform variable of the maintenance mode to be true/false and deploy the environment (in our case via the env0 UI). Conclusion The complete template source code can be found in this which includes all the Terraform code, scripts, our env0.yml and the maintenance page html file. We hope you find this useful, or get other ideas for more ways to use Terraform for your deployment workflows. Our next blog post will give you a sneak preview on how we are creating a maintenance mode on our API using Terraform. github repo About env0 env0 lets your team manage their own environments in AWS, Azure and Google, governed by your policies and with complete visibility & cost management. You can learn more about env0 and you can also . Feel free to drop us your thoughts below! here try it out yourself

Building a “Maintenance Mode” with Terraform and Github Pages

The OpenTelemetry Collector sits at the center of the OpenTelemetry architecture but is unrelated to the W3C Trace Context. In my , I use Jaeger instead of the Collector. Yet, it's ubiquitous, as in every OpenTelemetry-related post. I wanted to explore it further. tracing demo In this post, I explore the different aspects of the Collector: The data kind: logs, metrics, and traces Push and pull models Operations: reads, transformations, and writes First steps A long time ago, as we know it didn't exist; what we had instead was . Back then, monitoring was a bunch of people looking at screens displaying dashboards. Dashboards themselves consisted of metrics and only system metrics: mainly CPU, memory, and disk usage. For this reason, we will start with metrics. observability monitoring is one of the primary monitoring solutions. It works on a pull-based model: Prometheus scrapes compatible endpoints of your application(s) and stores them internally. Prometheus We will use the OTEL Collector to scrape a Prometheus-compatible endpoint and print out the result in the console. Grafana Labs offers a that generates random metrics to play with. For simplicity's sake, I'll use Docker Compose; the setup looks like the following: project version: "3"

services:
  fake-metrics:
    build: ./fake-metrics-generator                                         #1
  collector:
    image: otel/opentelemetry-collector:0.87.0                              #2
    environment:                                                            #3
      - METRICS_HOST=fake-metrics
      - METRICS_PORT=5000
    volumes:
      - ./config/collector/config.yml:/etc/otelcol/config.yaml:ro           #4 No Docker image is available for the fake metrics project; hence, we need to build it Latest version of the OTEL Collector at the time of this writing Parameterize the following configuration file Everything happens here As I mentioned above, the OTEL Collector can do a lot. Hence, configuration is everything. receivers:                                                                  #1
  prometheus:                                                               #2
    config:
      scrape_configs:                                                       #3
        - job_name: fake-metrics                                            #4
          scrape_interval: 3s
          static_configs:
            - targets: [ "${env:METRICS_HOST}:${env:METRICS_PORT}" ]

exporters:                                                                  #5
  logging:                                                                  #6
    loglevel: debug

service:
  pipelines:                                                                #7
    metrics:                                                                #8
      receivers: [ "prometheus" ]                                           #9
      exporters: [ "logging" ]                                              #9 List of receivers. A receiver reads data; it can be either push-based or pull-based. We use the pre-defined receiver prometheus Define pull jobs Job's configuration List of exporters. In contrast to receivers, an exporter writes data. The simplest exporter is to write data on the standard out Pipelines assemble receivers and exporters Define a metric-related pipeline The pipeline gets data from the previously-defined receiver and sends it to the exporter, , prints them prometheus logging i.e. Here's a sample of the result: 2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 83.090000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #1
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__embrace_world_class_systems: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__exploit_magnetic_applications: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__facilitate_wireless_architectures: Str(extranet)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(challenge)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__reinvent_revolutionary_applications: Str(support)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__strategize_strategic_initiatives: Str(internet_solution)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__target_customized_eyeballs: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__transform_turn_key_technologies: Str(framework)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__whiteboard_innovative_partnerships: Str(matrices)
2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 53.090000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #2
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__expedite_distributed_partnerships: Str(approach)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__facilitate_wireless_architectures: Str(graphical_user_interface)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(policy)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__reinvent_revolutionary_applications: Str(algorithm)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__transform_turn_key_technologies: Str(framework)
2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 16.440000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #3
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__exploit_magnetic_applications: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(graphical_user_interface)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__target_customized_eyeballs: Str(extranet) Beyond printing The above is an excellent first step, but there's more than printing to the console. We will expose the metrics to be scraped by a regular Prometheus instance; we can add a to visualize them. While it may seem pointless, bear with it, as it's only a stepstone. Grafana dashboard To achieve the above, we only change the OTEL Collector configuration: exporters:
  prometheus:                                                               #1
    endpoint: ":${env:PROMETHEUS_PORT}"                                     #2

service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      exporters: [ "prometheus" ]                                           #3 Add a exporter prometheus Expose a Prometheus-compliant endpoint Replace printing with exposing That's it. The OTEL Collector is very flexible. Note that the Collector is multi-input, multi-output. To both print data and expose them via the endpoint, we add them to the pipeline: exporters:
  prometheus:                                                               #1
    endpoint: ":${env:PROMETHEUS_PORT}"
  logging:                                                                  #2
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      exporters: [ "prometheus", "logging" ]                                #3 Expose data Print data The pipeline will both print data and expose them With the Prometheus exporter configured, we can visualize metrics in Grafana. Note that receivers and exporters specify their type every one of them must be unique. To comply with the last requirement, we can append a qualifier to distinguish between them, , and and i.e. prometheus/foo prometheus/bar. Intermediary data processing A valid question would be why the OTEL Collector is set between the source and Prometheus, as it makes the overall design more fragile. At this stage, we can leverage the true power of the OTEL Collector: data processing. So far, we have ingested raw metrics, but the source format may not be adapted to how we want to visualize data. For example, in our setup, metrics come from our fake generator, "business", and the underlying NodeJS platform, "technical." It is reflected in the metrics' name. We could add a dedicated source label and remove the unnecessary prefix to filter more efficiently. You declare data processors in the section of the configuration file. The collector executes them in the order they are declared. Let's implement the above transformation. processors The first step toward our goal is to understand that the collector has two flavours: a "bare" one and a contrib one that builds upon it. Processors included in the former are limited, both in number and in capabilities; hence, we need to switch the contrib version. collector:
  image: otel/opentelemetry-collector-contrib:0.87.0                        #1
  environment:
    - METRICS_HOST=fake-metrics
    - METRICS_PORT=5000
    - PROMETHEUS_PORT=8889
  volumes:
    - ./config/collector/config.yml:/etc/otelcol-contrib/config.yaml:ro     #2 Use the flavour contrib For added fun, the configuration file is on another path At this point, we can add the processor itself: processors:
  metricstransform:                                                         #1
    transforms:                                                             #2
      - include: ^fake_(.*)$                                                #3
        match_type: regexp                                                  #3
        action: update
        operations:                                                         #4
          - action: add_label                                               #5
            new_label: origin
            new_value: fake
      - include: ^fake_(.*)$
        match_type: regexp
        action: update                                                      #6
        new_name: $${1}                                                     #6-7
# Do the same with metrics generated by NodeJS Invoke the metrics transform processor List of transforms applied in order Matches all metrics with the defined regexp List of operations applied in order Add the label Rename the metric by removing the regexp group prefix Fun stuff: syntax is $${x} Finally, we add the defined processor to the pipeline: service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      processors: [ "metricstransform" ]
      exporters: [ "prometheus" ] Here are the results: Connecting receivers and exporters A connector is both a receiver an exporter and connects two pipelines. The example from the documentation receives the number of spans (tracing) and exports the count, which has a metric. I tried to achieve the same with 500 errors - spoiler: it doesn't work as intended. and Let's first add a log receiver: receivers:
  filelog:
    include: [ "/var/logs/generated.log" ] Then, we add a connector: connectors:
  count:
    requests.errors:
      description: Number of 500 errors
      condition: [ "status == 500 " ] Lastly, we connect the log receiver and the metrics exporter: service:
   pipelines:
     logs:
       receivers: [ "filelog" ]
       exporters: [ "count" ]
     metrics:
       receivers: [ "prometheus", "count" ] The metric is named , but its value stays at 1. log_record_count_total Logs manipulation Processors allow data manipulation; operators are specialized processors that work on logs. If you're familiar with the Elasticsearch Logstash Kibana stack, they are the equivalent of Logstash. As of now, the log timestamp is the ingestion timestamp. We shall change it to the timestamp of its creation. receivers:
  filelog:
    include: [ "/var/logs/generated.log" ]
    operators:
      - type: json_parser                                                   #1
        timestamp:                                                          #2
          parse_from: attributes.datetime                                   #3
          layout: "%d/%b/%Y:%H:%M:%S %z"                                    #4
        severity:                                                           #2
          parse_from: attributes.status                                     #3
          mapping:                                                          #5
            error: 5xx                                                      #6
            warn: 4xx
            info: 3xx
            debug: 2xx
      - id: remove_body                                                     #7
        type: remove
        field: body
      - id: remove_datetime                                                 #7
        type: remove
        field: attributes.datetime
      - id: remove_status                                                   #7
        type: remove
        field: attributes.status The log is in JSON format; we can use the provided JSON parser Metadata attributes to set Fields to read from Parsing pattern Mapping table Accept a range, , . The operator has a special interpreted value (and similar) for HTTP statuses. e.g. 501-599 5xx Remove duplicated data Logs At this point, we can send the logs to any log aggregation component. We shall stay in the Grafana Labs sphere and use Loki. exporters:
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push" We can also use logs from the collector itself: service:
  telemetry:
    logs: Finally, let's add another pipeline: service:
  pipelines:
    logs:
      receivers: [ "filelog" ]
      exporters: [ "loki" ] Grafana can also visualize the logs. Choose Loki as a datasource. Conclusion In this post, we delved into the OpenTelemetry collector. While it's not a mandatory part of the OTEL architecture, it's a useful Swiss knife for all your data processing needs. In case you're not stuck to a specific stack or don't want to, it's a tremendous help. The complete source code for this post can be found on . GitHub To go further: OpenTelemetry Collector OpenTelemetry Operators Originally at on November 12th, 2023 published A Java Geek

Delving Into OpenTelemetry Collector

ওপেনটেলিমেট্রি সংগ্রাহকের মধ্যে যোগ করা

Eintauchen in OpenTelemetry Collector

Profundizando en OpenTelemetry Collector

Plonger dans le collecteur OpenTelemetry

ओपनटेलीमेट्री कलेक्टर में गहराई से जाना

OpenTelemetry コレクターの詳細を調べる

OpenTelemetry Collector 살펴보기

Aprofundando-se no OpenTelemetry Collector

Углубляемся в сборщик OpenTelemetry

OpenTelemetry Collector'ı İncelemek

Đi sâu vào Bộ thu thập từ xa mở

深入研究 OpenTelemetry Collector

PHP and FastCGI The way PHP FPM receives client requests is via , which is a high-performance binary protocol for communication between servers, clients, and in general, programs of all kinds. FastCGI Originally, it was created to rectify shortcomings of the CGI (Common Gateway Interface) protocol in order to dramatically increase its performance. Over time, FastCGI emerged as a great protocol for high-performance server applications. The reason for this is that it allows client requests to be handled without starting up and shutting down server threads or processes; the inter-process communication is especially fast with Unix sockets. In addition, the protocol is binary and very lean, meaning it provides high performance. Typically, it's used on secure networks as it doesn't have any security built-in (which is one of the reasons for high performance), such as behind front-facing web server(s), on secure intranets, etc. Here, you will learn how to connect to any FastCGI server (in this case, PHP FPM and Vely) from virtually any programming language (that has C linkage, which is most of them), using ; that's the reason examples are written in C. FastCGI_API Prerequisites This example uses API that comes with the framework (at least 17.1.3 or later should be installed to run these examples). The examples are for Ubuntu 22, so you can install or . You can also it for other Linux distros. Vely Vely with apt packager from the source install Alternatively, if you are using API to connect to a non-Vely server (i.e., PHP or some other), you can do so without installing Vely - see "Using API without Vely" in ; in this case, you cannot run the Vely server example below. Note that in this case you also must install gcc beforehand. FastCGI_API What You Will Do Here You will connect to a PHP FPM and a Vely server using API, and run code on those servers on behalf of a client, meaning send data and receive a reply. Both single- and multi-threaded examples are included. Setup PHP FPM Server The PHP example is for Ubuntu with Apache, however, you can adapt it to your particular distribution. The PHP FPM version used is 8.1; if you're using a different version, replace "8.1" with your own. First, if you don't already have an Apache web server installed: sudo apt install apache2 Then, if you don't already have a PHP FPM server installed: sudo apt install php-fpm In order for your client program to be able to write PHP FPM's Unix socket, you can change the configuration for it - in this case, you'd make the group be the same as yours, thus, giving you permission to write to the socket. To do this, edit file: sudo vi /etc/php/8.1/fpm/pool.d/www.conf And change the line with "listen.group" to your group name (which is normally the same as your OS user name): listen.group = <your login user> Then restart the PHP FPM service: sudo service php8.1-fpm restart First, create a simple PHP program to just output what the input parameter was. Create file: sudo vi /var/www/html/example.php And copy this: <?php
echo "Input is " . $_GET['par'] . "\n";
?> Make sure the PHP file is accessible to PHP FPM (which runs in the same ownership context as the web server, meaning "www-data" user): sudo chown www-data:root /var/www/html/example.php A Simple Example With PHP FPM The C program to connect and call the above PHP file is simple. Note that the version of PHP used is "8.1", so if you're using a different one, replace "php8.1" with your own. You're also using a Unix socket to connect, the request method is "GET", and the rest is set up for a very simple invocation of a PHP script. "url_payload" is in the form of a query string, and PHP also expects SCRIPT_FILENAME to be set to where you saved the "example.php" file; this demonstrates the usage of environment variables. Overall, you can take this simple example and adjust it to serve your needs. Create client C file: vi php_api.c And copy this: #include "vfcgi.h"

void main ()
{
    // Request type vv_fc - create a request variable and zero it out
    vv_fc req = {0};

    req.fcgi_server = "/run/php/php8.1-fpm.sock"; // Unix socket
    req.req_method = "GET"; // GET HTTP method
    req.app_path = "/"; // application path
    req.req = "/example.php"; // request name
    req.url_payload = "par=99";

    char *env[3];
    env[0]="SCRIPT_FILENAME";
    env[1]="/var/www/html/example.php";
    env[2]=NULL;
    req.env = env;

    // Make a request
    int res = vv_fc_request (&req);

    // Check return status, and if there's an error, display error code and the
    // corresponding error message. Otherwise, print out server response.
    if (res != VV_OKAY) printf ("Request failed [%d] [%s]\n", res, vv_fc_error(&req));
    else printf ("%s", vv_fc_data(&req));

    // Free up resources so there are no memory leaks
    vv_fc_delete(&req);
} The example is very simple and fairly self-explanatory. A few things you must always have (see ): FastCGI_API "fcgi_server" member of "req" variable (of "vv_fc" type) says how you plan to connect to a FastCGI server. In the example, you'd use a local Unix socket for simplicity and performance; use a TCP socket only when connecting to another computer or if you have a good reason otherwise. "req_method" is the HTTP method, which, in this case, is "GET", but it could be "POST", "PUT", "DELETE", or anything you want. "app_path" is the SCRIPT_NAME. "req" is the request name, or in HTTP talk it's the leading path segment of PATH_INFO. Also provided is "url_payload", which, in terms of HTTP environment variables, is really the rest of PATH_INFO (divided by forward slashes "/", or up to "?" if there's one), plus QUERY_STRING (in the form of "name=value" pairs, or after "?", if there's one). Finally, "vv_fc_request()" calls the server, and you can use "vv_fc_error()" to get any error response and "vv_fc_data()" to get the data reply. Note that error and data may be interwoven; not to worry, the API will separate the two as proper streams. To delete memory used by this call, use "vv_fc_delete()". That's all for a simple example! To build the client, execute: gcc -o php_api php_api.c $(vv -i) Run it: ./php_api The result is, as expected: Content-type: text/html; charset=UTF-8

Input is 99 Multithreaded Example With PHP FPM The next example is an MT (multi-threaded) client connecting to PHP FPM. You will make 100 simultaneous calls to the PHP FPM server. The code that does this is very similar to the simple example with the exception that the "url_payload" is dynamically constructed so that you can display input from "0" through "99". In the "main()" function, you will create 100 threads and call the "call_server()" function that many times in parallel, then wait for all of them to finish. The result of each thread (i.e., if a call to the server was successful) is passed to "main()" as a return value of "call_server()". Create client C file: vi php_api_mt.c Copy the following: #include "pthread.h"
#include "assert.h"
#include "vfcgi.h"
#define REQ_LEN 200
#define MT_RUNS 100

void *call_server (void *inp)
{
    // Request type vv_fc - create a request variable and zero it out
    vv_fc req = {0};

    req.fcgi_server = "/run/php/php8.1-fpm.sock"; // Unix socket
    req.req_method = "GET"; // GET HTTP method
    req.app_path = "/"; // application path
    req.req = "/example.php"; // request name

    char *env[3];
    env[0]="SCRIPT_FILENAME";
    env[1]="/var/www/html/example.php";
    env[2]=NULL;
    req.env = env;

    req.url_payload = (char*)malloc (REQ_LEN); assert(req.url_payload);
    snprintf (req.url_payload, REQ_LEN, "par=%ld", (off_t)inp);

    int res = vv_fc_request (&req);

    // Check return status, and if there's an error, display error code and the
    // corresponding error message. Otherwise, print out server response.
    if (res != VV_OKAY) {
        fprintf (stderr, "Request failed [%d] [%s]\n", res, vv_fc_error(&req));
    }
    else {
        printf ( "%s", vv_fc_data(&req));
    }

    // Free up resources so there are no memory leaks
    vv_fc_delete(&req);
    return (void*)(off_t)res;
}

int main ()
{
    // Make a request
    pthread_t thread_id[MT_RUNS+1];
    int i;
    for (i = 0; i < MT_RUNS; i++){
        pthread_create(&(thread_id[i]), NULL, call_server, (void*)(off_t)i);
    }
    int bad = 0;
    void *thread_res[MT_RUNS+1];
    for (i = 0; i < MT_RUNS; i++){
        pthread_join(thread_id[i], &(thread_res[i]));
        int r = (int)(off_t)(thread_res[i]);
        if (r != VV_OKAY) bad++;
    }
    if (bad!=0) {
        fprintf (stderr, "Total [%d] bad\n", bad);
        return -1;
    } else {
        printf ("All okay return value\n");
        return 0;
    }
} To build the client, execute: gcc -o php_api_mt php_api_mt.c $(vv -i) -lpthread Run it: ./php_api_mt The result is, as expected (note that since clients work truly in parallel, the numbers are somewhat randomly dispersed): Content-type: text/html; charset=UTF-8

Input is 0
Content-type: text/html; charset=UTF-8

Input is 3
Content-type: text/html; charset=UTF-8

Input is 5
Content-type: text/html; charset=UTF-8

Input is 4
Content-type: text/html; charset=UTF-8

Input is 1
Content-type: text/html; charset=UTF-8

Input is 2
Content-type: text/html; charset=UTF-8

Input is 6
Content-type: text/html; charset=UTF-8

...

Input is 93
Content-type: text/html; charset=UTF-8

Input is 80
Content-type: text/html; charset=UTF-8

Input is 95
Content-type: text/html; charset=UTF-8

Input is 97
Content-type: text/html; charset=UTF-8

Input is 99
Content-type: text/html; charset=UTF-8

Input is 98
Content-type: text/html; charset=UTF-8

Input is 96
Content-type: text/html; charset=UTF-8

Input is 94
All okay return value Setup a Vely Server In order to run a client example with a Vely server, you need to set up and run a server first. First, create a Vely application named "example": sudo vf -i -u $(whoami) example To see highlights for Vely code in vim, run this just one time: vv -m Then create a source Vely file: vi echo.vely And copy this: #include "vely.h"

void echo() {
    out-header default
    input-param par
    @Input is <<p-out par>>
} Build the application: vv -q And start a server, in this case, with 5 worker processes: vf -w 5 example Once you've done this, you can proceed to build a client and call the server. Simple Example With Vely Create a file: vi api.c and copy this: #include "vfcgi.h"

void main ()
{
    // Request type vv_fc - create a request variable and zero it out
    vv_fc req = {0};

    req.fcgi_server = "/var/lib/vv/example/sock/sock"; // Unix socket
    req.req_method = "GET"; // GET HTTP method
    req.app_path = "/example"; // application path
    req.req = "/echo"; // request name
    req.url_payload = "par=99";

    // Make a request
    int res = vv_fc_request (&req);

    // Check return status, and if there's an error, display error code and the
    // corresponding error message. Otherwise, print out server response.
    if (res != VV_OKAY) printf ("Request failed [%d] [%s]\n", res, vv_fc_error(&req));
    else printf ("%s", vv_fc_data(&req));

    // Free up resources so there are no memory leaks
    vv_fc_delete(&req);
} The example is very similar to PHP FPM one. For Vely, see for more on application path (i.e., "app_path"), request path (i.e., "req"), and URL payload (i.e., "url_payload"); Vely's nomenclature is more geared toward how's data used in a logical sense, but the underlying usage (including environment variables) is exactly the same. request_URL To build the client, execute: gcc -o api api.c $(vv -i) Run it: ./api The result is: Content-type: text/html;charset=utf-8
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Status: 200 OK

Input is 99 This is the expected result. Multi-Threaded Example With Vely The next example is an MT (multi-threaded) client. You will make 100 simultaneous calls to a Vely server. The code that does this is very similar to the simple example, with one addition: you'll pass along an extra environment variable, in this case, "VV_SILENT_HEADER" with the value "yes", which will suppress HTTP header output from the server. Otherwise, the "url_payload" is dynamically constructed, so that you can display input "0" through "99". In the "main()" function, you will create 100 threads and call the "call_server()" function that many times in parallel, then wait for all of them to finish. The result of each thread (i.e., if a call to the server was successful) is passed to "main()" as a return value of "call_server()". Create client C file: vi api_mt.c Copy the following: #include "pthread.h"
#include "assert.h"
#include "vfcgi.h"
#define REQ_LEN 200
#define MT_RUNS 100

void *call_server (void *inp)
{
    // Request type vv_fc - create a request variable and zero it out
    vv_fc req = {0};

    req.fcgi_server = "/var/lib/vv/example/sock/sock"; // Unix socket
    req.req_method = "GET"; // GET HTTP method
    req.app_path = "/example"; // application path
    req.req = "/echo"; // request name
    char *env[3];
    env[0]="VV_SILENT_HEADER";
    env[1]="yes";
    env[2]=NULL;
    req.env = env;

    req.url_payload = (char*)malloc (REQ_LEN); assert(req.url_payload);
    snprintf (req.url_payload, REQ_LEN, "par=%ld", (off_t)inp);

    int res = vv_fc_request (&req);

    // Check return status, and if there's an error, display error code and the
    // corresponding error message. Otherwise, print out server response.
    if (res != VV_OKAY) {
        fprintf (stderr, "Request failed [%d] [%s]\n", res, vv_fc_error(&req));
    }
    else {
        printf ( "%s", vv_fc_data(&req));
    }

    // Free up resources so there are no memory leaks
    vv_fc_delete(&req);
    return (void*)(off_t)res;
}

int main ()
{
    // Make a request
    pthread_t thread_id[MT_RUNS+1];
    int i;
    for (i = 0; i < MT_RUNS; i++){
        pthread_create(&(thread_id[i]), NULL, call_server, (void*)(off_t)i);
    }
    int bad = 0;
    void *thread_res[MT_RUNS+1];
    for (i = 0; i < MT_RUNS; i++){
        pthread_join(thread_id[i], &(thread_res[i]));
        int r = (int)(off_t)(thread_res[i]);
        if (r != VV_OKAY) bad++;
    }
    if (bad!=0) {
        fprintf (stderr, "Total [%d] bad\n", bad);
        return -1;
    } else {
        printf ("All okay return value\n");
        return 0;
    }
} To build the client, execute: gcc -o api_mt api_mt.c $(vv -i) -lpthread Run it: ./api_mt The result is as expected (the input numbers are somewhat randomly dispersed since all clients work truly in parallel): Input is 0
Input is 3
Input is 4
Input is 6
Input is 5
Input is 2
Input is 1
Input is 7
Input is 8
...
Input is 71
Input is 74
Input is 75
Input is 78
Input is 77
Input is 70
Input is 65
Input is 76
Input is 67
All okay return value Conclusion You have learned how to connect to any FastCGI server (in this case, PHP FPM and Vely) from a program with C linkage (which can be used with any programming language that has it) using . FastCGI_API You have also learned how to make many parallel API calls using Linux threads. You can use this knowledge to call server code from any other application and receive results. Also published here

Connecting to PHP FPM With API: Multi-Threaded and Single-Threaded Calls: A Guide 

Let's learn about via these 51 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Terraform /Learn Repo 1. 6 Free Terraform Tools to Facilitate a Multi-Cloud Infrastructure Have a multi-cloud infrastructure? These free Terraform tools will help you to reduce costs and make your multi-cloud efficient. 2. Developer Trends from Thoughtworks’ Technology Radar 27 Thoughtworks’ Technology Radar is a regular time to take a look at what trends are changing our industry. 3. Serverless API with Terraform: GO and AWS [Part 2] In part 2, we'll go over deploying serverless API to AWS Cloud with Terraform. 4. How a Unicorn Startup in Japan Leveraged the Power of Microservices Startup experience in Mercari. Road to microservices from monolith. 5. Getting Started With Creating, Managing an AWS ECS Cluster With Terraform In this tutorial, you'll define AWS resources using Terraform and how resource definitions translate to actual resources created on AWS 6. Basics of AWS Tags & Terraform with S3 - Part 1 In this video course, you'll learn the basics of AWS tags and Terraform with an S3 bucket. 7. How to Test Terraform the Right Way In this blog post, you'll learn about why you should be testing IaC and the best platforms to get started with. 8. Static Code Analysis  for Infrastructure as Code Using Azure DevOps Pipelines This article explains how to set up static code analysis for infrastructure as code using Azure DevOps pipelines. 9. Should Deploying Production Infrastructure be Hard? Below I'll explain why I built terra_boi to deploy Clientelify, why I think it was downloaded 1,000 times in the first 10 days (despite not promoting it anywhere), and why deploying an application to production shouldn't suck. 10. How To Manage Infrastructure With Terraform What is Terraform? 11. Getting Started Provisioning an AWS EKS Kubernetes Cluster with Terraform Learn how to provision an AWS EKS Kubernetes cluster with Terraform. See a full step-by-step tutorial. 12. How Infrastructure as Code Is Forcing a Revolution in Cost Management You might have read our beta launch blog post, which describes how Infrastructure as Code (IaC) is driving the third datacenter revolution (the first two being virtualization and the move to public clouds). In this post, I want to go deeper into looking at how IaC is driving a new need and approach to cost visibility and management. 13. Automating User Management in the Company Have you ever faced the challenge of automating users in your company? 14. Introducing Driftctl: Your IaC Security Belt We recently released the first versions of driftctl, a new open-source project for infrastructure developers, DevOps, SRE, and cloud practitioners, with the goal of helping manage all kinds of drifts. 15. Deploying a Terraform Remote State Backend with AWS S3 and DynamoDB In this article, we are going to be deep diving into remote state management in terraform. 16. Another Reason to Use Docker Recently I was working on a project which includes Terraform and AWS stuff. While working on that I was using my local machine for terraform code testing and luckily everything was going fine. But when we actually want to test it for the production environment we got some issues there. Then, as usual, we started to dig into the issue and finally, we got the issue which was quite a silly one 😜. The production server Terraform version and my local development server Terraform version was not the same. 17. Building Maintenance Mode for your API Gateway Using Terraform In my previous blog post, I went through how to create a maintenance mode page for your application, and how to implement it using Terraform and github pages. But the website is just one part of an Application, and often there’s also a public (or private) API that also needs to have a maintenance mode. Let’s see how we can do that using Terraform on API Gateway. 18. How to Deploy a Django Application on AWS with Terraform The final goal of this guide is to create a scalable and reproducible setup of the Django Web Application on AWS using Terraform. 19. How To Sync Terraform Config And .tfstate File For Existing AWS Resources Terraform is a great (and dominant) infrastructure automation tool. It is multi-cloud, can build all sorts of resources, and in some cases supports API calls to build resources before the native tooling from cloud providers does. 20. Managing Active Directory Objects with Azure AD Provider for Terraform Learn how to manage Active Directory Objects with Azure AD Provider for Terraform. See examples of how to authenticate and grant the correct permissions. 21. Why you should take care of Infrastructure Drift When talking about infrastructure drift, you often get knowing glances and heated answers. Recording gaps in your infra between what you expected to be and the reality of what is, is a well known and widespread issue bothering hundreds of DevOps teams around the globe. 22. Managing a GitHub Organization With Infrastructure as Code This post will share our experience in adopting infrastructure as code (IaC) to manage GitHub organization resources. 23. Using Jupyter/TensorBoard In Any Cloud With One Command Using Terraform Provider Iterative for bespoke Machine Learning & Data Science on AWS, Azure, GCP and K8s 24. The Top 5 DevOps Tools and Services to Consider as a Startup A collection of DevOps tools can significantly enhance your Software Development Life Cycle (SDLC) and increase engineering productivity. 25. Change the Name of your AWS S3 Bucket in Terraform Without Breaking Things Learn how to rename an AWS S3 bucket in Terraform. See renaming process of Amazon s3 bucket in a few simple steps. 26. Exporting Your GKE Cluster to Terraform Cloud: A Guide with Challenges and Solutions In this article, I'll share our journey at ANNA Money from using our Kubernetes installation in GCP to the managed Kubernetes service by Google and GKE. 27. Hashicorp's Terraform: A Introduction Join us in the journey of learning Terraform. This blog series aims to provides a guidance for essentials to be known while working with Terraform. 28. A Handy Guide to Migrating Terraform State to GitLab CI/CD Migrating Terraform state to GitLab CI/CD is a GitOps best practice that allows multiple engineers to work together to develop infrastructure. 29. How Namespace and Terraform Can Manage Kubernetes Environments In the last several years, Kubernetes has become the “go to” standard for managing and orchestrating containerized workloads. Thanks to it’s vendor agnostic nature, you can easily run Kubernetes almost anywhere, and in fact, all the major cloud vendors offer a managed Kubernetes service (AWS EKS, Google GKE, and Azure AKS). 30. Building a “Maintenance Mode” with Terraform and Github Pages As you may have seen, we just launched our public beta of env0 last month. As part of the run-up to the launch, our dev team had to go through and make sure everything about our infrastructure was ready for ongoing public use: one element being creating a maintenance mode for both our Application and our public API. 31. How To Migrate An Existing Infrastructure into Terraform Terraform is a powerful tool to have in your toolset. 32. How To Setup Continuous Integration Pipeline By Using Terraform And GitLab CI Terraform is a fantastic tool for managing your cloud infrastructure, especially if your assets are hosted on multiple cloud providers. 33. Learn How to Build a CI/CD Pipeline Using GitLab for your Website We'll run through how to build and configure a CI/CD pipeline to deploy your business's website using AWS, Terraform & Gitlab. 34. How to Optimize for Iteration: Choosing Your Early Startup's Tech Stack As we’ve discussed in a previous blog post, product development is hypothesis testing. This is especially true in the early stages of a company when you need to confirm or reject your hypothesis as quickly as possible. This process is then repeated until you (hopefully) reach product-market fit. To get there, your team needs to be able to work and build at a pace that allows for this constant and rapid iteration. 35. What Really Propelled The Cloud Revolution: A Developer's Perspective An overview of the main trends that have pushed the Cloud industry — and the world — forward, in the past decade and a half. 36. Sanity-Checks with Playwright in Fargate container on schedule There is often the need to check whether your production system is working as expected. Playwright is a great way to test if everything is going to plan. 37. From “Feature branches” to “Feature environments” with Terraform “The advantage of feature branching is that each developer can work on their own feature and be isolated from changes going on elsewhere.” (FeatureBranch) 38. Serverless API with Terraform, GO and AWS, Part 1 Knowing how to build REST API with latest tech is cool. You know what's even cooler? It's being able to deploy it to the cloud! I'll walk you through the process of building simple, server-less application using GO, AWS (Lambda, API Gateway, Dynamodb) and Terraform. 39. Improve Your IaC With a Specialized Management Platform Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. 40. Kubernetes Terraforming on Linode: Linode + Terraform + Kubernetes = Cloud Nirvana Let's discuss the process of "Terraforming" a Kubernetes cluster on Linode using Terraform to help organizations efficiently manage and scale  infrastructure. 41. 10+ Best Resources to learn DevOps in Depth [Online Courses] DevOps is one of the most in-demand skills from employer and there are many job opportunities lying for full stack developers, distinguished engineers and DevOps professionals. If you are an experienced Java programmer or a full-stack web developer, and want to become a DevOps engineer then you have come to the right place. 42. Ad Hoc Environments for Django Applications With ECS, Terraform and GitHub Actions This article will show how software development teams can build on-demand environments for dog-food testing, quality review, internal and external demos and oth 43. Create Your New Reality With Wirehole And Other Awesome Automation Tools **UPDATE October 13, 2020 - The mjtechguy/terransible-wirehole Github repo now supports deploying to Digital Ocean and Hetzner Cloud as well! 44. GraphQL Is Not a Silver Bullet GraphQL is not a configuration language, also not a transformation language, a replacement for SQL, and definitely not Terraform. 45. Building an AWS EKS Fargate Cluster with Terraform How to build an AWS EKS Fargate cluster using terraform. 46. Terraform Configuration Syntax Overview All you need to know to get started about Terraform Syntax. 47. Is Terraform Still Useful? Terraform is an agentless infrastructure tool used mainly by DevOps for infrastructure management. Find out why you should use Terraform for infrastructure. 48. 6 DevOps Trends in 2022 That DevOps Engineers Should Adopt The role of DevOps engineers, their responsibilities, growth opportunities, a set of important soft & hard skills, and most importantly, DevOps trends in 2022. 49. Deploying an Open Source Secure Video Conference Server [A How To Guide] In this article, you'll create a new DigitalOcean account using a free credit link. Then, you will clone a GitHub repository, and use Terraform code to initialize, plan and apply resources to your account, using a preconfigured image from DigitalOcean and your custom domain name. Finally, you will access your account remotely via SSH and execute two scripts to configure both your video conference server and Let's Encrypt SSL to enable HTTPS. 50. Newsflash - "People with AWS Console Access Tend to Change Things" This blog post is a written transcript of the FOSDEM Talk: “Infrastructure drifts aren’t like Pokemon, you can’t catch ’em all”, by Stephane Jourdan – CTO and founder 51. Introduction to AWS Log Insights as CloudWatch Metrics A step-by-step description of how to create an AWS Lambda to convert Cloudwatch LogInsights into metrics Thank you for checking out the 51 most read stories about Terraform on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

51 Stories To Learn About Terraform

Let's learn about via these 46 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Infrastructure As Code /Learn Repo 1. How To Configure CloudFront Using CloudFormation Template I recently worked on implementing CloudFront for s3 bucket files. Most of the tutorials were doing that using console management(UI) but we at Blue Sky Analytics prefer “code as infrastructure”. So I had to do this using CloudFormation. With the help of a few StackOverflow links and tutorials, I was able to write CloudFormation Template. 2. How to Test Terraform the Right Way In this blog post, you'll learn about why you should be testing IaC and the best platforms to get started with. 3. The Relevance of Infrastructure-as-Code Concepts in Security So what is Infrastructure-as-Code (IaC) anyway? IaC is a process that automates the provisioning and management of cloud resources. 4. Change the Name of your AWS S3 Bucket in Terraform Without Breaking Things Learn how to rename an AWS S3 bucket in Terraform. See renaming process of Amazon s3 bucket in a few simple steps. 5. Using a Product-Led Growth Mindset to Improve User Experience How changing the development mindset to a product-led-growth one can improve the user experience by focusing the thought process for the team's members. 6. What Really Propelled The Cloud Revolution: A Developer's Perspective An overview of the main trends that have pushed the Cloud industry — and the world — forward, in the past decade and a half. 7. Static Code Analysis  for Infrastructure as Code Using Azure DevOps Pipelines This article explains how to set up static code analysis for infrastructure as code using Azure DevOps pipelines. 8. 6 Free Terraform Tools to Facilitate a Multi-Cloud Infrastructure Have a multi-cloud infrastructure? These free Terraform tools will help you to reduce costs and make your multi-cloud efficient. 9. How to Create an Azure API Management Instance using Bicep Lang via Azure DevOps The more I use Bicep, the more I love it. This is what ARM Templates should have been. 10. Deploy Docker Using Ansible 11. Smart E-bike Kits: A Tour Inside Our Servers The Idea 12. Learn About Infrastructure as Code in 5 Minutes and Why You Should Use It Infrastructure as Code (IaC) is the process of managing and configuring an infrastructure using configuration files, rather than manually editing configurations 13. Improve Your IaC With a Specialized Management Platform Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. 14. Managing a GitHub Organization With Infrastructure as Code This post will share our experience in adopting infrastructure as code (IaC) to manage GitHub organization resources. 15. Why Learn Any Other Programming Language When You Can Learn Typescript? Use TypeScript in all your projects and all use cases without any limitation. You can now do frontend, backend, infrastructure and machine learning in one progr 16. Code as Documentation and Assessment Documenting IT networking lab procedures in code rather than word-processor documents enables more authentic, equitable and consistent assessment. 17. AWS CloudFormation Template Basics Get started creating IaC using CloudFormation templates. 18. Why Understanding Infrastructure-as-code is Critical to Scaling Your Technology Infrastructure Infrastructure as code (IaC) is one of the most important trends in modern IT development. Read about the benefits of IaC and how to implement it. 19. How Infrastructure as Code Is Forcing a Revolution in Cost Management You might have read our beta launch blog post, which describes how Infrastructure as Code (IaC) is driving the third datacenter revolution (the first two being virtualization and the move to public clouds). In this post, I want to go deeper into looking at how IaC is driving a new need and approach to cost visibility and management. 20. How To Manage Infrastructure With Terraform What is Terraform? 21. Codify your SaaS Apps: The Answer to the Unmanaged SaaS Jungle A phenomenon we have encountered often, when helping companies overcome drift, is a common neglect of the entire SaaS toolchain. Learn why this is troubling. 22. Three Ways to Scale Your Apps with Render When your app is experiencing high traffic or utilization, you need to scale your service to handle that load. We’ll explore three ways to scale your apps. 23. Building Your Infrastructure With Just a Diagram This hands-on tutorial will teach you how to create infrastructure via the AWS Application Composer Console. 24. Check Out the Principles and Best Practices of Infrastructure as Code for 2023 Infrastructure-as-code is a very important concept to understand in the DevOps world today. 25. How To Migrate An Existing Infrastructure into Terraform Terraform is a powerful tool to have in your toolset. 26. Hashicorp's Terraform: A Introduction Join us in the journey of learning Terraform. This blog series aims to provides a guidance for essentials to be known while working with Terraform. 27. 2023 Will Be the Year of Kubernetes (and Other Predictions in the Cloud and Infrastructure Industry) Dive into the new year’s top cloud and infra-tech trends, straight from a cloud and infrastructure technology fanatic. 28. Introducing Driftctl: Your IaC Security Belt We recently released the first versions of driftctl, a new open-source project for infrastructure developers, DevOps, SRE, and cloud practitioners, with the goal of helping manage all kinds of drifts. 29. Cloud Application Infrastructure from Code (IfC) Presents a relatively new concept of Infrastructure from Code (IfC) as the next logical step in cloud automation 30. Project-as-Code: The Catalyst DevOps Needs If you are familiar with Docker, Terraform, and the CI platforms (eg: Jenkins, CircleCI, Codeship), you already know the power of Declarative DevOps. It can make development easier by being repeatable, predictable, and fast. Supporting technologies both hide complexities and offer important reuse by supporting simple structured syntax in an easy to create and read file. Each technology has codified much of its domain allowing developers to author and instrument with nothing more than a text editor. 31. I Got a New EKS Cluster Running And Terraform Code in 2 minutes AWS EKS + Terraform + Cloudskiff do the job 32. B2B Fintech Trends for 2021: Storage, Security, Cloud Migration, IaC As CTO at Altar.io, a large part of my work revolves around observing the forward-looking industry trends (including fintech trends) that affect how we create innovative products. 33. Why you should take care of Infrastructure Drift When talking about infrastructure drift, you often get knowing glances and heated answers. Recording gaps in your infra between what you expected to be and the reality of what is, is a well known and widespread issue bothering hundreds of DevOps teams around the globe. 34. The State of Cloud Automation and the Promise of Infrastructure Automation The Cloud has become a familiar feature of the tech stacks at most organizations and is becoming much more than a place for server resources. At present, almost everything in our digital landscape is connected to a Cloud in one way or another. 35. From “Feature branches” to “Feature environments” with Terraform “The advantage of feature branching is that each developer can work on their own feature and be isolated from changes going on elsewhere.” (FeatureBranch) 36. How To Setup Continuous Integration Pipeline By Using Terraform And GitLab CI Terraform is a fantastic tool for managing your cloud infrastructure, especially if your assets are hosted on multiple cloud providers. 37. Creating and Provisioning Azure Container Apps with Bicep Using Bicep, we can deploy Azure Container Apps quickly and easily! 38. Building a “Maintenance Mode” with Terraform and Github Pages As you may have seen, we just launched our public beta of env0 last month. As part of the run-up to the launch, our dev team had to go through and make sure everything about our infrastructure was ready for ongoing public use: one element being creating a maintenance mode for both our Application and our public API. 39. Application Deployment: Recipe for Great Serverless Strategy This is true for the most part, but Serverless apps also have a certain property that can make their deployment and maintenance time consuming. 40. Exporting Your GKE Cluster to Terraform Cloud: A Guide with Challenges and Solutions In this article, I'll share our journey at ANNA Money from using our Kubernetes installation in GCP to the managed Kubernetes service by Google and GKE. 41. What is Everything-as-Code? Examining  the Explosion of "as Code" Buzzwords If you’re confused when you read about “[some software term] as code” or “everything as code,” all you really need to know is that we’re talking about automation: The thing we use to do tedious tasks for us, or orchestrate tasks when they become too large and complex for manual methods. 42. Creating a Instance Scheduler using AWS CDK The AWS CDK is a software development framework to define cloud infrastructure as code and provision it through CloudFormation. The CDK integrates fully with AWS services and allows developers to use high-level construct to define cloud infrastructure in code. 43. Cloud-Native Yay or Nay: Reasons to Use Cloud-Native Technologies Cloud-native has been the talk of the town for quite some time now. Some developers think it’s just hyped way too much and will soon diminish from the limelight. Others think cloud-native will revolutionize software development and is here to stay! 44. Newsflash - "People with AWS Console Access Tend to Change Things" This blog post is a written transcript of the FOSDEM Talk: “Infrastructure drifts aren’t like Pokemon, you can’t catch ’em all”, by Stephane Jourdan – CTO and founder 45. How Do You Manage Helm Charts? Whether you love it or hate it, Helm is a ubiquitous tool for managing Kubernetes applications. You can use it in many different ways which is great, but can also be overwhelming. 46. Terraform Configuration Syntax Overview All you need to know to get started about Terraform Syntax. Thank you for checking out the 46 most read stories about Infrastructure As Code on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

env0

env0.com

31 employees

Since 2018

COMPANY RANKING

env0

EVERGREEN INDEX #1727

See more notable links about env0

Apply to get your Tech Company News Page live on HackerNoon.com today!

Read More Tech Stories Related to #env0

env0

env0

env0.com

31 employees

Since 2018

COMPANY RANKING

env0

EVERGREEN INDEX #1727

Related CompaniesRank

env0 Story MentionsComplete List

See more notable links about env0

Apply to get your Tech Company News Page live on HackerNoon.com today!

Read More Tech Stories Related to #env0

env0