Agentnoon

agentnoon

Effective metrics and monitoring play a key role in high-quality development, bug fixes, and handling of user requests and incidents. At Social Discovery Group, we employ a diverse range of tools to evaluate the performance of our products. In this article, we will share some hacks of how we managed to build a stable application monitoring and metrics system and reduced bug detection time on our products with the help of Datadog. SDG products are helping more than 250 mln people connect and build relationships globally, and our user base is constantly growing. The main factor in this success is our ability to promptly respond to user needs. We have a lot of experience working with various monitoring systems, including Datadog. Here's why: Currently, Datadog offers a variety of integration options with systems at different levels. You can explore these capabilities . here Datadog provides comprehensive documentation for configuring integrations with various services. We managed to build strong partnership relations. Here, we're sharing our experience in setting up and configuring APM metrics for applications within our Kubernetes cluster for Datadog. The article won't cover the deployment of projects in AKS, the CI/CD process, and other DevOps details. Instead, we will focus on the finer points of configuring Datadog monitoring for APM metrics. The technology stack used: Azure Services, Azure Kubernetes Service (AKS), ASP .Net Core 7, Datadog. For monitoring applications and services, we utilize the Datadog agent deployed within the cluster through a Helm chart with additional parameters from the values.yaml file. In the main agent configuration file, you'll need to enable the DogStatsD module and specify the port (default is 8125). For the data sent from an external host, the Datadog agent requires the following configuration: dogstatsd_non_local_traffic: true and apm_non_local_traffic: true. Here is the values.yaml file for one of the clusters, with some variables being passed at the deployment stage. The deployment is produced with Azure Devops. " 
datadog:
  apiKey: #{apiKey}#
  appKey: #{appKey}#
  clusterName:  #{ClusterName}#
  kubeStateMetricsEnabled: true
  clusterChecks:
    enabled: true
  dogstatsd:
    useSocketVolume: false
    nonLocalTraffic: true
  collectEvents: false
  apm:
    portEnabled: true #important to include
  env:
    - name: "DD_KUBELET_TLS_VERIFY"
      value: "false"
  systemProbe:
    collectDNSStats: false
  orchestratorExplorer:
    enabled: false
clusterAgent:
  image:
    repository: public.ecr.aws/datadog/cluster-agent
    tag: #{tag}#
  admissionController:
    enabled: false
agents:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
    doNotCheckTag:  true
clusterChecksRunner:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
" Then, you need to specify the agent's address for transmitting metrics to Datadog in the application settings. The dashboards for app monitoring are based on the internally used metrics; they had to be created independently. To set up monitoring of ASP.Net services, we used the official documentation that can be found by the . link Since the agent was already configured, one of the methods was to add the necessary lines to the image build and pass variables in the CI/CD system: DD_ENV, DD_SERVICE, DD_AGENT_HOST, to specify the environment, service name, and agent address, respectively. We also need to add the following to the dockerfiles for the services: "
RUN TRACER_VERSION=$(curl -s \https://api.github.com/repos/DataDog/dd-trace-dotnet/releases/latest | grep tag_name | cut -d '"' -f 4 | cut -c2-) \
    && curl -Lo /tmp/datadog-dotnet-apm.deb https://github.com/DataDog/dd-trace-dotnet/releases/download/v${TRACER_VERSION}/datadog-dotnet-apm_${TRACER_VERSION}_amd64.deb

# Copy the tracer from build target
COPY --from=build /tmp/datadog-dotnet-apm.deb /tmp/datadog-dotnet-apm.deb
# Install the tracer
RUN mkdir -p /opt/datadog \
    && mkdir -p /var/log/datadog \
    && dpkg -i /tmp/datadog-dotnet-apm.deb \
    && rm /tmp/datadog-dotnet-apm.deb
 
# Enable the tracer
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_DOTNET_TRACER_HOME=/opt/datadog
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json

" This method works, but it didn't seem like the most optimal solution. We decided to take it a step further and added the following to the deployments of our services: "
metadata.labels:
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1552'

spec.template.metadata.labels:
    admission.datadoghq.com/config.mode: service
    admission.datadoghq.com/enabled: 'true'
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1111'

spec.template.metadata.annotations:
    admission.datadoghq.com/dotnet-lib.version: v2.38.0

spec.template.spec.containers.name.env:
    - name: DD_TRACE_AGENT_URL
      value: datadog-agent.monitoring
    - name: DD_TRACE_STARTUP_LOGS
      value: 'true'
    - name: DD_LOGS_INJECTION
      value: 'true'
    - name: DD_RUNTIME_METRICS_ENABLED
      value: 'true'
    - name: DD_PROFILING_ENABLED
      value: 'true'
    - name: DD_APPSEC_ENABLED
      value: 'true'
" That’s what was changed in the agents: "
datadog:
  apm:
    socketEnabled: true
    portEnabled: true
    enabled: true
clusterAgent:
  admissionController:
    enabled: true
    mutateUnlabelled: false
providers:
  aks:
    enabled: true
" After all these steps, the data with highly detailed metrics for all services started flowing in Datadog under the APM -> Services section, and the graphs were automatically displayed. We had to tinker with the annotations settings for the second method; not everything started working smoothly right away. Regarding the notification system, it's worth mentioning that it is user-friendly and intuitive in Datadog. Notifications are created in the "Monitors -> Manage Monitors" section. The improvements we described above yielded several valuable outcomes. We now have a deeper understanding of how our system operates and adapts to various changes. Additionally, we've established a stable application monitoring and metrics system that operates independently of service builds, helping to reduce bug detection times. This, in turn, has allowed us to optimize our services and improve development speed and overall system quality.

How to Use Datadog for the APM Metrics Application

এপিএম মেট্রিক্স অ্যাপ্লিকেশনের জন্য ডেটাডগ কীভাবে ব্যবহার করবেন

So verwenden Sie Datadog für die APM Metrics-Anwendung

Cómo utilizar Datadog para la aplicación de métricas APM

Comment utiliser Datadog pour l'application APM Metrics

एपीएम मेट्रिक्स एप्लिकेशन के लिए डेटाडॉग का उपयोग कैसे करें

APM メトリクス アプリケーションに Datadog を使用する方法

APM 지표 애플리케이션에 Datadog을 사용하는 방법

Como usar o Datadog para o aplicativo de métricas APM

Как использовать Datadog для приложения APM Metrics

APM Metrics Uygulaması için Datadog Nasıl Kullanılır?

Cách sử dụng Datadog cho ứng dụng Số liệu APM

如何使用 Datadog 进行 APM 指标应用程序

In my 11 years as a developer, I have seen so many API's that have major security flaws. They either lack proper authentication or authorisation or both. Developers might feel like everything's ok, since those endpoints are usually not public. But it is a huge security loop hole which anyone can easily target. To better understand API security, let's create a demo project for the FBI. There will be an Admin who can enroll FBI Agents and change their clearance levels. FBI Agents with will be able to access public files, and agents with will be able to access pubic and classified files. Clearance Level 1 Clearance Level 2 But before we get started, here's some theory. How Authentication Works Our Agent has successfully cleared all their exams; time to enroll them. In order to do that they will provide their documents and in return will get their badge. In the above scenario, is like logging in – where once verified, the agent will be provided with a token (badge). This process is called . It determines whether agents are who they claim to be. providing documents Authentication We are going to use JSON Web Tokens (JWT) Bearer tokens for authentication. are a type of token that's generated by servers, and which contain details of the claims/roles of a user trying to login. Bearer tokens are mostly structured tokens like JWT. You can if you want to learn more. Bearer tokens read more about JWT here How Authorisation Works Now since the FBI Agent has gotten their badge, they can enter the FBI building. They are also able to access public files, but when trying to access classified files they get a . 401 error This is because FBI Agent is not to access classified files. determines what agents can and cannot access. authorised Authorisation As mentioned above, the JWT Bearer token contains claims/roles. Based on it, our server decides whether to give access to a private resource or not. Access Flow As you can see in the above diagram, on successful login the server returns a Bearer token. The client uses the bearer token in subsequent calls to access a private resource. These are the two main concepts that we are going to implement in our article. Enough with the theory, show me some code! How to Set Up our Project Create a new project by executing the command from your CLI. It will create a project with a sample WeatherForecast API. dotnet new webapi --name FBI Why work on WeatherForecast when we can work on FBI? Go ahead and delete the WeatherForecast.cs file. Add the necessary dependencies by executing these commands: dotnet add package Microsoft.IdentityModel.Tokens --version 6.9.0
dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer --version 5.0.4 In the ConfigureServices function in your Startup.cs file, add the below code: var TokenValidationParameters = TokenValidationParameters
{
    ValidIssuer = ,
    ValidAudience = ,
    IssuerSigningKey = SymmetricSecurityKey(Encoding.UTF8.GetBytes( )),
    ClockSkew = TimeSpan.Zero // remove token expire
}; new "https://fbi-demo.com" "https://fbi-demo.com" new "SXkSqsKyNUyvGbnHs7ke2NCq8zQzNLW7mPmHbnZZ" delay of when We are defining the parameters for validating a token. Make sure that the length of the string for generating SymmetricSecurityKey is 32. Next, setup the services to add authentication for API's like this: services ; .AddAuthentication ( =>
    {
        options. = JwtBearerDefaults.AuthenticationScheme;
    }) options DefaultScheme .AddJwtBearer ( =>
    {
        cfg. = TokenValidationParameters;
    }) cfg TokenValidationParameters The AddAuthentication method registers services required by authentication services. It also configures the JWT Bearer Authentication as the default scheme. The AddJwtBearer enables JWT-bearer authentication and setting the TokenValidationParameters defined above. Now let's add some Authorisation claims for our Agent and Admin. services.AddAuthorization(cfg =>
    {
        cfg.AddPolicy( , => policy.RequireClaim( , ));
        cfg.AddPolicy( , => policy.RequireClaim( , ));
        cfg.AddPolicy( , => policy.RequireClaim( , , ));
        cfg.AddPolicy( , => policy.RequireClaim( , ));
    }); "Admin" policy "type" "Admin" "Agent" policy "type" "Agent" "ClearanceLevel1" policy "ClearanceLevel" "1" "2" "ClearanceLevel2" policy "ClearanceLevel" "2" The AddAuthorization method registers services required for authorisation. We are also adding claims for Admin, Agent, ClearanceLevel1 and ClearanceLevel2 by calling AddPolicy. A claim is a name value pair that represents what the subject is. Since clearance level 2 can also access clearance level 1, we have put in ClearanceLevel1. You can read more about claims . "1", "2" here Lastly, in the Configure method, add the below line just above app.UseAuthorization(); app.UseAuthentication() ; The Admin Controller File Rename your file WeatherForecastController.cs to AdminController.cs. Do change the class name and constructor names as well. Finally, remove everything except the constructor. Microsoft.AspNetCore.Mvc; {
    [ ]
    [ ] : { { }
    }
} using namespace FBI.Controllers ApiController Route( ) "[controller]" public class AdminController ControllerBase ( ) public AdminController Let's create a login API for Admin so that they can get a token to perform other tasks. [ ]
[ ] { Claims = List<Claim>
            { Claim( , ),
            }; Key = SymmetricSecurityKey(Encoding.UTF8.GetBytes( )); Token = JwtSecurityToken( , ,
        Claims,
        expires: DateTime.Now.AddDays( ),
        signingCredentials: SigningCredentials(Key, SecurityAlgorithms.HmacSha256)
    ); OkObjectResult( JwtSecurityTokenHandler().WriteToken(Token));
} HttpPost Route( ) "[action]" IActionResult ( ) public Login [FromBody] User User // Authenticate Admin with Database TODO: // If not authenticate return 401 Unauthorized // Else continue with below flow var new new "type" "Admin" var new "SXkSqsKyNUyvGbnHs7ke2NCq8zQzNLW7mPmHbnZZ" var new "https://fbi-demo.com" "https://fbi-demo.com" 30.0 new return new new In the above code, User is a model with properties Username and Password. We are also creating an object of JwtSecurityToken using configurations that we have used in our Startup.cs file. The token is then converted to a string and returned in an OkObjectResult. You can now open Swagger and execute the API to see a bearer token. A bearer token will be returned as you can see below. Keep the token handy since we are going to use it in the next section. You can also visit to analyse your token. https://jwt.io How to Generate the Badge API Generating the badge for an Agent is a sensitive task and should only be authorised by an Admin. We are going to add an Authorize attribute for the GenerateBadge API. [ ]
[ ]
[ ] { CreatedResult( , Agent);
} HttpPost Route( ) "[action]" Authorize(Policy = ) "Admin" IActionResult ( ) public GenerateBadge [FromBody] Agent Agent // Add the agent to the database TODO: return new "Agent/1" Here, Agent is a model with properties Name as string and ClearanceLevel as int. Now when you go back to swagger and try to execute the GenerateBadge API it will give you a 401 Unauthorised response. We are getting this error because we have not passed the bearer token. To be able to add the Authorize header in Swagger, change the services.AddSwaggerGen as below: services.AddSwaggerGen(c =>
{
    c.SwaggerDoc( , new OpenApiInfo { Title = , Version = });
    c.AddSecurityDefinition( , new OpenApiSecurityScheme
    { = ParameterLocation.Header,
        Description = ,
        Name = , = SecuritySchemeType.ApiKey
    });
    c.AddSecurityRequirement(new OpenApiSecurityRequirement {
    { new OpenApiSecurityScheme
            {
                Reference = new OpenApiReference { = ReferenceType.SecurityScheme, Id = }
            },
        new string[] {}
    }
    });
}); "v1" "FBI" "v1" "Bearer" In "Please enter JWT with Bearer into field" "Authorization" Type Type "Bearer" When you refresh Swagger in your browser you will notice an button on the right side above the list of APIs. Authorize Click on the newly added Authorize button in Swagger which will open up a dialog. We need to mention what type of token it is. So first enter in the field then a space and then the token generated from the API from the previous section. Bearer /Admin/Login Click on the header to lock in the token. Now you are all set. When you execute the GenerateBadge API again you will get a token (analogous to badge). Keep this token handy, since we require it in the next section. Also make sure to for now. pass ClearanceLevel as 1 How to Set Up the Agent Controller Create a new file called AgentController.cs with the below content: Microsoft.AspNetCore.Mvc; {
    [ ]
    [ ]
    [ ] : { { }
    }
} using namespace FBI.Controllers ApiController Route( ) "[controller]" Authorize(Policy = ) "Agent" public class AgentController ControllerBase ( ) public AgentController As you can see above, we are authorising the whole controller for Agent's access only. So even Admin won't be able to access the API's we are going to create. How to Access the Record's APIs Let's add the APIs to access both public and classified files. [ ]
[ ]
[ ] { OkObjectResult( );
}

[ ]
[ ]
[ ] { OkObjectResult( );
} HttpGet Route( ) "[action]" Authorize(Policy = ) "ClearanceLevel1" ActionResult<String> ( ) public AccessPublicFiles return new "Public Files Accessed" HttpGet Route( ) "[action]" Authorize(Policy = ) "ClearanceLevel2" ActionResult<String> ( ) public AccessClassifiedFiles return new "Classified Files Accessed" We have added the Authorize attribute's for both API's such that public files can be accessed by ClearanceLevel1 and classified files can be accessed by ClearanceLevel2. If you try to access these API's with the Admin token you will get 403 Forbidden error. So go ahead and click on the button again and click on . Then, get the token from the above step and paste in the field with as a prefix Authorize logout Bearer . Now when you access the API you will see response 200 with the message . But when you try the classified API you get the 403 Forbidden error. /Agent/AccessPublicFiles Public Files Accessed How to Change the Clearance Level Fast forward 3 years and our performance has been mind bogglingly good. Management has now decided to promote them to ClearanceLevel2. Agent's The goes to the and asks them to provide a token/badge with Clearance Level 2. Agent Admin The calls the API to generate their own token first. They then enter it in the dialog. Admin /Admin/Login Authorize The admin then calls the API with value 2 in the ClearanceLevel. This generates a new token/badge which they then hand over to . /Admin/GenerageBadge Agent The enters this token/badge in the dialog and when they now call they are pleased to see the result . Agent Authorize /Agent/AccessClassifiedFiles Classified Files Accessed Conclusion You can find the whole project on GitHub. here API security is extremely important and shouldn't be taken lightly, even if it's for internal use only. Setup Authentication and Authorisation and you are halfway there. There are other other security measures you can take against DDoS attacks, accepting API's from a particular IP or domain only, and so on. How did you like the article? What other security measures do you usually take? Any feedbacks or comments? You can checkout out more tutorials . here Previously published at https://arjavdave.com/2021/03/31/net-5-setup-authentication-and-authorisation/

Authentication And Authorization In .NET 5: How It Works

Three years ago I’ve to the world. As of today, after 81 releases it’s in the dependency graph of and has surpassed ! introduced lint-staged more than 55.000 GitHub repositories 500.000 weekly downloads on NPM I would never expect lint-staged to become such a popular tool when I was releasing it back in 2015. One issue that took me two years to fix Shortly after lint-staged was introduced to — a popular open source project by , I’ve got an issue report entitled react-boilerplate Max Stoiber “Add better support for partially staged files” This (yes, I can remember its number without looking it up). This issue has of comments and of different opinions varying from “huh, it’s as simple as doing …” to “that’s impossible to achieve”. It’s so long that it would not fit into one single PNG file because it would exceed 30.000px — maximum allowed image height (sic!). issue is number #62 lots lots Screenshot of the issue #62 from GitHub. https://github.com/okonet/lint-staged/issues/62 I tried fixing the issue several times without luck. Last closest attempt was about a year ago on the train to 2018 where I held a presentation about lint-staged. AgentConf Unfortunately, even though I spent almost a few days hacking on it I could not solve it. This work still had some positive outcomes: I meet a few people during the conference who encouraged and helped me to keep trying (thanks to , , and Nejc Zdovc). Juho Vepsäläinen Michel Weststrate I wrote lots of tests so it became easier to do TDD and try different implementation ideas (which I did a lot). 🎉 Introducing lint-staged v8 Today, after more than two years and I’m very excited to finally release version 8 of lint-staged that supports partially staged files! 138 commits It uses the following algorithm: Before running linters, check if there are partially stated files (i.e. both with changes in commit index and the working copy) Stash and remove from the working copy all files that aren’t in the commit index Execute commands (linters and formatters) on these files Add formatters changes to index (separate task in the config) Restore the stashed working copy state if all linters exit with zero code (i.e. success). If there are errors running linters, restore stashed working copy. Compare new commit index and working copy and . If there are conflicting hunks, drop formatter’s changes. User modifications that were stashed should always take precedence over formatters. apply as many formatting changes to working copy as possible And this is how it looks in action: _This is the demo of how partially staged files support works in 🚫💩 lint-staged. For more information see…_asciinema.org 🚫💩 lint-staged with support for partially staged files Most importantly, y_ou don’t have to change your config to get support for partially staged files to work!_ The only “breaking change” in this version is the requirement of Node 8.6. npm install --save-dev lint-staged@latest I’ve been using it for more than a month on all my projects and consider it stable. If you happen to encounter issues with it, please report them on GitHub and help me fix them. Takeaways Don’t give up on your ideas! People kept saying that the problem I’m trying to tackle is impossible to solve. Some part of me kept denying it… One thing I’ve learned is that it’s worth trying something if you believe into it. Open Source is ❤️ The final solution is based on comments and ideas shared in the and wouldn’t be possible without huge support and encouragement coming from people around the globe. I’d also like to thank contributors who help me maintain the project: and . comments to the original issue Suhas Karanth Lufty Wiranda Open Source is 💩 The fact it took me more than 2 years to find the solution is just ridiculous but it goes back to justifying this kind of work. I spent countless amount of hours looking for the final solution. It took me a few more days to finalize and clean up the code, merge changes from upstream, and write this blog post. Just in last 2 months I spent more than 50 hours on it. Dedicating this amount of time becomes harder and harder for me to justify, keeping in mind I’m a father of two. If you think lint-staged is useful please consider funding the project at Open Collective .

🎉 Announcing lint-staged with support for partially staged files

Wikipedia define pilgrimage as “a journey or search of moral” with its “shrine”. While aboard another flight, leaving Lisbon and the latest edition of the WebSummit, I would like to tell you some things about the magic happening behind whatever screen you’re reading this. There is an entire world there, moving from a country to another every day. Where is the person working in the digital industry when he is ? not too far away from it, obviously, and you’ll probably find him in that kind of places. Maybe you will read WebSummit is good or bad in some articles. They’re both probably right. If you want to know if it could be good for you, read until the end. away from the keyboard The crème de la crème If you have some plans to , this is my shortlist, a non exhaustive list of the best meetings all over the world. You can think of these events as planets with a lot of satellites gravitating around them, an event is only the tip of the iceberg. travel if you work in the content industry you can’t avoid it. Some of the best independent artists are coming there to make legendary concerts. SXSW for goods, with its controversial booth babe, and because it’s the first big thing that ever appeared in the field. Alive since 1967, a millenium at the scale of digital era.  Since 2010 there has been a new kid in town, the It has disrupted and disturbed the established conferences like a whirlwind. Only a few years after the first edition it is already worldwide with , , . Each event is a ton of conferences in one, but also more than 2000 of the best startups exposing. CES WebSummit. Collision Rise Surge We are waiting for the next who will revolutionarize this again. It will come sooner than you expect, because this world needs a revolution per day. Paddy Cosgrave Paddy Cosgrave Code Coding is the ability to talk to machines. Nowadays it’s probably more profitable to learn one of the thousands machine language than human one. We call ourselves programmers, coders, developers … and there is even more conferences for us. I often hear that we were the rock stars of this decade … Some of us have been on tour for several years. Some of the best developers events you can’t miss : , and of course. , , and events are unmissable too, because they shape the future of what we will be able to do. AgentConf Smashing Conf DEF CON Google Apple Amazon Github Startup culture Silicon Valley, TV show As you can see there are conferences almost every day around the world. And you thought geeks never left theirs caves, or their sect ? This cliché is one of the reasons many events and meetups are organised. We fear we would look like the caricature if we didn’t do these kind of efforts. Pitch, disrupt, pivot, unicorn, you know everything. is the center of everything coming in this industry. And there is often no solid base behind it. Only energy and inspirational talks. Belief from , last year at WebSummit A memorable talk Gary Vaynerchuk Networking The first place in the world ? Bloombergs as defined it as and Forbes as Not enough for you ? Take a look at the stats. networking « Davos for geeks » « The best tech conference on the planet ». 60 000 people sharing a lot in common 2,600 of the world’s leading media 1,400 of the world’s most influential tech investors 1,200 world-class speakers Party Nights are important, for networking of course, but not only. In that world, people work almost every hour they are awake and they need as much time as possible to chill out. And why not signing millions dollars contract in a paper towel in one of the famous bars of Dublin. Do you know Uber ? Money money money released the government estimated economic impact of this year WebSummit : to Portugal and more than half a billion since the last year edition. It’s huge for an event of Observador 300 M$ only three days-duration. It is impossible to value how much money represent people coming to a conference like WebSummit represent. There is a lot of CEOs from the largest companies of the tech industry. Hundreds of investors, business angels and venture capital. For sure billions of dollars are walking in the aisle. You don’t believe me, do you ? Just an exemple to show you should trust me on this. I had the chance to discuss with representatives of the largest telecom operator in Kazakhstan, and guess what ? they were here in Lisbon, with Kazakstan government with an envelop of 1 billion of dollars to invest in tech. 🤑 Kazakhtelecom , Trending topics AI If last year was the year of AR this year is definitely about AI, just one letter has changed, easy. presented its automated car, as did with UberAir automated flying cars. presented some promising tools like the engine behind Alexa available to anyone with an AWS account. Waymo Uber Amazon Lex Stephen Hawking opened the ceremony with some advice at the age of artificial intelligence. Blockchain BTC/USD Last years and blockchain was trending. This year was definitely a new step. Today Bitcoin is adopted by general public. At the same time, Ethereum facilitated the development of new application on top of the blockchain. Bitcoin Ethereum The dream of decentralized internet seems to be closer to reality every day. Politics One year ago today, Trump was elected. It was during the previous edition of the WebSummit. And one of the most interesting talks about the election came from Tinder with some crazy numbers. It’s clear that lobbying changed side. Digital revolution exposes huge (de)regulation problems and tech actors have lobbied around politics for years to open some closed market. A new era is coming, politics rushing to these events to attract best specialists and ideas to their country with a lot of advantages like attractive taxes, places to work … Paddy Cosgrave and Marcelo Rebelo de Sousa The pilgrims To put it simply, more suits, less nerds t-shirts. A few years ago, most of the exposants were young geeks and now there are more casual people. The exposants seem more professional, dreamers and eccentrics are harder to find today. One of the explanations of this phenomena is the explosion of fin-tech and cryptocurrencies, of the 21st century ? “The Street” Another notable evolution, for the better, more women were present. Thanks to a hard work from the organisation to bring more women to the tech events, they were greatly represented as well on stage and on the audience . (35.4%) (42%) Rosario Dawson and Sara Sampaio Meet us If you’re attending some events we could meet us, or if you just want to exchange about your nomad experience, ping me on . twitter

Pilgrimages of the tech world

It’s time to install a MongoDB ReplicaSet on a Kubernetes cluster on Azure and try to kill it in all possible ways!Starring: Helm StatefulSet PersistentVolumes PersistentVolumeClaims Azure Kubernetes Cluster configuration Let’s install a Cluster with 1 Master and 3 Nodes all running Linux using ACS Engine with following commands. I described detailed steps on ACS Engine usage for installing a cluster in my article. Please refer to it for details. Here I list quickly commands to be used. Kubernetes Adventures on Azure — Part 3 (ACS Engine & Hybrid Cluster) Creation of Resource Group This is needed to group all resources for this tutorial in a single logical group, to be able to delete everything with a single command at the end. az group create --name k8sMongoTestGroup --location westeurope Cluster provisioning I usually create an ssh pair for my test on acs. Please check my article on how to do it. I will change the examples/kubernetes.json file to use previously created ssh pair, dnsPrefix and servicePrincipalProfile ( ). here following Deploy a Kubernetes Cluster suggestions from Microsoft my kubernetes.json file with changes in is: bold {"apiVersion": "vlabs","properties": {"orchestratorProfile": {"orchestratorType": "Kubernetes","orchestratorRelease": "1.7"},"masterProfile": {"count": 1,"dnsPrefix": " ","vmSize": "Standard_D2_v2"},"agentPoolProfiles": [{"name": "agentpool1","count": 3,"vmSize": "Standard_D2_v2","availabilityProfile": "AvailabilitySet"}],"linuxProfile": {"adminUsername": "azureuser","ssh": {"publicKeys": [{"keyData": " "}]}},"servicePrincipalProfile": {"clientId": " ","secret": " "}}} ivank8stest ssh-rsa yourpubkeyhere yourappclientid yourappsecrete Create your cluster with: acs-engine deploy --subscription-id Wait for the cluster to be up running: INFO[0010] Starting ARM Deployment (k8sMongoGroup-2051810234). This will take some time…INFO[0651] Finished ARM Deployment (k8sMongoGroup-2051810234). Connect to it using kubeconfig file generated during deployment in _output folder. export KUBECONFIG=~/acs/acs-engine/_output/ /kubeconfig/kubeconfig.westeurope.json ivank8stest Following commands can be used to determine when cluster is ready: kubectl cluster-infokubectl get nodes NAME                        STATUS    AGE       VERSIONk8s-agentpool1-33584487-0   Ready     46m       v1.7.4k8s-agentpool1-33584487-1   Ready     46m       v1.7.4k8s-agentpool1-33584487-2   Ready     46m       v1.7.4k8s-master-33584487-0       Ready     46m       v1.7.4 Now you can open Kubernetes Dashboard if you want to use a UI to check you cluster status: and then open a browser at kubectl proxy http://127.0.0.1:8001/ui Helm MongoDB Charts Helm is the Package Manager for Kubernetes. It simplifies installation and maintenance of products and services like: MongoDB Redis RabbitMQ and many other We will use it to install and configure a MongoDB Replica Set. Helm installation Prerequisites, installation steps and details can be found in the article from Microsoft. Use Helm to deploy containers on a Kubernetes cluster With all prerequisites in place, Helm installation is as simple as running command: helm init --upgrade Clone charts repository Let’s clone charts repository to be able to examine and change MongoDB chart files before deploying everything on our cluster: git clone https://github.com/kubernetes/charts.git Now go in the folder. Here you will find all artifacts composing an Helm Chart. If needed you can change file to tailor installation based on your needs. For now let’s try a standard installation. /charts/stable/mongodb-replicaset values.yaml Run following command: and wait for following output: helm install . NAME:   foppish-angelfishLAST DEPLOYED: Sun Sep 10 20:42:42 2017NAMESPACE: defaultSTATUS: DEPLOYED RESOURCES:==> v1/ServiceNAME                                  CLUSTER-IP  EXTERNAL-IP  PORT(S)    AGEfoppish-angelfish-mongodb-replicaset  None        <none>       27017/TCP  5s ==> v1beta1/StatefulSetNAME                                  DESIRED  CURRENT  AGEfoppish-angelfish-mongodb-replicaset  3        1        5s ==> v1/ConfigMapNAME                                        DATA  AGEfoppish-angelfish-mongodb-replicaset        1     5sfoppish-angelfish-mongodb-replicaset-tests  1     5s NOTES:... DONE! MongoDB Replicaset is up and running! Helm is ultra easy and powerful! Helm is ultra easy and powerful! MongoDB installation test From output of helm install pick up the NAME of your release and use it in the following command: export RELEASE_NAME= foppish-angelfish Here we follow a different path from Helm Chart. Let’s open an interactive shell session with remote Mongo server! kubectl exec $RELEASE_NAME-mongodb-replicaset-0 --mongo --shell OUTPUT MongoDB shell version v3.4.8connecting to: mongodb://127.0.0.1:27017MongoDB server version: 3.4.8type "help" for help......A LOT OF WARNING (I will check these in a future post to clean them up, if possible)......rs0:PRIMARY> Who is the primary? In theory Pod 0 should be the master as you can see rom rs: > prompt. If this is not the case tun following command to find the Master: PRIMARY rs0:SECONDARY> db.isMaster().primary Take note of the primary Pod because we are going to kill it soon and connect to it using last kubectl exec used above. Failover testing We need to create some data to check persistence across failures. We are already connect to the mongo shell, creating a document and leaving the session is as simple as: rs0:PRIMARY> db.test.insert({key1: 'value1'})rs0:PRIMARY> exit Use following command to monitor changes in replica set: kubectl run --attach bbox --image=mongo:3.4 --restart=Never --env="RELEASE_NAME=$RELEASE_NAME" -- sh -c 'while true; do for i in 0 1 2; do echo $RELEASE_NAME-mongodb-replicaset-$i $(mongo --host=$RELEASE_NAME-mongodb-replicaset-$i.$RELEASE_NAME-mongodb-replicaset --eval="printjson(rs.isMaster())" | grep primary); sleep 1; done; done'; OUTPUTfoppish-angelfish-mongodb-replicaset-0 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",foppish-angelfish-mongodb-replicaset-1 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",foppish-angelfish-mongodb-replicaset-2 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",......... Kill the primary! Here it is: kubectl delete pod $RELEASE_NAME-mongob-replicaset-0 MongoDB will start an election and another Pod will become master: foppish-angelfish-mongodb-replicaset-1 “primary” : “foppish-angelfish-mongodb-replicaset-1.foppish-angelfish-mongodb- And in the meantime Kubernetes will immediately take corrective actions instantiating a new Pod 0. Kill’em all Now we have to simulate a real disaster: let’s kill all Pods and see the StatefulSet magically recreating everything with all data available. kubectl delete po -l "app=mongodb-replicaset,release=$RELEASE_NAME"kubectl get po --watch-only After few minutes our MongoDB replicaset will be back online and we can test it again to see if our data are still there. Final check and clean up Run following command to verify that key created is still there: kubectl exec $RELEASE_NAME-mongodb-replicaset-1 --mongo --eval=”rs.slaveOk(); db.test.find({key1:{\$exists:true}}).forEach(printjson)” As always you can delete everything with a simple Azure CLI 2 command: az group delete --name k8sMongoTestGroup --yes --no-wait How to expose this replicaSet externally? This is a topic for a e post. It seems trivial as a but it is not so easy. If you expose the existing service, it will be LoadBalanced on 3 nodes behind it and this is wrong. We need 3 LoadBalancers, exposing 3 services, 1 for each Pod in the StatefulSet. Moreover we have to activate Authentication and SSL to ensure best practice from perspective. futur kubectl expose security I will find best way to do it, while playing with Heml, Kubernetes, MongoDB and Azure!

Stories, News & Mentions about AGENTNOON

Agentnoon’s Stories on HackerNoon