LetSync

letsync

If you are familiar with Stripe, you know how big of a player they are in the online payment processing field.  Their API not only allows programmers to easily create one-time payments for sites such as e-commerce stores, but also provides quick integrations for monthly subscriptions and routing payouts.  If new to Django and Stripe, check out our recent on integrating one time payments.  Otherwise, let's get into setting up monthly payments with Django and Stripe. article Why a Monthly Subscription? Monthly subscriptions are a common practice found across the web especially among companies who promote software-as-a-service (SAAS) as their delivery model.  For example, SAAS companies such as Hubspot (marketing), Dropbox (data storage), and Mailchimp (email marketing) all offer tiered pricing options to their prospective customers.  Many consider this model to be favorable given that revenue is easier to predict once basic metrics are calculated (customer acquisition cost, lifetime value, churn rate). Predictable revenue of course creates stability and produces more accurate financial forecasts. Mailchimp pricing page Django Setup Start by setting up a virtual environment and creating a basic Django project.  We'll create a new virtual environment called . saas on Mac, use instead of for all commands Note: python3 py C:\Users\Owner\Desktop\code>py -m venv saas Next change directory into the virtual environment, install Django, and setup your project and app. C:\Users\Owner\Desktop\code>cd saas

C:\Users\Owner\Desktop\code\saas>Scripts\activate

(saas) C:\Users\Owner\Desktop\code\saas>pip install Django

(saas) C:\Users\Owner\Desktop\code\saas>django-admin startproject mysite

(saas) C:\Users\Owner\Desktop\code\saas\mysite>py manage.py startapp main Add main app to the in . settings.py mysite settings.py INSTALLED_APPS = [ , , , , , , ,
] 'main.apps.MainConfig' #add this 'django.contrib.admin' 'django.contrib.auth' 'django.contrib.contenttypes' 'django.contrib.sessions' 'django.contrib.messages' 'django.contrib.staticfiles' Create in the folder and include in . urls.py main mysite > urls.py mysite > urls.py django.contrib admin django.urls path, include urlpatterns = [
	path( , include( )), path( , admin.site.urls),
] from import from import #add include '' 'main.urls' #add path 'admin/' Stripe Integration Start by installing the official library for connecting to Stripe's API. pip install --upgrade stripe Next create a Stripe account and create products in their dashboard.  While this can be performed using Stripe CLI's we'll use the dashboard since we are already on the page from creating an account.  Make sure you are in test mode and can only view test data.  By default, you should be in test mode after creating an account.  Click products in the left-hand side navigation menu.  Create two new products: Name the product Premium Plan Add description "paid plan for advanced features" Use standard pricing Enter $15.00 and ensure "recurring" is selected Keep billing period as "monthly" Name the product Enterprise Plan Add description "enterprise plan for advanced features" Use standard pricing Enter $30.00 and ensure "recurring" is selected Keep billing period as "monthly" Now let's sync the product in the Stripe dashboard.  While we could create a model and store the relevant product information, such as as product id, as model fields, an easier solution is to simply install the and use the sync command.  We also add need to add our API keys in .  Note your live keys should always be secured and never listed in the settings.  For more information on securing environment variables, check out Python securing. dj-stripe package settings.py pip install dj-stripe mysite/settings.py STRIPE_TEST_PUBLIC_KEY = STRIPE_TEST_SECRET_KEY = STRIPE_LIVE_MODE = DJSTRIPE_WEBHOOK_SECRET = 'pk_test_E52Nh0gTCRpJ7h4JhuEX7BIO006LVew6GG' 'sk_test_L87kx7GQNbz9tajOluDts7da00mSbze3dW' False # Change to True in production "whsec_xxx" Finally migrate the database. Note: if database is sqlite, migrating make take longer than usual. py manage.py migrate Add products to database automatically with the following command: py manage.py djstripe_sync_plans_from_stripe View the admin page to see the changes we just made.  First create an admin user. Include an email such as since we'll use this as the Stripe customer name then visit .   You should see a variety of new models including our newly created products as well as plans for the products. test@example.com http://127.0.0.1:8000/admin/ py manage.py createsuperuser Checkout Page Now that we have our data synced, let's create a checkout page where users can select their plan and checkout. We'll start with the view and checkout template. Note: we have imported the Bootstrap CDN in home.html: views.py django.shortcuts render, redirect stripe json django.http JsonResponse djstripe.models Product django.contrib.auth.decorators login_required render(request, ) products = Product.objects.all() render(request, ,{ : products}) from import import import from import from import from import # Create your views here. : def homepage (request) return "home.html" @login_required : def checkout (request) return "checkout.html" "products" checkout.html {% extends "home.html" %}

{% block content %} {% for p in products %} {{p.name}} {{p.description}} {% for plan in p.plan_set.all %} {{ plan.human_readable_price }} {% endfor %} {% endfor %} Checkout Enter card details.  Your subscription will start immediately Plan: Total: Loading... Subscribe {% endblock %} < = > script src "https://js.stripe.com/v3/" </ > script < > br < > br < = > div class "container " < = > div class "row " < = > div class "col-6" < = = > div class "card mx-5 shadow" style "border-radius: 10px; border:none; " < = > div class "card-body" < = > h5 class "card-title font-weight-bold" </ > h5 < = > p class "card-text text-muted" < = = = = = = > svg class "bi bi-check" width "1em" height "1em" viewBox "0 0 16 16" fill "currentColor" xmlns "http://www.w3.org/2000/svg" < = = /> path fill-rule "evenodd" d "M10.97 4.97a.75.75 0 0 1 1.071 1.05l-3.992 4.99a.75.75 0 0 1-1.08.02L4.324 8.384a.75.75 0 1 1 1.06-1.06l2.094 2.093 3.473-4.425a.236.236 0 0 1 .02-.022z" </ > svg </ > p < > h5 </ > h5 < = > div class "text-right" < = = = = > input type "checkbox" name "{{p.name}}" value "{{p.id}}" onclick "planSelect('{{p.name}}' ,'{{plan.human_readable_price}}', '{{plan.id}}')" </ > div </ > div </ > div </ > div </ > div < > br < > br < > hr < > br < > br < > div < = > div class "row" < = > div class "col-12" < = = > div class "card mx-5 shadow rounded" style "border-radius:50px;border:none" < = > div class "card-body" < = > h5 class "card-title font-weight-bold" </ > h5 < = > p class "text-muted " </ > p < = > div class "row" < = > div class "col-6 text-muted" < > p </ > p < > p </ > p </ > div < = > div class "col-6 text-right" < = > p id "plan" </ > p < = > p id "price" </ > p < = > p hidden id "priceId" </ > p </ > div </ > div < > br < = > form id "subscription-form" < = = > div id "card-element" class "MyCardElement"  </ > div  < = = > div id "card-errors" role "alert" </ > div < = = > button id "submit" type "submit" < = = = > div class "spinner-border  spinner-border-sm text-light hidden" id "spinner" role "status" < = > span class "sr-only" </ > span </ > div < = > span id "button-text" </ > span </ > button </ > form </ > div </ > div </ > div </ > div </ > div </ > div Now that we have a template for selecting the subscription plan. Let's add some styling and setup the credit card form by using Stripe Elements, a pre-built set of UI components. < > {


    .StripeElement { : border-box; : ; : ; : solid transparent; : ; : white; : ; : box-shadow ease; : box-shadow ease;
    } { : ;
    } { : ;
    } { : ;
    } { : none;
    } { : (120%);
    } { : ; : ; : ; : ; : ; : ; : ; : -apple-system,BlinkMacSystemFont,sans-serif; : ; : ; : normal; : break-word; : border-box; : inherit; : visible; : button; : antialiased; : ; : inherit; : transparent; : ; : ; : ; : none; : none; : none; : ; : ; : (--radius); : ; : ; : ; : ; : pointer; : all . ease; : block; : (0,0,0,.07); : ; : (--button-color);
    }

</ > style body box-sizing height 40px padding 10px 12px border 1px border-radius 4px background-color box-shadow 0 1px 3px 0 #e6ebf1 -webkit-transition 150ms transition 150ms .StripeElement--focus box-shadow 0 1px 3px 0 #cfd7df .StripeElement--invalid border-color #fa755a .StripeElement--webkit-autofill background-color #fefde5 !important .hidden display #submit :hover filter contrast #submit font-feature-settings "pnum" --body-color #f7fafc --button-color #556cd6 --accent-color #556cd6 --gray-border #e3e8ee --link-color #fff --font-color #697386 --body-font-family --radius 4px --form-width 400px -webkit-box-direction word-wrap box-sizing font overflow -webkit-appearance -webkit-font-smoothing margin 0 font-family -webkit-tap-highlight-color font-size 16px padding 0 12px line-height 32px outline text-decoration text-transform margin-right 8px height 36px border-radius var color #fff border 0 margin-top 16px font-weight 600 cursor transition 2s display box-shadow 0 4px 5.5px 0 rgba width 100% background var style .getElementById( ).disabled = ;

stripeElements(); {
  stripe = Stripe( ); ( .getElementById( )) { elements = stripe.elements(); style = { : { : , : , : , : , : { : }
    	}, : { : , : }
    };


    card = elements.create( , { : style });

    card.mount( );

    card.on( , { el = .getElementById( );
      el.classList.add( );
    });

    card.on( , { el = .getElementById( );
      el.classList.remove( );
    });

    card.on( , {
      displayError(event);
    });
  } } { displayError = .getElementById( ); (event.error) {
    displayError.textContent = event.error.message;
  } {
    displayError.textContent = ;
  }
} document "submit" true ( ) function stripeElements 'pk_test_E52Nh0gTCRpJ7h4JhuEX7BIO006LVew6GG' if document 'card-element' let // Card Element styles let base color "#32325d" fontFamily '"Helvetica Neue", Helvetica, sans-serif' fontSmoothing "antialiased" fontSize "16px" "::placeholder" color "#aab7c4" invalid color "#fa755a" iconColor "#fa755a" 'card' style '#card-element' 'focus' ( ) function let document 'card-errors' 'focused' 'blur' ( ) function let document 'card-errors' 'focused' 'change' ( ) function event //we'll add payment form handling here ( ) function displayError event let document 'card-errors' if else '' This should allow the credit card form to render.  First, we disabled the subscribe button and then called to create and then mount the credit card form.  Next, we  add a small script to update the template with the selected subscription plan and to re-enable the subscribe button if a plan is selected. stripeElements() { inputs = .getElementsByTagName( ); ( i = ; i<inputs.length; i++){
			inputs[i].checked = ; (inputs[i].name== name){

				inputs[i].checked = ;
			}
		} n = .getElementById( ); p = .getElementById( ); pid = .getElementById( );
		n.innerHTML = name;
		p.innerHTML = price;
		pid.innerHTML = priceId; .getElementById( ).disabled = ;


	} ( ) function planSelect name, price, priceId var document 'input' for var 0 false if true var document 'plan' var document 'price' var document 'priceId' document "submit" false Now we need to handle the payment form submission. We'll add the first code block to the end of the function.  First grab the form by its ID and add an event listener.  After preventing the default form submission, we change the loading state of the form in order to prevent any issues associated with double clicking the subscribe button.  Finally, with the inputted card information, we create a payment method and submit this data to our server along with the price id of the selected subscription. stripeElements() As demonstrated, we use Django's CSRF token to authorize the submission to our server. Note: Stripe's documentation includes some additional validation that you should check out if interested.  Also, unlike the documentation, we create a customer and subscription in one request to the server instead of sending separate requests. paymentForm = .getElementById( ); (paymentForm) {

		paymentForm.addEventListener( , {
			evt.preventDefault();
			changeLoadingState( ); createPaymentMethod({ card });
	  });
	}

} { billingName = ;

  stripe
    .createPaymentMethod({ : , : card, : { : billingName,
      },
    })
    .then( { (result.error) {
        displayError(result);
      } { paymentParams = { : .getElementById( ).innerHTML, : result.paymentMethod.id,
      };
      fetch( , { : , : { : , : ,
        }, : , : .stringify(paymentParams),
      }).then( { response.json(); 
      }).then( { (result.error) { result;
        } result;
      }).then( { (result && result.status === ) { .location.href = ;
      	};
      }).catch( {
          displayError(result.error.message);

      });
      }
    });
} changeLoadingState = { (isLoading) { .getElementById( ).disabled = ; .querySelector( ).classList.remove( ); .querySelector( ).classList.add( );
	} { .getElementById( ).disabled = ; .querySelector( ).classList.add( ); .querySelector( ).classList.remove( );
	}
}; //we'll add payment form handling here let document 'subscription-form' if 'submit' ( ) function evt true // create new payment method & create subscription ( ) function createPaymentMethod { card } // Set up payment method for recurring usage let '{{user.username}}' type 'card' card billing_details name ( ) => result if else const price_id document "priceId" payment_method "/create-sub" method 'POST' headers 'Content-Type' 'application/json' 'X-CSRFToken' '{{ csrf_token }}' credentials 'same-origin' body JSON ( ) => response return ( ) => result if // The card had an error when trying to attach it to a customer throw return ( ) => result if 'active' window '/complete' ( ) function error var ( ) function isLoading if document "submit" true document "#spinner" "hidden" document "#button-text" "hidden" else document "submit" false document "#spinner" "hidden" document "#button-text" "hidden" Add url paths for creating a subscription/customer and when payment is complete. main/urls.py django.urls path . views

app_name = urlpatterns = [
	path( , views.homepage, name= ),
	path( , views.checkout, name= ),
	path( , views.logout_request, name= ),
	path( , views.login_request, name= ),
	path( , views.register, name= ),
	path( , views.create_sub, name= ), path( , views.complete, name= ), ] from import from import "main" "" "homepage" "checkout" "checkout" "logout" "logout_request" "login" "logout_request" "register" "register" "create-sub" "create sub" #add "complete" "complete" #add Next, create a view to handle creating a Stripe customer and subscription.  The customer and subscription should also be stored in our local database.  Luckily, we can easily syn our data using dj-stripe. main/views.py ... stripe json django.http JsonResponse djstripe.models Product django.contrib.auth.decorators login_required djstripe django.http HttpResponse

... request.method == : data = json.loads(request.body)
	    payment_method = data[ ]
	    stripe.api_key = djstripe.settings.STRIPE_SECRET_KEY

	    payment_method_obj = stripe.PaymentMethod.retrieve(payment_method)
	    djstripe.models.PaymentMethod.sync_from_stripe_data(payment_method_obj) : customer = stripe.Customer.create(
	            payment_method=payment_method,
	            email=request.user.email,
	            invoice_settings={ : payment_method
	            }
	        )

	        djstripe_customer = djstripe.models.Customer.sync_from_stripe_data(customer)
	        request.user.customer = djstripe_customer subscription = stripe.Subscription.create(
	            customer=customer.id,
	            items=[
	                { : data[ ],
	                },
	            ],
	            expand=[ ]
	        )

	        djstripe_subscription = djstripe.models.Subscription.sync_from_stripe_data(subscription)

	        request.user.subscription = djstripe_subscription
	        request.user.save() JsonResponse(subscription) Exception e: JsonResponse({ : (e.args[ ])}, status = ) : HTTPresponse( ) import import from import from import from import import from import @login_required : def create_sub (request) if 'POST' # Reads application/json and returns a response 'payment_method' try # This creates a new Customer and attaches the PaymentMethod in one API call. 'default_payment_method' # At this point, associate the ID of the Customer object with your # own internal representation of a customer, if you have one. # print(customer) # Subscribe the user to the subscription created "price" "price_id" "latest_invoice.payment_intent" return except as return 'error' 0 403 else return 'requet method not allowed' Also add a view function for the payment complete page: render(request, ) : def complete (request) return "complete.html" Go through and test the integration.  As suggested by Stripe, use "4242 4242 4242 4242" as the test credit card and view the results upon submitting your payment.  Click on "Subscriptions" under "Customers" in your Stripe dashboard to view the newly created subscription and customer.  You should see something similar as below: Conclusion Thanks for reading about using Django and Stripe to create monthly subscriptions. Hopefully this provides a foundation for creating your own SAAS project.  We plan on writing additional pieces on Django and Stripe to cover topics such as routing payments with Stripe Connect, the underlying payment technology behind Lyft, Postmates, Kickstarter, and more. Also if you have any suggestions on other Django integrations that you'd like to read about, leave a comment below. Previously published at https://www.ordinarycoders.com/blog/article/django-stripe-monthly-subscription

Setting up Subscriptions And  Recurring Payments Using Django and Stripe

Continue reading in our blog You’ve started a web project. Congrats! This means you (or your colleague) will inevitably become a Product Owner: even if the title is not spoken aloud, you’ll have to do the work. This post is our message of advice: from us (Developers) to You, our future App Holder (no matter if you’re professional or beginner). Why are we writing this post? To make life easier for both of us: Product Owners and Product Creators. To break the wall between people who know what to do and people who know how to do it. For you to enjoy developing your app with us, we’re sharing this priceless advice. It accumulates the pain and wisdom of years of work. “Objection! Nothing valuable comes for free!” you may say. Right. We have our interest: to let Product Owners know . This will make our life happier. the critical attitude we expect from them Some basic notions Before we get to the core, let’s sync our understandings of a Product Holder role in Agile development process. Otherwise, we might be talking about different people and responsibilities. That’s the first secret unveiled: a Product Owner a Project Manager! is absolutely not Who is a Product Owner? The person who knows Product the best, believes in it, and drives others. A Product Owner is a person with the Product Vision. It means knowing: what the Product will do (with smallest details) what problems will it solve, and for whom Whenever there’s a question about the Product — a Product Owner always has an answer. At the same time, a Product Owner is a who: Leader Sees the direction for building the Product Knows the result to be achieved Motivates the team with belief, drive, and enthusiasm As a consequence of that total product knowledge, a Product Owner acts like a Proxy (Connector) between the Stakeholders and Developers, coordinating Product wishes and capacities of the Development Team. What does a Product Owner do? becomes the Knowledge Keeper for the Product. What’s vital: knowing and are equally important! Product Vision: what the product is what it’s not The instruments for putting down this Vision are (the plan with the business goals and technologies, used for achieving them) and (the list of Products, sold/developed by a company, often — in different stages of Scrum development/production). Product Roadmap Product Pipeline gathers new features the future Product must have, maintains and prioritizes that knowledge base. Product Backlog: The core competence here is estimating the of these functionalities, and thus prioritize the Backlog, answering the question Business Value “What gives us more business value with less efforts/expenses?” writes documents that explain to Developers: User Stories: Why the new feature is needed, and Expectations after the feature is implemented (these may appear in ) Acceptance Criteria The document is usually a detailed one and contains usage examples, screenshots, mockups, and other things that make explanations easier. We’ll cover the art of writing cool user stories in a separate blog post. Sprint and Release Planning: Decides which features must be done in the next development cycle ( ), and thus — which functionalities have to be postponed and done later. Sprint Make sure these features are fully functional in the last stable version of the Product (Release). participates in: Coordination of Development Team’s Work: Working meetings with Developers ( ) Stand Up/Scrum meetings Discussions of new user stories for the upcoming sprint ( ) Sprint Grooming Follow-up discussions on what was good and went wrong during the completed sprint ( ). Retrospective Meeting connects Developers and Stakeholders on all possible issues. This is exactly where the famous motto of , , is applied the best. Collaboration: Agile Manifesto “Individuals and interactions over processes and tools” Although not absolutely complete, this list includes the key responsibilities of a Product Owner. So what’s the advice?! BE AVAILABLE … to answer questions Developers always have questions. Hundreds of them. The sad graph of interdependence states: the less details a User Story has, the more questions arise during the work. Surprisingly, it’s good for you: questions show that Developers care about your product, trying to foresee problems and make it better. This leads to a ground-breaking notion. Remember: Developers are on your side! Make you teammates sure you’re always near to support them with all the info you have. Then, they’ll be sure to be moving in a right direction, and all the words in the world won’t express how important that is. Try to answer questions as soon as you can. … to discuss Developer’s think and plan before they start — it’s a part of their mentality. Thus, they may foresee issues before they appear. Being open for discussions and proposals may save days (!) of work and billions of nerve cells. … to give feedback Say what you don’t like. Share concerns. Ask questions yourself. Anything to let Developers know how do you feel about their work. Web development is a two-way road. As a family, it needs feedback from both sides. And a small wish: when you like the work results, don’t forget to say it. It is a great motivator. … to solve issues Issues, misunderstandings, problems are OK. It’s a normal (and inevitable) part of development. The only man who never makes a mistake is the man who never does anything. (Theodore Roosevelt). When there’s an issue, face it with your team. The earlier you address it, the less problems it will cause. Again, remember: your teammates are there to back you up in any difficult case. The trick is that Developers always try their best to help, offering variants and solutions. But they will never know the product as well as you do. So, their solutions won’t be the most optimal ones. That brings us to another valuable notion: Make sure you’re always involved. Otherwise, you’ll get . Do not delegate too much responsibilities to your Developers. their app, not yours … to plan work Try to drag as many features to the new sprint as Developers can handle — based on previous sprint experience. Don’t let lots of tasks stuck and demotivate your team. Be ready to say “No” to other Stakeholders who insist on adding as much features as possible. is key value here. Prioritization … to trust Developers in technical details This is particularly important with Sprint and Release Planning. For example: sometimes it may prove not productive to start working on some tasks — when they are not connected with others logically. Writing code for these features will be difficult. Then, it’s better to follow Developers’ recommendations and fill the sprints with tasks they find related. Such tasks will be implemented consistently, step-by-step, and the resulting code will be optimal and well-structured (which is critical for further scalability). … to make decisions on the Product This is the toughest skill. You cannot be a natural born decision maker. But you can learn. And, if you’re sure about your product idea and have a good team at your back, you can do it. Optimistic conclusion One of our QA Engineers, a person with enormous customer experience, once said: “… In a worst case, a Product Owner may have no serious experience with web apps or software development in general. The critical thing is to be open for communication. Sooner or later, we’ll get all the right questions to ask. Having the answers, we will get us through the entire work, and finally have a great result”. More useful articles you can find in our blog Primary source

Web Development: the Most Precious Advice to a Product Owner

AWS LetsEncrypt Lambda or Why I Wrote a Custom TLS Provider for AWS Using OpenTofu and Go

AWS LetsEncrypt Lambda বা কেন আমি OpenTofu এবং Go ব্যবহার করে AWS এর জন্য একটি কাস্টম TLS প্রদানকারী লিখেছি

AWS LetsEncrypt Lambda oder warum ich mit OpenTofu und Go einen benutzerdefinierten TLS-Anbieter für AWS geschrieben habe

AWS LetsEncrypt Lambda o por qué escribí un proveedor TLS personalizado para AWS usando OpenTofu and Go

AWS LetsEncrypt Lambda ou pourquoi j'ai écrit un fournisseur TLS personnalisé pour AWS à l'aide d'OpenTofu and Go

AWS LetsEncrypt लैम्ब्डा या मैंने OpenTofu और Go का उपयोग करके AWS के लिए एक कस्टम TLS प्रदाता क्यों लिखा

AWS LetsEncrypt Lambda または OpenTofu と Go を使用して AWS 用のカスタム TLS プロバイダーを作成した理由

AWS LetsEncrypt Lambda 또는 OpenTofu 및 Go를 사용하여 AWS용 사용자 지정 TLS 공급자를 작성한 이유

AWS LetsEncrypt Lambda ou por que escrevi um provedor TLS personalizado para AWS usando OpenTofu and Go

AWS LetsEncrypt Lambda или почему я написал собственный TLS-провайдер для AWS с использованием OpenTofu и Go

AWS LetsEncrypt Lambda veya OpenTofu ve Go'yu Kullanarak AWS için Neden Özel bir TLS Sağlayıcısı Yazdım?

AWS LetsEncrypt Lambda hoặc Tại sao tôi viết Nhà cung cấp TLS tùy chỉnh cho AWS bằng OpenTofu và Go

AWS LetsEncrypt Lambda 或为什么我使用 OpenTofu 和 Go 为 AWS 编写自定义 TLS 提供程序

RSA vs ECC comparison. Issuing LetsEncrypt certificates using certbot and acme.sh clients wrapped in Docker image. Nginx setup Table of contents RSA vs ECC comparison. Why you might need ECDSA certificate? How to Generate RSA and EC keys/CSR using . openssl What is ? How to issue free domain validated certificates in automatic fashion? LetsEncrypt CA How to generate RSA and/or ECDSA certificates through while still using and clients under the hood? Docker image certbot acme.sh How to configure and test Nginx for hybrid RSA/ECDSA setup? RSA vs ECC comparison RSA is a most popular public-key cryptography algorithm. Certificates with RSA keys are the gold standard and the present of the current Internet PKI . It’s old and battle tested , and that’s highly important from the security perspective. Elliptic curve cryptography is an alternative approach to public-key cryptography over the current RSA standard. security technology RSA algorithm can be used for encryption and digital signing, while ECC can only be used for signing. The security of a key depends on its size and its algorithm. Some algorithms are easier to break than others. Breaking an RSA key requires to factor the product of two large numbers. Breaking an ECC key requires to find the discrete logarithm between points on an elliptic curve, and there is no progress so far to achieve this. Algorithms require different key size for the same level of security. ECC can use smaller key sizes. Here is table with key size comparison: +----------------------+-----------------+--------------------+| Symmetric Key length |  RSA key length |   ECC key length   |+----------------------+---------- ------+--------------------+| 80                   | 1024            | 160                || 112                  | 2048            | 224                || 128                  | 3072            | 256                || 192                  | 7680            | 384                || 256                  | 15360           | 512                |+----------------------+-----------------+--------------------+ For example, 256-bit ECC key is equivalent to RSA 3072-bit key, providing 128 bits of security: Smaller keys are less computationally intensive for generating signatures because the math involves smaller numbers. However, while ECC is faster at signature generation, it’s slower than RSA at signature verification. Let’s measure this using openssl as a benchmarking tool: $ openssl speed ecdsap256 rsa2048 sign/s     verify/s rsa 2048 bits             679.0      23489.0256 bit ecdsa (nistp256)  15581.9    6211.7 Smaller ECC public key means smaller certificate size — less data to pass around, quicker to download, and . faster TLS handshake If you want more security, RSA does not scale well — you have to increase the RSA modulus size far faster than the ECDSA curve size. 1024 bit RSA keys are obsolete, 2048 are the current standard size. If you need to go farther, you’d stuck. First, if CA does not provide 4096 bit RSA keychain, signing your own 4096 bit RSA key with a 2048 RSA intermediary doesn’t make sense. Second, note that every doubling of an RSA private key degrades TLS handshake performance approximately by 6–7 times. So, if you need more security, choose ECC. While ECC has some benefits, there are also some drawbacks. The technology is not that mature and tested as RSA. And there is a compatibility question, however seems it can be used on most operating systems and modern browsers. Check out these resources for . ECC compatibility details There a lot of elliptic curves out there. Most popular and supported by major browsers are P-256, P-384, P-521, x25519. While P-256 and P-384 are part of algorithms, P-521 and x25519 are not. Google Chrome has , regarding NSS/Firefox. To maximise interoperability with existing browsers and servers, stick to P-256 and P-384 curves. NIST’s Suite B dropped support for the P-521 curve same is going on prime256v1 secp384r1 To view list of all available ECC curves, that OpenSSL library supports: openssl ecparam -list_curves To view supported curves of your browser, use . SSL Labs Client Test To conclude, ECDSA certificates pros and cons: (+) smaller key and certificate size, faster TLS handshake (+) from server perspective, requires less CPU and memory better performance (+) stronger algorithm, harder to break (+) scales better when you need more security (-) not as mature and time-tested as RSA, ECC is relatively new compared to RSA (-) lack of compatibility and lack of widespread support. (-) ECC is faster and generating signatures, but signature verification is computationally intensive and slower that RSA You don’t need to choose between RSA and ECC exclusively. You can setup hybrid configuration, serving ECDSA certificate first, with a fallback to RSA certificate for non-supporting clients. Generate RSA and ECC keys/CSRs using openssl Usually, before you send a request to a CA to issue a certificate, you need to generate private key and CSR (certificate signing request). It’s super easy with tool. openssl Generate RSA key at a given length: openssl genrsa -out example.key 2048 Generate EC key with a given curve: openssl ecparam -genkey -name secp384r1 | openssl ec -out ec.key Generate a CSR from existing private key with a given subject info: openssl req -new  -key example.key -out example.csr -subj "/CN=example.com" -sha256 Or you can combine both key and CSR creation within a single command: openssl req -nodes -newkey rsa:2048 -keyout example.key -out example.csr -sha256 LetsEncrypt CA If you want to experiment with ECDSA and RSA certificates, the best option is to use , which allows to generate free domain validated certificates in automated fashion. LetsEncrypt Certificate Authority In order to obtain a certificate, you need to prove the ownership of the domain. Usually this process requires manual work: generate a private key and a CSR (Certificate Signing Request) with relevant subject info and common name, send a CSR to the CA, and finally prove the domain’s ownership using a selected challenge method: ; CA sends email with a verification link one of the address like admin@example.com. You need to follow the link to complete the verification. email you need to download a file with a specially crafted token and then upload it to your’s domain root folder, so it’s accessible at URI http; http://example.com/.well-known/pki-validation you must create a special CNAME record in the DNS configuration for your domain dns; LetsEncrypt automates this process by using a client that can talk (Automatic Certificate Management Environment). Client typically runs on your web host, and communicates to LetsEncrypt CA or another ACME-compatible server. Client receives unique token from the server, generates key from it, starts a standalone web server listening on port 80 and serves key at special URI, like . In case of DNS challenge method, client can automatically add CNAME record to your DNS configuration depending on DNS server/provider. LetsEncrypt CA then makes HTTP or DNS request to your domain to retrieve the key derived from the token. Successful response proves the domain ownership, and CA issues the requested certificate. ACME protocol http://example.com/.well-known/acme-challenge You don’t generate private key and CSR on your own, this is handled by the client software on your web host. Note, that private keys are never generated/leaked on CA servers, and stored on your web host exclusively. Clients typically provide some way to adjust some settings through client, like key length, subject alternative names, or if you need further customisation, you can fallback to providing custom CSR for more flexibility and control. Note, that Lets Encrypt’s certificates are only valid for 90 days. This is to encourage users to automate certificate issuance and renewal process. Rate limits and staging server Be aware, that LetsEncrypt CA production servers put strict : rate limits certificates per Registered Domain (20 per week) up to 100 alternative names per certificate duplicate certificate limit of 5 certificates per week While you’re trying and experimenting, most likely you will hit the latter one, so it’s better to use LetsEncrypt with much relaxed limits. staging environment The staging environment intermediate certificate (“ ”) is issued by a root certificate not present in browser/client trust stores, so you might want to add as trusted one while testing. Fake LE Intermediate X1 “Fake LE Root X1” Generate LetsEncrypt certificates using Docker image There are variety of ACME clients: , , , and . I’m not going to dive into client specifics here — anyway docs would explain better. certbot acme.sh lego others Personally, I’ve found that easier and faster way to get things done is to issue certs through some Docker image. This is for lazy people, and those, who don’t want to spend much time digging with LetsEncrypt and parsing docs. I’ve prepare one: on Docker Hub and . This Docker image provides a simple single entrypoint to obtain and manage SSL certificates from LetsEncrypt CA. It encapsulates two popular ACME clients: and , which are used to obtain RSA and/or ECDSA certificates respectively. We need both, because certbot is not capable of issuing ECDSA certificates (to be more correct, only thru custom CSR, but then you lose the ability to renew, revoke and further manage such certificate). asamoshkin/letsencrypt-certgen Github link certbot acme.sh Here is an example of issuing both ECDSA (prime256v1 curve) and RSA (2048) certificates for the single domain foobbz.site docker run \  -v /var/ssl:/var/ssl \  -p 80:80 \  -e DOMAINS=foobbz.site \  --rm \  asamoshkin/letsencrypt-certgen issue The requirement is to run this image on the server, configured for the domain you want to get certificates for (where your DNS A record points to). Server’s port 80 should be open by a firewall, so LetsEncrypt CA server can perform validation challenge. Once done, certificates, keys and related files are stored at path, on a volume you’ve mounted into the container before. /var/ssl/$domain_common_name /var/ssl # tree /var/ssl/var/ssl└── foobbz.site    ├── certs    │   ├── cert.ecc.pem    │   ├── cert.rsa.pem    │   ├── chain.ecc.pem    │   ├── chain.rsa.pem    │   ├── fullchain.ecc.pem    │   └── fullchain.rsa.pem    └── private        ├── privkey.ecc.pem        └── privkey.rsa.pem All files are encoded in PEM format: , - generated certificates (RSA or ECDSA) cert.rsa.pem cert.ecc.pem - chain of intermediate CA certificates (e.g. Fake LE Intermediate X1) chain.[type].pem - certificate bundled with any intermediate CA certificates. This is suitable for Nginx directive , which requires a bundle, instead of leaf certificate. fullchain.[type].pem ssl_certificate - private key file privkey.[type].pem You’re not limited to a single certificate for a single domain. You can issue several certificates for several domains, or single certificate covering multiple domains using SAN (X.509 subject alternative names extension). You can opt to generate ECDSA or RSA certificate only with custom key length or elliptic curve. First, prepare a file with domains lists. Each line represents individual certificate to be issued. First name within each line is a common name, whereas subsequent comma-separated names are certificate alternative names (SAN). # cat /root/domains.txtfoobbz.site,www.foobbz.site,web.foobbz.sitefoobbz2.site,www.foobbz.site Run a command to issue two ECDSA “secp384r1” certs only (no RSA cert) for and domains: foobbz.site foobbz2.site docker run \  -v /var/ssl:/var/ssl \  -v /root/domains.txt:/etc/domains.txt \  -p 80:80 \  -e RSA_ENABLED=0 \  -e ECDSA_KEY_LENGTH=ec-384 \  -e DOMAINS=/etc/domains.txt \  --rm \  asamoshkin/letsencrypt-certgen issue The main use case for the image is to trigger a one-shot command to issue certificates, but you can also further manage your certificates by renewing, revoking or deleting them using same image. You can use this image ad-hoc at a build time, at a run-time prior to Nginx/Apache startup, or by running it from a cron job to renew certificates on regular basis. The idea is that LetsEncrypt stuff is encapsulated within a single container , and you don’t need to pollute your Nginx/Apache container. More complete list of features: automate issuance and managing LetsEncrypt SSL certificates generate DV certificates for 1..N domains, support multi-domain SAN (Subject alternative names) certificates generate RSA and/or ECDSA certificate with configurable key params: RSA key length (2048, 3072, 4096) and elliptic curve for EC key (prime256v1, secp384r1) choose DV challenge verification method: standalone or webroot renew certificates when they’re about to expire or force renewal revoke certificates by contacting LetsEncrypt CA use either LetsEncrypt staging or production server For further details and examples, check out on Github. You can also just check out the source code to learn how to consume and clients. samoshkin/docker-letsencrypt-certgen certbot acme.sh Also, there is a worth checking out. ZeroSSL docker image Nginx configuration for hybrid RSA/ECDSA setup Once you’ve generated certificates, it’s time to configure your web server. Nginx allows hybrid side by side RSA and ECDSA certificates, and will serve one or another during TLS handshake depending on agreed cipher suite. server {listen 443 ssl default_server_;_  server_name foobbz.site www.foobbz.site_;_ ssl_certificate /var/ssl/foobbz.site/certs/fullchain.rsa.pem_;_ssl_certificate_key /var/ssl/foobbz.site/private/privkey.rsa.pem_;_ # RSA certificates ssl_certificate /var/ssl/foobbz.site/certs/fullchain.ecc.pem_;_ssl_certificate_key /var/ssl/foobbz.site/private/privkey.ecc.pem_; # ECDSA certificates Other directives }_ Also you need to tell Nginx to prefer ECDSA over RSA for authentication by correctly ordering your cipher suites (notice how suites with “ECDSA” go first, followed by suites with “aRSA”): ssl_prefer_server_ciphers on;ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:DHE+AESGCM:DHE:!RSA!aNULL:!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!CAMELLIA:!SEED"; That’s it, now start your Nginx, and let’s test our configuration using openssl as a TLS client and extract certificate. Or you can use to view certificates. Qualys SSL Labs Server Test $ openssl s_client -host foobbz.site -port 443 -cipher ECDHE-ECDSA-AES128-GCM-SHA256 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' | openssl x509 -noout -text Looking at a public key, we can ensure that ECDSA certificate is served. Subject Public Key Info:Public Key Algorithm: id-ecPublicKeyPublic-Key: (256 bit)pub:04:47:ee:86:9f:c9:5a:81:16:89:38:f7:5b:9d:ba:6b:8b:d2:aa:8e:1e:54:67:f7:f1:44:84:5f:10:df:7f:df:16:f4:2f:d6:c1:78:b8:71:68:e9:ee:78:82:fc:2e:ae:96:e9:a3:b7:26:c0:ed:41:39:2a:48:f9:0f:28:10:4e:15ASN1 OID: prime256v1NIST CURVE: P-256 Note, that leaf ECDSA certificates are still signed by LetsEncrypt’s RSA certificate chain (Let’s Encrypt Authority X3, DST Root CA X3). LetsEncrypt does not use dedicated EC certificates to sign to build complete EC chain. 0 s:/CN=foobbz.sitei:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X31 s:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3i:/O=Digital Signature Trust Co./CN=DST Root CA X3 Now, let’s use another cipher suite (ECDHE-RSA-AES128-GCM-SHA256) to test RSA configuration. openssl s_client -host foobbz.site -port 443 -cipher ECDHE-RSA-AES128-GCM-SHA256 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' | openssl x509 -noout -text We get 2048 bits RSA public key. By the way, notice the difference in length between them. Subject Public Key Info:Public Key Algorithm: rsaEncryptionPublic-Key: (2048 bit)Modulus:00:ea:cb:5e:22:dd:93:fe:63:21:c4:bd:b9:07:78:fc:ef:66:38:7a:19:bf:68:58:39:16:86:ee:f6:2c:18:68:9c:32:c0:5b:a4:76:e8:e0:40:0e:6d:29:7c:bc:04:67:b4:1b:05:e8:72:53:24:dc:4f:a6:3d:48:41:2e:83:99:fa:13:20:88:b8:e5:5d:34:57:01:c6:eb:fc:c1:67:e8:e4:ec:58:2c:a2:ce:51:ea:99:c0:bb🇩🇪61:8a:40:76:80:50:48:25:c6:7f:0e:a4:a6:61:e7:25:67:b1:74:ee:1f:f1:75:e8:76:a0:c5:5d:9c:40:48:8b:d3:95:e1:27:d2:d6:ca:14:e4:39:ac:4d:0d:35:23:89:db:4b:ef:60:84:0b:4d:15:76:0e:3c:f5:52:1c:20:ce:d8:03:25:22:7a:37:84:fb:d2:1b:00:ff:31:69:55:65:7d:42:d1:31:99:0c:d6:29:41:36:06:bf:0d🆎31:1a:e6:b0:6a:76:67:2c:7b:c0:5b:34:55:49:e2:4c:d9:e4:40:99:1c:c1:1d:6a:88:c1:53:af:ee🆎b5:2e:e6:76:ff:1c:33:e2:ca:7e:d7:93:e6:23:df💿78:a6:39:f4:04:a2:44:d0:a6:cc:f1:51:2f:5d:dc:5e:ea:ff:57:d7:f1:82:d4:48:11Exponent: 65537 (0x10001) So, that’s it. Thank you for reading this post. Edit #1 that Docker container does not have enough entropy to generate key material. One solution I’m aware of is mounting from host machine into of the container. See this . Reddit user wuunderbar pointed out /dev/urandom /dev/random StackOverflow answer docker run -v /dev/urandom:/dev/random ... Resources asamoshkin/letsencrypt-certgen — Docker Hub — https://hub.docker.com/r/asamoshkin/letsencrypt-certgen/ samoshkin/docker-letsencrypt-certgen: Generate, renew, revoke RSA and/or ECDSA SSL certificates from LetsEncrypt CA using certbot and acme.sh clients in automated fashion — https://github.com/samoshkin/docker-letsencrypt-certgen Elliptic Curve Cryptography (ECC Certificates) | DigiCert.com — https://www.digicert.com/ecc.htm ECC — https://support.globalsign.com/customer/portal/articles/1994347-ecc ECC Compatibility — https://support.globalsign.com/customer/portal/articles/1995283-ecc-compatibility ECDSA: The digital signature algorithm of a better internet — https://blog.cloudflare.com/ecdsa-the-digital-signature-algorithm-of-a-better-internet/ Let’s Encrypt — Free SSL/TLS Certificates — https://letsencrypt.org/ Certbot — https://certbot.eff.org/ ZeroSSL: Free SSL — https://zerossl.com/ Neilpang/acme.sh: A pure Unix shell script implementing ACME client protocol — https://github.com/Neilpang/acme.sh RSA and ECDSA performance | securitypitfalls — https://securitypitfalls.wordpress.com/2014/10/06/rsa-and-ecdsa-performance/ Testing out ECDSA certificates — https://scotthelme.co.uk/ecdsa-certificates/

RSA and ECDSA hybrid Nginx setup with LetsEncrypt certificates

People don’t understand SSL/TLS There is a new . So lots of started screaming because they read phishing and in the same sentence. report that associates Let’s Encrypt with phishing people in Twitter Paypal Let’s is a foundation that gives certificates away for free. Not only that. Let’s Encrypt is trying to make installing and renewing a certificate effortless. Encrypt I decided to read the report which is more a blog post. Apparently the argument of the report is that Let’s Encrypt doesn’t do a proper validation before giving away the certificates. So many sites use the services in what appear to be Paypal phishing sites. But the report doesn’t say that domain name sellers are not doing proper validation before selling a phishing domain. But here comes the twist. The blog post appears in a site called . hashedout Under hashedout’s icon you can see in small letters “by The SSL store”. It turns out that the blog is from a company that . sells certificates A company that sells certificates creates a report criticising a foundation that gives certificates away for free. Yes. I know that they are different kind of certificates. In fact the argument is that the certificates they sell are given only after a proper validation. Anyway, what kind of certificates do they sell? Cheap!!! Yeah, a lot of Symantec, Thawte and GeoTrust. The same Symantec, Thawte and GeoTrust that have been (including extended validation certificates, those that have a name in green too) without proper validation. incorrectly emitting certificates The same Symantec, Thawte and GeoTrust that by at least a major browser. are going to be distrusted There is a serious problem with PKI. But guys, . you have a nerve

LetsEncrypt, Paypal and selling fear

In Kubernetes, Ingress resources are frequently used as traffic controllers, providing external access to services within the cluster. Ingress is essential for routing incoming traffic to your service; however, there may be scenarios in which you want to prevent search engines from indexing your service's content: it might be a development environment or something else. This blog post will walk you through the process of blocking your site's indexing on Kubernetes Ingress using a robots.txt file, preventing search engine bots from crawling and indexing your content. Prerequisites To proceed with the tutorial, you should have a basic grasp of Kubernetes basic objects, Ingress resources, and the official HAProxy ingress controller. You will also need access to the Kubernetes cluster and the necessary permissions to make configuration changes. Keep in mind that for this article, I assume that the HAProxy ingress controller is set as the default controller. Otherwise, if you did not select HAProxy as the default controller, you must add the ingressClassName option to all Ingress code examples. Step 1: Create an Ingress Kubernetes Resource In the first part of our journey, we'll set up a small Ingress resource to expose our service outside of the Kubernetes cluster. Pay attention: for the time being, all web crawlers will have access to the service. To apply the code below, use the command kubectl apply -f ingress.yaml. # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example Step 2: Modify the Ingress Configuration The robots.txt file is used to control how search engines index documents. Its file specifies which URLs search engine crawlers can access on your website. The most basic file that restricts access to the web service looks like this: User-agent: *
Disallow: / HAProxy does not require you to add this file to your web server or website. This can be achieved with the following configuration, which should be added to the backend section for the specific group of servers: acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path K8S Annotations regulate all manipulations of the HAProxy frontend/backend configuration for a single ingress resource. The full list of HAProxy annotations can be found in the official documentation on GitHub. In our case, we need to use haproxy.org/backend-config-snippet with the HAProxy snippet for blocking any indexing. To do this, edit, open your Ingress resource YAML file, and add the following annotation to the metadata section: #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... Step 3: Apply the Configuration Changes After changing the Ingress YAML file, save it and apply it to the Kubernetes cluster with the kubectl command: kubectl apply -f ingress.yaml. The Ingress controller will detect the changes and update the configuration accordingly. Step 4: Verify the Configuration Inspect the generated robots.txt file to confirm that indexing prevention is working properly. The Ingress controller generates this file based on the annotation you supply. Retrieve the external IP or domain associated with your Ingress resource and add /robots.txt to the URL. Example: $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / As we can see, the answer contains a robots.txt file that prevents any search indexing. Step 5: Test Indexing Prevention To verify that search engines are not indexing your site, you can run a search for your website on popular search engines. Keep in mind that search results sometimes take some time to reflect changes, so the indexing status may not be fully updated right away. Conclusion Annotations make it easy to avoid search engine indexing while using HAProxy Kubernetes Ingress. By adding the appropriate annotation to your Ingress resource, you can prohibit search engine bots from crawling and indexing your website's content. A similar approach can be used with other ingress controllers, such as Nginx, Traefic, and others. A similar annotation can also be used for K8S Gateway API resources, which are actively replacing Ingresses. As a final note, robots.txt is a time-honored way for website creators to specify whether or not their sites should be crawled by various bots. However, it turns out that AI crawlers from large language model (LLM) companies frequently ignore the contents of robots.txt and crawl your site regardless. To avoid such situations, utilize password security, noindex, or enterprise load balancer features like HAProxy AI-crawler, which may be also configured as a K8S annotation. In Kubernetes, Ingress resources are frequently used as traffic controllers, providing external access to services within the cluster. Ingress is essential for routing incoming traffic to your service; however, there may be scenarios in which you want to prevent search engines from indexing your service's content: it might be a development environment or something else. This blog post will walk you through the process of blocking your site's indexing on Kubernetes Ingress using a robots.txt file, preventing search engine bots from crawling and indexing your content. Prerequisites To proceed with the tutorial, you should have a basic grasp of Kubernetes basic objects, Ingress resources, and the official HAProxy ingress controller . You will also need access to the Kubernetes cluster and the necessary permissions to make configuration changes. Kubernetes official HAProxy ingress controller Keep in mind that for this article, I assume that the HAProxy ingress controller is set as the default controller. Otherwise, if you did not select HAProxy as the default controller, you must add the ingressClassName option to all Ingress code examples. ingressClassName Step 1: Create an Ingress Kubernetes Resource In the first part of our journey, we'll set up a small Ingress resource to expose our service outside of the Kubernetes cluster. Pay attention: for the time being, all web crawlers will have access to the service. To apply the code below, use the command kubectl apply -f ingress.yaml . kubectl apply -f ingress.yaml # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example Step 2: Modify the Ingress Configuration The robots.txt file is used to control how search engines index documents. Its file specifies which URLs search engine crawlers can access on your website. robots.txt The most basic file that restricts access to the web service looks like this: User-agent: *
Disallow: / User-agent: *
Disallow: / HAProxy does not require you to add this file to your web server or website. This can be achieved with the following configuration, which should be added to the backend section for the specific group of servers: acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path K8S Annotations regulate all manipulations of the HAProxy frontend/backend configuration for a single ingress resource. The full list of HAProxy annotations can be found in the official documentation on GitHub . in the official documentation on GitHub In our case, we need to use haproxy.org/backend-config-snippet with the HAProxy snippet for blocking any indexing. To do this, edit, open your Ingress resource YAML file, and add the following annotation to the metadata section: haproxy.org/backend-config-snippet #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... Step 3: Apply the Configuration Changes After changing the Ingress YAML file, save it and apply it to the Kubernetes cluster with the kubectl command: kubectl apply -f ingress.yaml . The Ingress controller will detect the changes and update the configuration accordingly. kubectl apply -f ingress.yaml Step 4: Verify the Configuration Inspect the generated robots.txt file to confirm that indexing prevention is working properly. The Ingress controller generates this file based on the annotation you supply. Retrieve the external IP or domain associated with your Ingress resource and add /robots.txt to the URL. Example: $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / As we can see, the answer contains a robots.txt file that prevents any search indexing. robots.txt Step 5: Test Indexing Prevention To verify that search engines are not indexing your site, you can run a search for your website on popular search engines. Keep in mind that search results sometimes take some time to reflect changes, so the indexing status may not be fully updated right away. Conclusion Annotations make it easy to avoid search engine indexing while using HAProxy Kubernetes Ingress. By adding the appropriate annotation to your Ingress resource, you can prohibit search engine bots from crawling and indexing your website's content. A similar approach can be used with other ingress controllers, such as Nginx, Traefic, and others. A similar annotation can also be used for K8S Gateway API resources, which are actively replacing Ingresses. K8S Gateway API As a final note, robots.txt is a time-honored way for website creators to specify whether or not their sites should be crawled by various bots. However, it turns out that AI crawlers from large language model (LLM) companies frequently ignore the contents of robots.txt and crawl your site regardless. To avoid such situations, utilize password security, noindex, or enterprise load balancer features like HAProxy AI-crawler , which may be also configured as a K8S annotation. HAProxy AI-crawler

How to Block Search Engine Indexing in Kubernetes with HAProxy

Wazuh on Docker: Secure Your Dashboard with Let's Encrypt and Deploy Agent with Kubernetes Daemonset

How to Streamline CI/CD With Octopus and TeamCity

অক্টোপাস এবং টিমসিটির সাথে কীভাবে সিআই/সিডি স্ট্রিমলাইন করবেন

So optimieren Sie CI/CD mit Octopus und TeamCity

Cómo optimizar CI/CD con Octopus y TeamCity

Comment rationaliser le CI/CD avec Octopus et TeamCity

ऑक्टोपस और टीमसिटी के साथ CI/CD को कैसे सुव्यवस्थित करें

Octopus と TeamCity で CI/CD を効率化する方法

Octopus와 TeamCity를 사용하여 CI/CD를 간소화하는 방법

Como simplificar CI/CD com Octopus e TeamCity

Как оптимизировать CI/CD с помощью Octopus и TeamCity

Octopus ve TeamCity ile CI/CD Nasıl Kolaylaştırılır

Cách hợp lý hóa CI/CD với Octopus và TeamCity

如何使用 Octopus 和 TeamCity 简化 CI/CD

is a distributed unit of infrastructure. It can provide information about FreeBSD and Linux operating systems. It also supports metrics transmission to Clickhouse, InfluxDB, and Prometheus. Alligator The main interface of alligator is . It connects with spreading Prometheus as a monitoring system in many infrastructures. However, the Prometheus ecosystem has some problems. OpenMetrics First of all - the specialized and simplified agents. Every service on the host needs a specialized exporter. This is the reason for having a lot of exporters per host (like a node_exporter, redis_exporter, sentinel_exporter, and so on). Secondly, simplified agents. A lot of agents can be used for different tasks (like an SNMP, blackbox, etc.), and it inflates infrastructure for such daemons to collect information. Sometimes, a pull model has problems with multi-worker applications and cron jobs - nothing can give metrics from hosts when Prometheus makes a request, or it is connected with problems related to synchronizing results between workers. It forces us to use statsd/pushgateway. Then the question comes up: how to remove outdated metrics? How to make it a high-availability service? The approach of the alligator is different - it is a smart agent for the Prometheus. Alligator consists of a system exporter and an application exporter. It also provides a push model for pushgateway, statsd, and graphite emulation protocol. Metrics Export Alligator supports such sets of metrics: FreeBSD and GNU/Linux system metrics. Filesystem metrics (file’s checksum, sizes, and access level). Certificate metrics (PEM and P12). Some popular applications (ElasticSearch, uwsgi, Druid, RabbitMQ, and others). Ephemeral metrics. It supports pushgateway, statsd, and graphite protocol, and it can collect metrics from file on FS. It also has a cluster mode for centralized installation of pushgateway. Simple JSON to metrics parser. Simple HTTP/TCP checker. Use Cases A part of supported technologies for collecting metrics are reviewed in the . The rest I am describing here. official documentation Application Server Metrics First of all, alligator can collect application server metrics. For instance, alligator can gather metrics from uwsgi instances with this configuration: aggregate {
    uwsgi tcp://localhost:1717;
} and rerun the uwsgi server with the : expose stats port --stats 127.0.0.1:1717 Another example, fastcgi-application: aggregate {
    php-fpm fastcgi://127.0.0.1:9000/stats?json&full;
} and write this option in : php-fpm config pm.status_path = /stats After this, you will have common metrics from the application server. Application Metrics I recommend using synchronization structures to collect metrics in the application internally. Alternatively, you can make only 1 worker per instance and use such an abstraction as orchestration (Kubernetes, opennebula, and others). And next, use alligator as a push server for metrics. Alligator has many collectors to collect it: pushgateway prom-aggregation-gateway statsd graphite file These approaches ensure that metrics will be collected from all workers of the application Applications have to send metrics to alligator. To remove influence on one metric by different workers, you can use two different approaches: mark metric with the worker ID from prometheus_client import Counter, push_to_gateway, CollectorRegistry
    registry = CollectorRegistry()
    c = Counter('my_requests_total', 'HTTP Failures', ['method', 'worker_id'], registry=registry)
    c.labels(method='get', worker_id=uwsgi.worker_id()).inc()
    c.labels(method='post', worker_id=uwsgi.worker_id()).inc()
    push_to_gateway('localhost:1111', job='myapp', registry=registry) use instead of metric type if you can. File collector cannot calculate increment, but others can increment if the counter is a type of metric. You can only send increment value to alligator, and alligator will calculate the sum. Counter Gauge More information about the difference between metric types can be found or on the . here official page What Metrics Do I Recommend to Measure in Applications? Firstly, it is any outgoing requests to external systems. In worker applications, it often is the reason to overflow workers or cause of low speed of query processing. My recommended list of metrics: count of requests to external services grouped by request types request time from external services (avg, or if possible, by quantiles/buckets of time) grouped by request types status of requests grouped by request type (success or errors) It would be nice if you could group not only by request type but more by network layer of requests: DNS resolve count, time, and status connect count, time, and status write/read count, time, and status It is also important to measure internal application metrics: ingoing requests grouped by types processing ingoing request time avg (quantile or buckets if possible) by type status of ingoing request by type cache status (hit/miss if it exists) I guess you can make another list of metrics, but these metrics are the most important to reveal problems and give main observability about application health. If your application is working in podman or docker environment, you can expose the container statistics. Alligator mechanics are similar to Cadvisor's when collecting container statistics. To enable, you have to use option cadvisor in the system context: Container checks. system {
    cadvisor;
} . This is a good way to monitor applications or external sources for reliability. Let’s check the resources on two ports every minute: Blackbox checks aggregate_period 60;
aggregate {
    blackbox http://127.0.0.1:80/check;
    blackbox https://127.0.0.1:443/check;
} Alligator produces network and application response metrics for this resource (response code, response time, and certificate expiration date). If the application has a page with JSON, alligator can also convert JSON to metrics: aggregate {
    jsonparse https://jsonplaceholder.typicode.com/todos;
} The next level of blackbox metrics - checking the process, service, and listening port: system {
	services uwsgi.service;
	process uwsgi;
}


query {
	expr 'count by (src_port, process) (socket_stat{process="uwsgi", src_port="80"})';
	make socket_match;
	datasource internal;
} And in the end: launching external programs and proactive actions (like watchdog): aggregate {
	process 'exec:///usr/local/bin/mycheckscript.sh';
} If the script can produce a metric that can determine the exact application downtime, this metric can be used for proactively restart: query {
    expr 'myapplication_status < 1';
    make app_is_down;
    datasource internal;
    action app_is_down;
}
action {
    name app_is_down;
    type shell;
    expr 'exec://systemctl restart uwsgi;
    datasource internal;
} System Metrics Any application metrics are also important. But without system metrics of servers, you won’t be able to clearly understand what happens in your system. System metrics are a set of metrics that help us gain insight into the health of our systems. This is the basis for making decisions if something goes wrong. An alligator with an empty configuration doesn’t collect any metrics at all. Any system metrics are classified into the following categories: base - CPU, memory, time, logical resources, and other common metrics about the server disk - disk usage and I/O network - interface statistics, network counters, sockets statistics smart - collecting S.M.A.R.T. statistics cpuavg - analog for load average when alligator make by CPU-only load firewall - counters from the system firewall process - check process (uptime, ulimits usage, status, CPU, memory, and disk consumption) packages - metrics with a list of installed packets in the system services - systemd services stats This is configured in a block called “system” in alligator.conf: system {
    base;
    disk;
    network;
    process;
    cpuavg period=5;
} What about cron jobs? How can it be connected to the Prometheus via alligator? There are several ways to do this. The simple way is to output metrics to a file: aggregate {
    prometheus_metrics file:///var/lib/stats/metrics.txt state=save;
} The parameter called ‘state’ enables autosaving at a point of the file when alligator is stopped. It makes us sure that we can stop the alligator and run again, and it rereads only that part of the file that has not been read yet. If we want to reread a file from the start (every 10 sec), we have to use . The third way - is to read only realtime data. To enable this mode, we have to remove any state from the configuration file. state=begin The other way is to use statsd or extended pushgateway protocol (called prom-aggregation-gateway): It only needs to have this section in the configuration: entrypoint {
	tcp 1111;
    metric_aggregation count;
} For instance, we have a code: #!/bin/sh
TASKS=0
do_something() {
        echo 0
        TASKS=$1
}

res=`do_something 1`
echo "amount_of_done_tasks {method=\"do_something\", status=\"done\"} $TASKS
task_exit_code {method=\"do_something\"} $res" | curl --data-binary @- http://localhost:1111/ This script will make two metrics in alligator: amount_of_done_tasks {status="done", method="do_something"} 0.000000
task_exit_code {method="do_something"} 0.000000 Sometimes, we might get a situation when the script is not working, but the metric exists (because alligator doesn’t know that the application doesn't work). To solve this problem, alligator has TTL mechanics. You can specify the lifetime in seconds for a request: curl -H "X-Expire-Time: 30" --data "living_thirty_seconds_metric 4" http://localhost:1111/ Or in the configuration file: entrypoint {
    ttl 30;
    tcp 1111;
    metric_aggregation count;
} The next way is using statsd protocol (udp or tcp). TCP server for statsd is enabled by default. If you want to use udp, specify the UDP port in config: entrypoint {
        ttl 30;
        tcp 1111;
        udp 1111;
} And your job script might have a section like this: from statsd import StatsClient, TCPStatsClient
from time import sleep

statsd = StatsClient(host='localhost',
                     port=1111,
                     prefix=None,
                     ipv6=False)

statsd.incr('baz')
statsd.incr('baz') Graphite protocol is also supported by alligator, and it works like graphite_exporter. Internet Web Server If I have an internet web server, what options do I have? First of all, https is the most popular protocol for web servers today. It forces us to check the certificate expiration time: To check directories with certificates: x509
{
	name nginx;
	path /etc/nginx/ssl/;
	match .crt;
}
x509
{
	name letsencrypt;
	path /etc/letsencrypt/live;
	match .pem;
} Secondly, to check the nginx listening ports: query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="80"})';
	make socket_match;
	datasource internal;
}
query {
	expr 'count by (src_port, process) (socket_stat{process="nginx", src_port="443"})';
	make socket_match;
	datasource internal;
} To check nginx -t to exclude the situation, when nginx after server reboot cannot be started: aggregate {
	process 'exec:///sbin/nginx -t';
} To check system, web-server process, and firewall counters system {
    base;
    disk;
    network;
    cpuavg period=5;
    services nginx.service;
    process nginx;
    firewall ipset=on;
} We also make alligator port accessible only on local interfaces: entrypoint {
    handler prometheus;
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    tcp 1111;
} or you can use the notation tcp in the tcp directive. <localIP>:<port> Statsd Metric Mapping Simple statsd doesn’t work with labels. To create a label from a statsd metric name, you must use mapping with metric name templates to change new names and create new labels. Due to this, I recommend using an extended format of Statsd, like DogStatsd (alligator also supports this extension), which supports labels in statsd protocol. But, some programs have already made pure statsd protocol without labels. One such example is Airflow. You can catch metrics from Airflow like this: airflow.ti.finish.upload_job.load_data.success:1|c
airflow.ti.finish.upload_job.other_data.success:1|c where the name is the following template: airflow.ti.finish.<job name>.<process name>:<increment by>|type The mapping for such metric name can be: entrypoint {
    tcp 1111;
    udp 1111;
    mapping {
        template   airflow.ti.finish.*.*.*;
        label dag_id "$1;
        label task_id "$2";
        label state "$3";
        name airflow_ti_finish;
        match glob;
    }
} This mapping will create such a metric: airflow_ti_finish {state="success", dag_id="upload_job", task_id="load_data"} 1.000000
airflow_ti_finish {state="success", dag_id="upload_job", task_id="other_data"} 1.000000 Statsd or Pushgateway Cluster Firstly, you can only make some amount of servers statsd or pushgateway and simply proxying to them. The first problem with using this: if you want to use gauges, you will have problems with calculating. If there is a distributed application sending a metric with the same name and labels, the gauge metric will also be replaced by the application. The first way to fix this - is to specify different labels for each application (such as hostname and/or worker_id/process_id). But that is not all. You also have a problem with the same gauge metric: this metric can be a balancer on different machines. Here is what to do with a metric that was sent at different times, but now they are interfering with each other: 10:12 http_request_time_avg {appid="43"} 12
10:14 http_request_time_avg {appid="43"} 22
10:55 http_request_time_avg {appid="43"} 122
11:58 http_request_time_avg {appid="43"} 1232 You can use nginx upstreams with a balancing method called hash: upstream alligators {
    server srv1.example.com:1112;
    server srv2.example.com:1112;
    server srv3.example.com:1112;
    server srv4.example.com:1112;
    server srv5.example.com:1112;
    hash $remote_addr consistent;
} Then one backend will always push all metrics to one server. On the other hand - cluster mode in alligator. Let's set it up! Firstly, we allocated 3 ports: for getting metrics 1111 for replication works 1112 for getting metrics from other applications 80 Imagine that our server is listening to the internet. Then, we need to secure our push metric ports. We can use the directive “return” with arg “empty” to remove any answers outside. We also know that internet browsers don't send POST queries to other domains if CORS doesn't allow this. Thirdly, make an authentication for POST requests. entrypoint {
   allow 127.0.0.1;
   allow 10.0.0.0/8;
   tcp 1111;
}
entrypoint {
   allow 10.0.0.0/8;
   tcp 1112;
   cluster replication;
   instance srv1.example.com:1112;
}
entrypoint {
   return empty;
   allow 0.0.0.0/0;
   ttl 300;
   tcp 80;
   metric_aggregation count;
   auth basic user:setme;
   auth_header Authorization;
   header access-control-allow-headers 'Authorization';
   header access-control-allow-methods POST;
   header access-control-allow-origin *;
} That's good. We annotate that we have a cluster called “replication” and the current instance name. Since this moment, alligator allows to answer collected oplog to anyone from network 10.0.0.0/8 on port 1112. How to command an alligator to join other instances of cluster? For this, we will use the “cluster” directive. We can specify oplog size (the limit of in-memory stored metrics that will wait to sync to other instances), and distribute metrics by its metric name (sharding key in our case, but you can use any other label). The replication factor hides replicated metrics while another primary server is running in the cluster. If the primary server goes down, hidden metrics will be exposed and made available to Prometheus from other instances. cluster {
   name replication;
   size 10000;
   sharding_key __name__;
   replica_factor 2;
   type oplog;
   servers srv1.example.com:1112 srv2.example.com:1112 srv3.example.com:1112 srv4.example.com:1112 srv5.example.com:1112;
} Check the External Sites Status Blackbox mechanics for collecting information about network services have already been mentioned. Nevertheless, that is not all that alligator can be used for. Alligator also has a puppeteer parser for loading external pages with a lot of site resources (which can load a page DOM). puppeteer {
    https://site1.example.com;
    https://site2.example.com;
    https://example.com;
} They might measure domain lookup name in metric: puppeteer_domainLookupEnd - puppeteer_domainLookupStart Connect time: puppeteer_connectEnd - puppeteer_connectStart Time to load DOM: puppeteer_DomContentLoaded Status of loading: puppeteer_eventSourceResponseStatus And even errors in the browser console: puppeteer_eventConsole This is not all that a puppeteer collector can do. However, this engine needs nodejs to be installed on a machine with puppeteer dependencies. Install puppeteer dependency: cd /var/lib/alligator/
npm i puppeteer@14.3.0
npm i ps-node-promise-es6 Conclusion Monitoring systems is a really big challenge despite the amount of tools to solve it. You can choose between a monitoring system or tools in one system of monitoring (such as VictoriaMetrics, Thanos, some amount of exporters, and so on). I guess that the approaches outlined in the article can be applicable to any other tools of monitoring. Alligator is a tool that can simplify it. The main goal of any monitoring is to make your system observable. You must have all measured information from your systems to ensure that it always operates correctly.

Alligator Is a Prometheus Monitoring Agent: Everything You Need to Know About It

How to Use Terraform to Configure AWS Verified Access With an OIDC Google Provider

Stories, News & Mentions about LETSYNC

Letsync’s Stories on HackerNoon