OpenSwap

openswap

It’s been almost a decade since Docker and containers changed the landscape of modern software delivery. Early in their evolution, there was around container security because they didn’t have a lot of the familiar architectural security elements when compared to VMs. Today, mature tooling and processes along with a widespread understanding of the risks and mitigation have persuaded many holdouts to adopt containers as another tool in their infrastructure strategies. a lot of fear This blog takes an updated look at the most mature and popular tools in the various stages of the container hardening process, giving you 7 strategies and related tools to try as you head into 2023. The strategies covered are: Container Image Creation Minimal Containers Container Image Scanning Policy Enforcement Container Encryption Container Signing Host Operating System Vulnerability Scanning 1.  Container Image Creation Container creation requires an understanding of the software packages, users, and environment settings that are necessary to deploy a workload. Container builds use a manifest of instructions for building containers, which are read and executed sequentially to generate a new container. A variety of tools can produce containers: Dockerfile Buildah Packer Packer is a favorite of mine and I’ll use it as an example. It builds containers using the . An advantage of using Packer is having a common workflow for building operating system as well as containers. Packer also has a " " capability. The post-processor creates a framework for container tagging, encrypting, signing, and container scanning. Packer can then push to artifact repositories after the build and post-processing steps. Docker builder plugin golden images post-processor Once the build step is complete, a container is pushed to an approved repository. To push to approved repositories, users should have an authentication and authorization step to grant access to a container repository. The credentials for accessing container repositories can then be stored in a . secrets manager 2. Minimal Containers A security benefit of using containers is having a limited number of operating system packages inside a container. Containers with a precise set of software packages decrease the number of possible security vulnerabilities in a container. In the last few years, there has been an increase in the number of minimal Linux distributions for containerization, and you may want to consider adopting some of them if you’re looking for more ways to improve your container security posture. A few of the container images that use very minimal operating systems with just enough packages installed to run include: Alpine Ubuntu Minimal Chainguard’s Wolfi Another way to limit the size of containers and the number of unnecessary software packages is to do . A container that serves a Golang REST API would have two stages in the multi-stage build: multi-stage builds The primary stage would be labeled as the compile stage. The compile stage needs an operating environment with enough packages to build the Go binary. The second stage would use a minimal base operating system and the Go binary created in the compile stage. Minimal containers, whether they’re ready out of the box or homegrown via multi-stage builds, minimize your threat surface because there are fewer packages and operating system components to exploit. 3. Container Image Scanning Scanning container images is a critical security step in the container lifecycle. You can run and enforce image scans at different stages of the software delivery lifecycle. Scans can execute… Before an image is pushed to a repository In the image repository Before running in an orchestration environment like Kubernetes There are many popular scanning tools to choose from. Some of the popular options include: Prisma Cloud Snyk (GitLab's ) Trivy default container scanner Grype If you want to scan a container before an image is pushed to a repository, an image build tool like Packer can do this during the build process, scanning images and checking for container vulnerabilities. If you want to scan a container when it’s in the image repository, tools like and are able to scan a container repository and monitor security drift as containers become stale and new common vulnerability and exposure reports are released on containers. Sysdig Jfrog Scanning tools can also continuously monitor web feeds for new vulnerabilities and send alerts when an image is out of compliance. At the enterprise level, there are tools that go even further and automate base-image upgrades when they find vulnerabilities. , the most commonly used infrastructure as a code provisioning tool can integrate with container scanning tools to ensure that images have been scanned before being deployed onto specific architecture. And if your company runs the commercial versions of Terraform, they can use native or third-party tools and guardrails to automatically scan and enforce rules on container images before running in an orchestration environment  (e.g. Kubernetes, Docker Swarm, and Nomad). Terraform 4. Policy Enforcement Policy checks are a great way to enforce security standards on a system as part of the software delivery lifecycle.  In order to add a check before allowing a service to be run in Kubernetes, secure environments require additional gate checks called . admission controllers The admission controller can scan containers for vulnerabilities and check for compliance at runtime. These gate checks should be codified with (PaC). Just like infrastructure as code, policies can be tracked, version controlled, and reused like modules or functions in programming languages. Example policies could include: policy as code Validation of container signatures for Software Bill of Materials Enforcing approved container registry whitelists Container scan severity thresholds - (Prohibit the deployment of services that have high severity CVEs in packages) Enforce the use of necessary tags or labels on services Ensuring container applications are not running as the root user Look for a policy as code or other guardrail features in the tools you use. For example, if you want to run checks and enforce them on containers before they’re deployed to an orchestrator,  use a native feature like admission controllers in Kubernetes or Sentinel policy as code in Nomad (enterprise version). Terraform can also use Sentinel (enterprise version) to enforce policy before deploying a container onto a single instance server. 5. Container Encryption Encryption is an additional security measure with an authentication and authorization step before the manipulation of a container. When using encryption tools for containers, the container itself is encrypted to include all the internal files and the metadata associated with the container. This encryption happens at the build step before being pushed to an external repository. Typical key encryption formats are: JWE (RFC7516), PGP (RFC4880), and PKCS7. The encryption key can be stored in a secrets manager. is getting the but there are other options here as well: Vault broadest adoption Azure Key Vault AWS Secrets Manager GCP Secrets Management These secret managers can also add an authentication step to encrypt and decrypt containers. CI/CD platforms, container scanning tools, and other services also need access to the encryption key stored in the secrets manager, so make sure it has a broad integration ecosystem. Vault, for example, integrates with open-source and commercial tools to add an encryption and decryption step into the software delivery lifecycle. (a tool for moving encrypted image repositories), (for building and running encrypted containers), and Containerd's (to encrypt and decrypt Open Container Initiative (OCI) containers) are examples of tools that can add a cryptographic workflow in your container image pipeline. Skopeo Singularity Imgcrypt When importing containers from outside sources, it’s important to have tools that can encrypt them and keep them up to date with vulnerability patches. Running container scans and encrypting containers before they are pushed to trusted sources are two great ways to continuously secure your software supply chain and image repository. To run a container in an orchestration pipeline, the service needs to authenticate and be authorized to access the decryption key. This additional step in the pipeline adds file encryption and provides least-privilege access to the container at runtime. 6. Container Signing is critical for defending customer data, especially with the industry’s ubiquitous use of open-source software. Many organizations like SLSA and OpenSSF’s governing boards have created frameworks for measuring the security and integrity of the software supply chain. Containers can have digital signatures applied to commits as metadata with tools like . Securing the software supply chain Sigstore's cosign Additional metadata like the author of the commit or vulnerability scan results can be used as a validation gate check before deploying to a container orchestration platform. The commit signature can use a self-signed key with cosign to verify the authenticity of the steps or allow integrations with external public key infrastructure (PKI) servers. In PKI workflows, Vault can act as a certificate authority and generate programmatically. PKI certificates The benefit of using Vault certificates for signing a commit is the authentication process. Authenticating before each commit affirms that the human or machine executing the signing process is authorized to sign container images. Dynamic, short-lived PKI certificates shorten the threat window in which an adversary might gain access to a key and use it to inject bad code or improperly sign a container. Tools like add a timestamp of the signature verification to an immutable ledger. This ledger can be used to validate the signature authenticity even after the short-lived certificate has expired. Sigstore's Rekor 7. Host Operating System Vulnerability Scanning Vulnerabilities in the host operating system provide threat vectors to the containers running on the host system. Hence, the host operating system running the container needs to be monitored and scanned for vulnerabilities.  OS scanning tools have been around for ages but some of the popular commercial and open-source host vulnerability scanners are: Palo Alto’s Prisma Cloud Snyk OpenSCAP OpenVaS NMAP It’s good security hygiene to keep operating systems up to date and secure but it can be a pain to manually update hosts consistently. One way to speed this up is to automate the process and treat your host systems like cattle vs. pets. Container orchestration platforms allow the host operating system to become commoditized and much easier to cull. If configured correctly, host operating systems can be shut down and containers will ultimately find a new home on an available server in the cluster with the appropriate amount of CPU, memory, and disk. As operating systems are scanned and CVEs are released, automated triggers from scanning tools can , like Packer (image builder) and Terraform (infra provisioning), to quickly patch an entire cluster of servers with no downtime for running services. Finding a workflow to get patching quickly will ensure there aren’t any underlying attack vectors allowing access to the container from the underlying operating system. kick off a toolchain Final Notes Container security is a must in modern cloud architecture. Hopefully, this list of current tools will help improve your process for securing the creation, signing, encrypting, verifying, validating, provisioning, connecting, and running of containers. More strategies? Did I miss anything?  Let me know in the comments! For more information, check out the following resources: NIST Container deployments NIST Application Container Security Guide

Container Security Tools to Try in 2023

This article is part of the with which introduces the space and how it differs from on-premise security. covers the security consideration for building the cloud foundation. Enterprise Cloud Security Series Part I - Introduction Part II - Foundation This article focuses on building security in the application development process. The security of application can, at a high level, be classified into the following: Application is being developed with security in mind and is secure Application development infrastructure is secure Application is secure while running Application runtime infrastructure is secure. Developing a secure application requires that the application is designed and developed to be secure on a secure development platform ( ). Reflections on Trusting Trust Application security in DevOps CNCF has done a great job putting together a security based on the , , , and . It covers the typical lifecycle of development, distribution, deployment, and runtime of applications. Due to its focus on cloud-native development, it is a bit light on some aspects like unique considerations for securing SaaS and PaaS services. white paper NIST Application Security Container Guide Center for Internet Security (CIS) NIST Security Strategies for Micro-service OpenSCAP The overall approach of DevOps is to ensure that security is achieved as the following types of artifacts flow through the process. — represents a collection of all the artifacts that are part of the runtime. Application — consists of SaaS, PaaS, IaaS, and application configuration that drives the provisioning and runtime behavior of the application. Configuration — primarily deployed to IaaS (Infrastructure as a Service), CaaS (container as a service), FaaS (Function as a Service), and PaaS (Platform as a Service) as part of the application. Code — provides the platform on which the code runs. This typically includes VM and container images. Image As part of the DevOps, the following stages form the part of the process — involves the developers writing code, configuration, and sometimes putting together the image on which the solution is going to run. Developers are also responsible for testing and may also be either responsible for or involved in threat modeling, given their understanding of the domain and technology involved. Develop — focuses on packaging the outcome of the development process to either use it within or share it with other teams, business units, and organizations for use. This typically involves more comprehensive testing to ensure that artifacts are “production-ready” for release. The packaged artifacts are typically pushed to the registry for staging purposes. Distribute — involves storing the artifacts for future use by downstream systems and users. Registry staging — focuses on setting up the application in a landing zone designated for the application. Deployment — consists of an application and associated infrastructure that provides the service that the application was designed for. Runtime Across these stages, the various aspects of security must be monitored and addressed to ensure that the application is “ ”. This “ ” approach to application security can reduce the time, effort, and cost of fixing these security issues during runtime. The following aspects of security are typically addressed as part of DevOps processes. secure-by-default shift-left Application — focuses on identifying, communicating, and understanding threats and mitigations within the context of protecting something of value, i.e., data and/or service. This is an important part of the development process to provide appropriate guidance for users to deploy applications securely. During the deployment phase, threat modeling is typically performed to identify all the runtime threats based on the deployment architecture. Threat modeling — the changes to code, configuration, and images should be reviewed to ensure that it aligns with functional and non-functional requirements and enterprise guidelines. Code review — ensure that the code, configuration, and images are aligned with functional and non-functional (including security) requirements. Depending on the requirements, the testing would typically involve unit testing, system testing, integration testing, performance testing, and others during the entire development lifecycle. With regards to security, is one of the typical testing performed to ensure that implementation is secure by design. Testing penetration testing — looks at all the third-party artifacts like VM and container images, libraries that are pulled from external sources like maven repositories, docker container repository, and cloud marketplace and performs vulnerability analysis, signing, and publishing to enterprise registry for use by application teams. Third-party artifact analysis — involves checking all the components that form part of the runtime environment, including code, configuration, image, network, against pre-defined enterprise policies to ensure that the application is secure after deployment. Security policy validation — ensures that all the application and infrastructure logs are set up correctly to flow into the SIEM and/or performance/ telemetry platform after deployment. Log Flow validation — focuses on verifying whether the deployed application is deployed such that it is aligned with the resiliency (failover, disaster recovery) requirement identified. Resiliency validation — ensures that deployed infrastructure is set up appropriately (e.g., installation of any agent, access to the environment from forensic automation tools) to quickly perform evidence collection for forensic investigations in case of any incident. Forensic validation Image — focuses on identifying the various artifacts that form part of each layer of container images and validating the vulnerability associated with the artifact. Docker file vulnerability scan — looks for security and configuration issues with Kubernetes workload. Kubernetes manifest scan — looks for potential points of exploits in the VM image. VM Image vulnerability scan — strips unnecessary software and changes configuration to limit potential vulnerabilities that can be exploited by attackers. Image hardening — adds digital fingerprint to the image, which can be tested later to verify trust. Image signing and validation — needs to be managed to ensure that unapproved or insecure images or image layers are not pulled in as part of any of the phases. Public registry access — ensures that images being developed are approved or aligned with enterprise policies. Marketplace restriction — ensures that any intellectual property stored on the image is protected. Image encryption Code — focuses on white box testing by analyzing source code before it is compiled to find vulnerabilities that may be exploitable at runtime depending on the deployment architecture and other factors. Static Analysis (SAST) — reviews all the dependencies associated with code to ensure that corresponding licenses align with enterprise policies and limit exposure. Dependency License Scan — adds digital fingerprint to packages and artifacts created as part of compilation and packaging processes. Artifact and package signing — identifies any existing exploit associated with dependencies that are being used by code. Dependency vulnerability scan — ensures that only authorized libraries are used. External library access — looks for references to public and unauthorized repositories in the code, configuration, and build process. Public Repo access scan Configuration — IaaC (Infrastructure as a Code) focuses on defining (and provisioning) infrastructure using a configuration or code. The configuration (e.g., cloud formation, terraform, azure ARM, GCP deployment manager) can be scanned using predefined and custom policies to ensure that infrastructure is secure (e.g., the disk is encrypted, logs are enabled, access is configured, etc.) and configured correctly. Infrastructure as code scan — is the runtime equivalent of where the same policies are validated against the runtime to ensure that the deployed infrastructure aligns with security and business policies. Runtime policy validation IaaC, — looks for shared secrets, keys, private certificates in the registry. Shared secret scanning DevOps platform security Security of the DevOps platform consists of securing all the technical components, enterprise processes with the idea that resistance to developers’ flow increases as the application moves from development to production. Each enterprise has to find the right balance of the developer’s freedom with security and operational controls needed to ensure business objectives. A sample CI/CD flow is shown below: DevOps consists of different components that need to be secured. Apply the criteria to structure the approach below. V.L.A.D.R. Code repository/Source Control Management Depending on the delivery model (IaaS, PaaS, or SaaS) and applicable shared responsibility, the enterprise may be responsible for managing the basic platform security aspects like technology vulnerabilities, logging, and resiliency. Code repository poses a unique set of security challenges like process vulnerabilities, access, and data which should be considered appropriately. The code repository is a type of database since it contains business and trade secrets, algorithms that can result in a loss if exposed. Keeping that in mind, code repositories should be handled similarly to data within the enterprise. This would typically involve separating the code repository based on sensitivity. Organization policies to secure data at rest by encryption with organization-managed keys may mean that SaaS code repository can not be used for sensitive code. Access to the code repository forms another important part of the process. Strong controls regarding user’s access should be adequately be supplemented by control over identities (e.g., API keys) and access of scripts and other automation processes (e.g., automatic key rotation, service identity for authentication). It is important to ensure that API keys and similar tokens have the same level of access control as user/service accounts (e.g., SAML authentication may enforce device authentication, which API keys may miss). All the access should be based on the principle of least privilege with, where possible, the use of UEBA (User & Entity Behavior Analysis) to continuously fine-tune the access and with periodic access reviews. Additional network access control may be required to be put in place for sensitive repositories to reduce the attack surface. Code repository in most of the cases also stores other aspects of the DevOps (e.g., CI/CD pipeline definition, application configuration, SAST/DAST configuration, vulnerability exceptions), which can become a source of process vulnerabilities. It is important to ensure that appropriate pre-commit hooks; scanning processes are in place to detect changes to CI/CD pipeline that circumvent enterprise processes, shared secrets/API keys/private certificates in the code base, ignore false-positive vulnerability without exception, and so on. The enterprise should review the developers’ toolsets and use-cases to identify these process vulnerabilities and deploy controls to address them. Registry Besides basic platform security aspects like technology vulnerabilities, logging, and resiliency that enterprise may be responsible for depending on the delivery model and corresponding shared responsibility, registry-specific process vulnerabilities and access model along with data continue to be the focus. Depending on the degree of control in place, applications may not have access to a public repository during the development and distribution phase. The repository may operate as a proxy for external dependencies like libraries, VM Images, container images, and tool plugins (e.g., approved terraform providers). Moreover, additional controls may be in place to scan the external dependencies and generated artifacts for vulnerabilities, sensitive information. Access to the registry is another aspect that plays an important role. This access can be of significant importance when the repository operates as a proxy to public repositories. The ability to add or remove the external repository allowed can have a significant impact. Most registries also support the ability to sign and encrypt the artifacts, which provide data-level security. CI/CD Automation CI/CD automation platform provides a way to execute the build process and deploy the artifacts. This platform needs to ensure that the process being executed is aligned with enterprise policies (e.g., the application should not have critical severity vulnerability before deployment in production) and deployment is being with the least privilege access. In addition to that, the build and packaging tools should be from trusted sources and publishers (including extensions like terraform provider plugins) to ensure that artifacts being generated are not compromised during the build process. Due to the involvement of secrets/keys, it is important that automation is executed on the platform with limited accessibility and with an adequate level of encryption of data at rest and in motion. Furthermore, adequate environment hygiene (e.g., delete all the files after build, encrypt and store state files using BYOK on an alternate storage system) should be maintained during and after execution of the build and deployment process to ensure that details about the application, including secrets/service account passwords are not present on the file system after the CI/CD pipeline has been executed. Adding security to development and operational processes in any enterprise can be a significant daunting task further exacerbated by the changes in culture within most enterprises. Maintaining this balance between developer’s choice and enterprise security posture can be a significant challenge with the rollout of DevOps within organizations. Also published at Medium's subdomain .

Enterprise Cloud Security Guide: Secure Application Development

In our last post “ ”, we introduced how to set up price data in five minutes and we got a lot of good feedback and contributions to the open source . How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes bitcoin MarketStore The data speed is really important Today, I wanted to tell you how fast MarketStore is using the same data so that you can see the performance benefit of using the awesome open source financial timeseries database. Faster data means more backtesting and more training in machine learning Faster data means more backtesting and more training in machine learning for our trading algorithm. We are seeing a number of successful machine learning-based trading algos in the space, but one of the key points we learned is . It’s important not just for backtesting, but also for . the data speed is really important training AI-style algorithms since it by nature requires an iterative process This is another post to walk you through step by step. But TL;DR, it is really fast. Setup Last time, we showed how to setup the historical daily bitcoin price data with MarketStore. This time, we store all the minute-level historical prices using the same mechanism called background worker, but with a slightly different configuration. root_directory: /project/data/mktsdblisten_port: 5993# timezone: "America/New_York"log_level: infoqueryable: truestop_grace_period: 0wal_rotate_interval: 5enable_add: trueenable_remove: falseenable_last_known: falsetriggers: module: ondiskagg.soon: "*/1Min/OHLCV"config:destinations:- 5Min- 15Min- 1H- 1Dbgworkers: module: gdaxfeeder.soname: GdaxFetcherconfig:query_start: "2016-01-01 00:00"base_timefame: “1Min”symbols:- BTC Almost 2.5 years with more than 1 million bars The difference from last time is that background worker is configured to fetch 1-minute bar data instead of 1-day bar data, starting from 2016–01–01. That is . You will need to keep the server up and running for a day or so to fill all the data, since GDAX’s historical price API does not allow you to fetch that many data points quickly. almost 2.5 years with more than 1 million bars Again, the data fetch worker carefully controls the data fetch speed in case the API returns “Rate Limit” error. So you just need to sleep on it. Additional configuration here is something called “on-disk aggregate” trigger. What it does is to aggregate 1-minute bar data for lower resolutions (here 5 minutes, 15 minutes, 1 hour, and 1 day). Check the longer time horizon to verify the entry/exit signals In a typical trading strategy, . So it is a pretty important feature. You would need pretty complicated LEFT JOIN query to achieve the same time-windowed aggregate in SQL. But with MarketStore, all you need is this small section in the configuration file. you will need to check the longer time horizon to verify the entry/exit signals even if you are working on the minute level The machine we are using for this test is a typical Ubuntu virtual machine with 8 of Intel(R) Xeon(R) CPU E5–2673 v3 @ 2.40GHz, 32GB RAM and SSD. The Benchmark We are going to have a DataFrame object in which holds all the minute level historical price data of bitcoin since January of 2016 from the server. We compare MarketStore and PostgreSQL. python Unfortunately lots of people in this space are using some sort of SQL database PostgreSQL is not really meant to be the data store for this type of data, but for this purpose since there is no other alternative. That’s why we built MarketStore. unfortunately lots of people in this space are using some sort of SQL database The table definition of the bitcoin data in PostgreSQL side looks like this. btc=# \d pricesTable "public.prices"Column |            Type | Modifiers--------+-----------------------------+-----------t      | timestamp without time zone |open   | double precision            |high   | double precision            |low    | double precision            |close  | double precision            |volume | double precision            | The code looks like this. # For postgresdef get_df_from_pg_one(conn, symbol):tbl = f'"{symbol}"'cur = conn.cursor() order by timestamp, so the client doesn’t have to do it cur.execute(f"SELECT t, open, high, low, close, volume FROM {tbl} ORDER BY t")times = []opens = []highs = []lows = []closes = []volumes = []for t, open, high, low, close, volume in cur.fetchall():times.append(t)opens.append(open)highs.append(high)lows.append(low)closes.append(close)volumes.append(volume) return pd.DataFrame(dict(open=opens,high=highs,low=lows,close=closes,volume=volumes,), index=times) # For MarketStoredef get_df_from_mkts_one(symbol):params = pymkts.Params(symbol, '1Min', 'OHLCV')return pymkts.Client('http://localhost:6000/rpc').query(params).first().df() You don’t need much client code to get the DataFrame object The input and output is basically the same, in that one symbol name is given, query the remote server over the network, and get one DataFrame. One strong benefit of MarketStore is since the wire protocol is designed to give an array of numbers efficiently. you don’t need much client code to get the DataFrame object The Result First, PostgreSQL %time df = example.get_df_from_pg_one(conn, 'prices')CPU times: user 8.11 s, sys: 414 ms, total: 8.53 sWall time: 15.3 s And MarketStore %time df = example.get_df_from_mkts_one('BTC')CPU times: user 109 ms, sys: 69.5 ms, total: 192 msWall time: 291 ms Both results of course look the same like below. In [21]: df.head()Out[21]:open high low close   volume2016-01-01 00:00:00  430.35 430.39 430.35  430.39 0.07272016-01-01 00:01:00  430.38 430.40 430.38  430.40 0.94782016-01-01 00:02:00  430.40 430.40 430.40  430.40 1.63342016-01-01 00:03:00  430.39 430.39 430.36  430.36 12.56632016-01-01 00:04:00  430.39 430.39 430.39  430.39 1.9530 In [22]: df.shapeOut[22]: (1198274, 5) 50 times difference A bitcoin was about $430 back then… Anyway, . Remember, you may need to get the same data again and again for different kinds of backtesting and optimization as well as ML training. you can see the difference between 0.3 vs 15 seconds which is about 50 times difference Also you may want to query not just bitcoins but also other coins, stocks and fiat currencies, since the entire database wouldn’t fit into your main memory usually. Scalability advantage in MarketStore MarketStore can serve multiple symbol/timeframe in one query pretty efficiently, whereas with PostgreSQL and other relational databases you will need to query one table at a time, so there is also scalability advantage in MarketStore when you need multiple instruments. Querying 7.7K symbols for US stocks To give some sense of this power, here is the result of querying 7.7K symbols for US stocks done as an internal testing. %time dfs = example.get_dfs_from_pg(symbols)CPU times: user 52.9 s, sys: 2.33 s, total: 55.3 sWall time: 1min 26s %time dfs = example.get_dfs_from_mkts(symbols)CPU times: user 814 ms, sys: 313 ms, total: 1.13 sWall time: 6.24 s Again, the amount of data is the same, and in this case each DataFrame is not as large as the bitcoin case, yet the difference to expand to large number of instruments is significant (more than 10 times). You can imagine in real life these two (per instrument and multi-instruments) factors multiply the data cost. Alpaca has been using MarkStore in our production Alpaca has been using MarkStore in our production for algo trading use cases both in our proprietary customers and our own purposes. It is actually amazing that this software is available to everyone for free, and we leverage this technology to help our algo trading customers ( e). signup for early access is her Please follow and for fresh posts on Financial Market, Algorithmic Trading, Technology. Alpaca Automation Generation You can find us , if you use twitter. @AlpacaHQ If you’re a hacker and can create something cool that works in the financial market, please check out our project “ ” where we provide simple REST Trading API and real-time market data for free. Commission Free Stock Trading API Brokerage services are provided by Alpaca Securities LLC ( ), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc. alpaca.markets

How to Achieve 50x Faster Bitcoin Price Data Queries

In my , I showed you how to use a like to create a secure tunnel to your private resources. In this post, I will walk you through step by step on how to setup a secure bridge to your remote from your home network with a as a . previous article VPN Software Solution OpenVPN AWS AWS VPC subnets Raspberry PI Customer Gateway To get started, find your public-facing IP address: Home Router Next, sign in to , navigate to and create a new : AWS Management Console VPC Dashboard VPN Customer Gateway Next, create a : Virtual Private Gateway And attach it to the target : VPC Then, create a with the and the : VPN Connection Customer Gateway Virtual Private Gateway Note: Make sure to add your to the section. Home CIDR subnet Static IP Prefixes Once the is created, click on “ ” tab, you should see two tunnels for redundancy: VPN Connection Tunnel Details It may take a few minutes to create the VPN connection. When it’s ready, select the connection and choose “ “, and open the configuration file and write down your and : Download Configuration Pre-shared-key Tunnel IP I used a (Quand Core CPU 1.2 GHz, 1 GB RAM) with , with enabled (default username & password: / ), you can login and start manipulating the PI: Raspberry PI 3 Raspbian SSH server pi raspberry kernel support must be installed. Therefore, you must install on your PI: IPsec openswan sudo apt-get install -y openswan lsof Update the file as below: /etc/ipsec.conf Create a new in : IPsec Connection /etc/ipsec.d/home-to-aws.conf : Your Raspberry PI private IP. left : Your Home Router public-facing IP. leftid : CIDR of your Home Subnet. leftsubnet : Virtual Private Gateway Tunnel IP. right : CIDR of your VPC. rightsubnet Add the tunnel to : pre-shared key /var/lib/openswan/ipsec.secrets.inc 89.95.X.Y 52.47.119.151 : PSK “irCAIDE1NFxyOiE4w49ijHfPMjTW9rL6” To enable the , edit , and ensure the following lines are uncommented: IPv4 forwarding /etc/sysctl.conf Run to reload it. Then, restart service: sysctl -p IPsec service ipsec restart Verify if the service is running correctly: If you go back to your you should see the 1st tunnel status changed to : AWS Dashboard, UP Add a new route entry that forwards traffic to your home subnet through the VPN Gateway: Note: Follow the same steps above to setup the 2nd tunnel for resiliency & high availablity of VPN connectivity. Launch an in the private subnet to verify the VPN connection: EC2 instance Allow SSH only from your : Home Gateway CIDR Once the instance is created, connect via using the server private ip address: SSH Congratulations ! you can now connect securely to your private EC2 instances. To take it further and connect from other machines in the same , add a static route as described below: Home Network 1 — Windows route add 10.0.0.0 MASK 255.255.0.0 192.168.1.81 2 — Linux sudo up route add -net 10.0.0.0 netmask 255.255.0.0 gw 192.168.31.232 3 — Mac OS X sudo route -n add 10.0.0.0/16 192.168.31.232 Test it out:

Setup Raspberry PI 3 as AWS VPN Customer Gateway

How to Turn Your OpenAPI Specification Into an AI Chatbot With RAG

FastAPI Got Me an OpenAPI Spec Really... Fast

ফাস্টএপিআই আমাকে একটি ওপেনএপিআই স্পেক দিয়েছে... দ্রুত

FastAPI hat mir wirklich ... schnell eine OpenAPI-Spezifikation besorgt

FastAPI me consiguió una especificación OpenAPI realmente... rápido

FastAPI m'a offert une spécification OpenAPI vraiment... Rapide

FastAPI ने मुझे एक OpenAPI विनिर्देशन वास्तव में... तेजी से दिलाया

FastAPI で OpenAPI 仕様を本当に早く入手できました

FastAPI를 통해 OpenAPI 사양이 정말... 빨라졌습니다.

FastAPI me deu uma especificação OpenAPI muito... rápida

FastAPI дал мне спецификацию OpenAPI действительно... быстро

FastAPI Bana Gerçekten Bir OpenAPI Özelliği Sağladı... Hızlı

FastAPI đã cung cấp cho tôi thông số OpenAPI thực sự... Nhanh

FastAPI 让我快速获得了 OpenAPI 规范

When I first saw on in 2018, I immediately fell in love with the technology. As I listened to the presentation, a scene from ' ' came to mind, where they spoke briefly about the technology. In the scene, precogs lie in a bathtub, dreaming up visions of the future, which are then read from their brains using light. How absurd, I thought. Light can't penetrate brain tissue. How could it possibly be used to read thoughts? Well, Jepsen's talk revealed that light can indeed go through not just the brain tissue, but even the skull. The problem is that it scatters so much that it becomes completely unusable. This is where the brilliant technology of OpenWater comes in… Mary Lou Jepsen's TED talk OpenWater Minority Report Jepsen and her team found a solution to focus light through living tissue using appropriately structured light. Essentially, they built a microscope whose lens system is made of flesh and bones. It sounds strange. But that's exactly what it is. They can see into the human body by treating the living tissue itself as part of the system. Jepsen beautifully demonstrates this in her TED talk, where she places a piece of tissue in the path of a laser, and yet, a focused point appears on the wall instead of scattered light. To understand how this is possible, let's look at Jepsen's patent. On the left side of the diagram, there's a "hologram projector" (110), consisting of a laser emitter (105/103) and a special display (113). This display appropriately adjusts the phase of the light beams passing through it. But why is it called a “hologram projector“? The essence of is that, instead of capturing the intensity of light beams on a photographic plate, it records their phase. While this process loses color information, in return, if we illuminate it with the same laser used to make the recording, the light beams emerge in the exact phase as they would from the real object. That's why we perceive the hologram as three-dimensional. In Jepsen's patent, the display plays the same role as the photographic plate in a hologram, but while the photographic plate is static, this display is dynamic. Unlike traditional holograms, these holograms are incomprehensible to the human eye. However, they offer the possibility to focus light through human tissue. It's akin to placing a lens in front of a blurry image to obtain a clear, focused picture, except here, the lens is made of human flesh and bones. hologram technology On the right side of the diagram, a "holographic camera" (160) is depicted, which differs from a traditional camera in that it captures a "hologram." This means that, like the projector, it records the phase of light instead of its intensity. The operation of the system is very similar to how a traditional hologram is recorded on a photographic plate using a reference laser, but here, instead of a photographic plate, there's an image pixel array (170). This setup enables the camera to capture the nuanced phase information of the light. In the center, marked as "diffuse medium" (130), is the human tissue that we want to observe. This is the medium onto which the hologram is projected, and it's calibrated until the tissues in the observed area focus the light. The light that exits the tissues is captured by the camera, from which the necessary information can be extracted. In a nutshell, this is how Jepsen's microscope (built from flesh and bones) works. For those who are interested in a deeper understanding of the system, I recommend reading the . (I also had a more detailed about the topic from 2019.) patents article With this ingenious device, we can create images of the human body with a resolution superior to that of an MRI, and there's no need for massive magnets and expensive equipment. Jepsen's tool is affordable, accessible to everyone, and can fit in a box the size of a mobile phone or even be integrated throughout the day into our clothing. But why would someone want to wear an MRI-like device all day? Do you know what the real treatment for cancer is? No, it's not a special drug, but rather early detection. Cancer starts its malicious life as one renegade cell. It grows quietly and is quite harmless for a long time. The problem is that it spreads stealthily and unnoticed throughout the human body, and often by the time it's detected, it's too late. If everyone could scan themselves daily with an MRI, we could detect cancer in time before it causes harm, eliminating the need for anyone to die from it. But this is not only true for cancer. Many serious diseases can be easily treated if detected early. Considering the leading causes of death, a system like this could offer almost everyone a longer and better quality of life. Therefore, OpenWater's development is not just a tool, but a complete reform of healthcare, which Jepsen and her team call the "Silicon Hospital." What OpenWater offers in healthcare is fascinating, yet my enthusiasm for this technology lies elsewhere. Those who regularly read my articles know . I believe that following the Internet and Mobile communication (and maybe the VR revolution), the next major technological revolution will be the BCI revolution. OpenWater presents a very promising solution for this, as its device allows for neuron-level observation of the brain, all in a non-invasive manner (no surgery required, unlike Neuralink, for example). Experiments are currently underway to , , or based on fMRI data. These experiments are very promising, but they require the patient to lie inside a large tube (the fMRI machine). Jepsen's technology would replace this with a simple cap. Imagine a net cap capable of recording our dreams while we sleep. I am a big fan of Brain-Computer Interfaces (BCIs) reconstruct dreams visual images human speech Another very exciting possibility is to focus on the spinal cord instead of the brain. Just think about it: all information from our brain to the entire body passes through a "bundle of wires" about 4 square centimeters in diameter (the spinal cord). The brain controls our limbs, our organs, and our entire body through this. If we can interpret this information, we might restore the ability to walk paralyzed people or build a virtual reality for them where they can live a full life. Since this is a smaller area than the entire brain, observing it might be much simpler than observing the entire brain, and the possibilities are fascinating. Even more exciting is the possibility that with focused light, we can not only peer into the human body but also induce changes within it. For instance, if cancer has already developed in our body, we can use light to destroy cancerous cells or deliver light-activated toxins that target only the cancer cells. Similarly, we can activate neurons with light, making not just thought reading possible, but also the projection of images directly into the brain. It sounds like science fiction, doesn't it? In 2008, DARPA launched its program. One of the projects supported within this program is the , which aims to achieve visual telepathy. The project received $18 million in funding from DARPA. The solution, similar to OpenWater's approach, would use light to read the brain's state and magnetic fields to activate neurons. However, this would involve using vector viruses and genetic engineering to modify brain cells so they can be activated by magnetic fields. This latter part might not sound so appealing, but perhaps it's not necessary. With OpenWater's technology, perhaps light alone could be sufficient for both writing to and reading from neurons. Next-Generation Nonsurgical Neurotechnology (N3) Rice University MOANA project Whatever the winning technology turns out to be, it's clear that many are taking Brain-Computer Interface (BCI) technology very seriously, investing millions of dollars in its development. This is hardly surprising, as such technology could potentially cure blindness, directly integrate prostheses with the brain, or even enable fully immersive virtual reality experiences. Perhaps the very technologies that current researchers are developing for better VR headsets (such as higher resolution miniature displays, dedicated processors, etc.) could in the future be used to project images directly into our brains using lasers. This merging of advanced imaging and neurotechnology could revolutionize not just healthcare, but the way we interact with technology itself. One might think that if someone possesses such a significant technology, they would guard its details as closely guarded secrets, providing access only for substantial sums of money. However, chose a different path: in January 2024, they announced that they would make the technology open-source, including hardware blueprints, source codes, patents, measurement data, etc. It's difficult to determine what is more revolutionary: the technology itself or the philosophy and business model behind it. This approach not only accelerates innovation by allowing others to build upon and improve the technology but also democratizes access, potentially leading to widespread applications in various fields beyond just healthcare and computing. OpenWater If you're a researcher, developer, BCI expert, or simply someone interested in technology, you can access everything on and the . I would encourage everyone whose field is even slightly related to OpenWater to review the materials and contribute to the technology's development. This is genuinely a breakthrough that could bring about a change in the whole of humanity. GitHub company's website I realize that 2024 has only just begun, but if what OpenWater promises holds, then I'd venture to say that Jepsen has stolen the show. Making OpenWater open-source could very well be the most important tech announcement of 2024.

OpenWater: A Revolutionary Open-source Wearable MRI and BCI Device

ওপেনওয়াটার: একটি বিপ্লবী ওপেন সোর্স পরিধানযোগ্য এমআরআই এবং বিসিআই ডিভাইস

OpenWater: Ein revolutionäres tragbares Open-Source-MRT- und BCI-Gerät

OpenWater: un revolucionario dispositivo portátil de resonancia magnética y BCI de código abierto

OpenWater : un appareil d'IRM et BCI portable open source révolutionnaire

ओपनवाटर: एक क्रांतिकारी ओपन-सोर्स पहनने योग्य एमआरआई और बीसीआई डिवाइस

OpenWater: 革新的なオープンソースのウェアラブル MRI および BCI デバイス

OpenWater: 혁신적인 오픈 소스 웨어러블 MRI 및 BCI 장치

OpenWater: um dispositivo revolucionário de ressonância magnética e BCI vestível de código aberto

OpenWater: революционное носимое устройство МРТ и ИМК с открытым исходным кодом

OpenWater: Devrim Yaratan Açık Kaynaklı Giyilebilir MRI ve BCI Cihazı

OpenWater: Thiết bị MRI và BCI có thể đeo được nguồn mở mang tính cách mạng

OpenWater：革命性的开源可穿戴 MRI 和 BCI 设备

What are we talking about here? As part of my job, I advise a lot of customers who use AWS. Many of them build complex systems on top of services. AWS OpenSearch is typically included in the system landscape to cover a wide range of different business cases, some of which require an Internet-accessible cluster with HTTP basic authentication. AWS does not give this capability out of the box; thus, in this article, I will demonstrate how to leverage AWS native services to publish your OpenSearch cluster to the Internet. AWS The current limitations OpenSearch cluster can be deployed in two modes with the following features VPC based Public domain Only private endpoint in the following format https://vpc-domain-name Public endpoint in the following format https://search-domain-name Cluster health tab does not include shard information, and the Indices tab isn't present. All information about cluster health and indexes are available Security is based on AWS security groups Security is based on IP-based access policies Dashboards are available only from VPC and requires VPN or something else for being accessible Dashboards are available publicly for IP range that’s configured via IP-based access policies AWS currently does not provide any out-of-the-box way for publishing an OpenSearch cluster with basic HTTP authentication to the Internet for any IP range (e.g.: ). It is possible to configure an AWS IAM Policy to allow a certain IP range or to utilize a publicly open cluster with SAML authorization; however, this is not suitable for cases such as mobile clients or TV set-top boxes, where the client can have any unpredictable IP address and dev team does not have the capacity for SAML implementation. 0.0.0.0/0 Maybe in the future, there will be something like an RDS proxy, but so far, AWS has no public plans for this feature. Solutions The official AWS proposal based on Nginx The AWS documentation recommends using a reverse proxy based on a highly available EC2 Nginx cluster. It's a good approach, but it requires a lot of extra effort for Nginx maintenance, which would be inconvenient in the case of a large number of clusters. In addition, as AWS does not provide highly available Nginx as a service, you will be unable to configure your infrastructure using a single tool such as Terraform, Pulumi, or CloudFormation. OpenSearch The solution based on AWS native services My approach, which I recommend for use in production environments, is based on AWS Application load balancer (ALB). Nginx can be skipped because ALB provides the same functionality as Nginx for all basic scenarios and requires no additional configuration or automation work. AWS ALB points to an IP address target group that contains a set of private OpenSearch IP addresses, and the OpenSearch cluster should be configured in VPC mode. It is also important to configure HTTP redirects at the ALB level in this configuration, as seen in the image below. ALB targets should also require the configuration of health checks because without them, OpenSearch IP addresses will be removed from IP pools. Keep in mind that the http code must be added to the list of successful health check codes when you are using basic HTTP authentication. 401 Configuring the ALB target group once is insufficient. Any OpenSearch cluster change can cause the IP address to change, so to ensure that the target groups contain all OpenSearch IPs at all times, the IP addresses pool should be maintained by a simple Lambda function that runs once per hour and verifies that all OpenSearch IPs are present and actual. The complete AWS function code is provided below. import boto3
import socket

def lambda_handler(event, context):
    # Initialize AWS clients
    elasticsearch = boto3.client('es')
    elbv2 = boto3.client('elbv2')

    # Retrieve target group ARN and OpenSearch domain name from event
    target_group_arn = event.get('targetGroupARN')
    open_search_domain_name = event.get('openSearchDomainName')

    # Describe OpenSearch cluster to retrieve DNS endpoint
    response = elasticsearch.describe_elasticsearch_domain(DomainName=open_search_domain_name)
    opensearch_dns = response['DomainStatus']['Endpoints']['vpc']

    # Resolve DNS to get IP addresses
    opensearch_ips = socket.gethostbyname_ex(opensearch_dns)[2]
    print(f"OpenSearch IPs are: {opensearch_ips}")

    # Describe existing IPs in the EC2 target group
    target_group_info = elbv2.describe_target_health(TargetGroupArn=target_group_arn)
    existing_ips = [target['Target']['Id'] for target in target_group_info['TargetHealthDescriptions']]

    # Compare IPs to determine new ones and remove old ones
    ips_to_add = list(set(opensearch_ips) - set(existing_ips))
    ips_to_remove = list(set(existing_ips) - set(opensearch_ips))

    # Update the target group
    if ips_to_add:
        elbv2.register_targets(TargetGroupArn=target_group_arn, Targets=[{'Id': ip} for ip in ips_to_add])
    if ips_to_remove:
        elbv2.deregister_targets(TargetGroupArn=target_group_arn, Targets=[{'Id': ip} for ip in ips_to_remove])

    return {
        'statusCode': 200,
        'body': 'Updated EC2 target group with OpenSearch IPs.'
    } The lambda function can be deployed in VPC mode. It’s more secure but not mandatory. In the beginning, you may provide only permissions for your Lambda’s IAM role, as shown in the code snippet below. es:* {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/test-user"
        ]
      },
      "Action": [
        "es:*"
      ],
      "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*"
    }
  ]
} For a variety of reasons, connecting to an OpenSearch cluster with a floating IP address is inconvenient. All modern architectures use DNS to hide a pure IP address. We'll do the same via AWS Route53. Unfortunately, the Route53 DNS record is insufficient because ALB would display problems linked to HTTP certificate mismatches. We must generate a valid TLS certificate and attach it to an ALB listener to avoid HTTPS connection issues while using DNS. can be used to issue a purchased certificate, but I prefer free Let's Encrypt certificates. AWS does not natively support Let's Encrypt and Certificate Manager integration, so I created an AWS Lambda that automatically issues and renews TLS certificates a long time ago and proposes using it in the suggested design. This Lambda's sources may be found at AWS Certificate Manager https://github.com/kvendingoldo/aws-letsencrypt-lambda. Security highlights Security is not covered in this article, but I can not highlight two critical points: Don't forget to configure your AWS security groups correctly; you only need to allow the required port (443 in our example). Use AWS Trust Advisor to ensure that all useless ports are blocked. It would be a good idea to have different users with different types of rights when using OpenSearch HTTP basic authentication. Do not use an admin user with full cluster access. It would be preferable to have multiple users, each with access to a subset of OpenSearch indexes. You can keep all password information in AWS Secret Manager for simplified password management. Summary It is possible to avoid using a High Availability Nginx cluster in your architecture. It will save you money and configuration time. Instead of using tools like Terraform or Pulumi, use the lambda function to keep your ALB's target group up to date. It is critical to check the target pool's validity on a regular basis. Don't forget about security when it comes to implementation. Take note of AWS subnets, security groups, and OpenSearch users. Use as many AWS native components as possible. All of them can be configured using IaC tools like Terraform, Pulumi, or CloudFormation. Although EC2 instances can be configured using IaC tools, the provisioning, configuration, and launch times are substantially longer than with pure AWS ALB. Useful links AWS OpenSearch access proxy AWS Let’s encrypt lambda

How to Make AWS OpenSearch Publicly Accessible Without nginx Proxy

Greetings Comrades! One of the strangest logging setups I ever had - AWS ALB controller logs. Really, look at the scheme: 1. S3 Bucket for ALB Logs I assume that you already have an ALB Controller in your EKS Cluster. So, we should create an S3 bucket and enable the SQS trigger with Terraform: ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} 2. S3 Trigger to SQS Queue ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} 3.  Installing and Configuring Vector I always use ArgoCD for installing something in the Kubernetes cluster, so here is a manifest: apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb 4. Enabling Our Logs There are three ways you can enable logging: Old-fashioned documentation from AWS - link With Terraform aws_lb resource - link And the easiest way is - Kubernetes Ingress annotation (our way): alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs Now, you should be able to see logs in your OpenSearch console! That`s all! Greetings Comrades! One of the strangest logging setups I ever had - AWS ALB controller logs. Really, look at the scheme: 1. S3 Bucket for ALB Logs I assume that you already have an ALB Controller in your EKS Cluster. So, we should create an S3 bucket and enable the SQS trigger with Terraform: ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} 2. S3 Trigger to SQS Queue ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} 3.  Installing and Configuring Vector I always use ArgoCD for installing something in the Kubernetes cluster, so here is a manifest: apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb 4. Enabling Our Logs There are three ways you can enable logging: Old-fashioned documentation from AWS - link Old-fashioned documentation from AWS - link link With Terraform aws_lb resource - link With Terraform aws_lb resource - link link And the easiest way is - Kubernetes Ingress annotation (our way): And the easiest way is - Kubernetes Ingress annotation (our way): alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs Now, you should be able to see logs in your OpenSearch console! That`s all!

ALB Controller Logs With Vector and OpenSearch: A Guide

The key to providing an easy webhook integration experience for your users lies in your event types. The first thing your users are interested in seeing when integrating with you is what events you send and what information those events provide. You should consider event types (and their schemas) as part of your API. Hence, they need to be accurate and up-to-date. Using Your OpenAPI Specification If you have an API you offer to your customers, it's quite likely you have an OpenAPI specification already documenting it. You can now use that same OpenAPI file as a quick and easy way to create and update Svix Event Types. The OpenAPI 3.1 spec already defines a section . If your API docs define this field, you have all you need to create event types in Svix with no additional changes. webhooks as part of the standard This can be a great option if you want to avoid error-prone manual changes and don't need to update event types very often. If you update event types frequently, a might be more appropriate. programmatic integration How It Works In the Event Types section in your , select "Import from OpenAPI", and upload your OpenAPI file. dashboard Once your spec is uploaded, Svix will parse the section and suggest event types based on your existing definitions. webhooks All you need to do is set the event type name and hit 'Save'. For better readability, you might want this name to be slightly different than the name in your API definition. If an event type with that name already exists, it will get updated. You used your existing API definition to add event types to Svix; no extra work is needed. Schemas, descriptions, and examples can all be read from the OpenAPI spec and can be kept up to date with your API this way. That's it! Check out to learn more about how to get the most out of this integration. the docs For more content like this, make sure to follow us on , or for the latest updates for the , or join the discussion on . Twitter , GitHub RSS Svix webhook service our community Slack Also published here

How to Create Event Types Via OpenAPI Import

Stories, News & Mentions about OPENSWAP

Openswap’s Stories on HackerNoon