Proxytool is a developer tool to help you develop faster, deploy sooner with less code and errors

Proxytool

proxytool

How to Access YouTube in Restricted Areas — 10 Methods

Cómo acceder a YouTube en áreas restringidas: 10 métodos

Comment accéder à YouTube dans les zones restreintes – 10 méthodes

प्रतिबंधित क्षेत्रों में YouTube तक कैसे पहुंचें - 10 तरीके

制限エリアで YouTube にアクセスする方法 — 10 の方法

Como acessar o YouTube em áreas restritas – 10 métodos

Cách truy cập YouTube ở khu vực hạn chế - 10 phương pháp

如何在受限区域访问 YouTube — 10 种方法

What Are Supply Chain Attacks? Supply chain attacks involve compromising a company that has some sort of access to the network of another company. A third party may be used to attack the intended target. There are different variants of a supply chain attack. One approach is to attack one company to launch attacks against other companies. The SolarWinds attack serves as an example of this. Hackers installed malware on the software of the company. When an update was sent to its clients, the malicious software was installed, unbeknownst to the clients or SolarWinds. Supply Chain Attack Challenges for Organizations Another supply chain attack example involves hackers attacking a company by going through a third party with a business/network connection to their main target. The best example of this is the cyber attack against Target. The supermarket chain’s card readers had malware installed onto them by hackers. The credit card details of millions of customers were stolen as a result. Target’s relationship with a third-party company that serviced air conditioning units was instrumental in the attack. In order for the company to access air conditioning units remotely, it established a VPN (Virtual Private Network) connection with Target. After an employee of the company fell victim to a phishing attack, the hackers were able to gain access to the VPN connection and spread the malware for the main attack. With new supply chain attack events, come more questions about what represents effective management of supply chain cyber security risks. There are many ways to execute supply chain attacks. The 3CX attack is a great example of this. In a rare case, enterprise phone provider 3CX was compromised through another supply chain attack. A malware-laced version of financial software, X_Trader, was used by hackers to compromise the systems of 3CX. Cybersecurity company, Madiant (which was hired to investigate the breach) suspected the attackers to be North Korean state-backed hackers. A report by Google’s Threat Analysis group corroborates this finding. The report shows that Trading Technologies’ website was compromised in February 2022 as part of a North Korean operation targeting users of FinTech and cryptocurrency. The investigation by Mandiant shows that an employee at 3CX downloaded a version of the X_Trader software in April 2022 (without realizing it contained malicious programming) from Trading Technologies’ website, which hackers digitally signed with the company’s then code-signing certificate to make the software look legitimate. The X_Trader installer was digitally signed by a valid code signing certificate (which was set to expire in October 2022) with the subject of “Trading Technologies International, Inc.” Despite the software being reportedly retired in 2020 by Trading Technologies, it was still made available for download on the Trading Technologies website in 2022. Using the software, the hacking group gained backdoor access to the device of the employee. The access was used to move laterally through 3CX’s network and ultimately compromise the desktop phone application of 3CX. A Fast-Reverse Proxy tool was key in moving laterally in the environment of 3CX. Information-stealing malware was planted inside the corporate networks of customers, marking the first time concrete evidence has been found of a software supply chain attack that leads to another supply chain attack. “The identified software supply chain compromise is the first we are aware of which has led to an additional software supply chain compromise,” wrote Mandiant’s research team in a new disclosure blog. The rare event shows the creative ways access to networks can be used to distribute malware, crossing over into verticals and using follow-on intrusion campaigns. Malware, Veiledsignal, was instrumental in providing the threat actor with administrative-level access and persistence to the compromised system. Using this, they stole the credentials of the 3CX employee. 10 Million Affected by Cybersecurity Attacks What may come as a surprise to many is that supply chain attacks surpassed malware-based attacks in 2022 by 40%. It would be interesting to know how and why this happened. 10 million people were affected by the attacks. You may ask yourself, how? In contrast, malware-based attacks affected 4.3 million people. This points to the many vectors to exploit in supply chains. It’s hard to avoid activities that expose supply chains to cyber security risks. Looking Forward and Risk Mitigation The number of supply chain attacks increased by 600%. 88,000 reports surfaced of organizations compromised through their supply chains. Mitigation can be achieved through auditing third parties, using auditors to ensure accountability, and compliance with security standards. Network access may also be limited so that a vendor only gets access to the information and privileges it needs to carry out its work. There are so many vectors and vulnerabilities that can lead to supply chain attacks. More creative methods of exploiting vulnerabilities have surfaced with time, giving rise to a greater need for cyber security strategies that not only take entities into consideration but also their vendors and others in the supply chain who may have a greater responsibility for the cyber security interests of other entities than themselves. Education, automation, and segmentation will be critical in tightening the gaps in supply chains that make it easier for threat actors to compromise systems. Sources: https://www.mandiant.com/resources/blog/3cx-software-supply-chain-compromise https://www.csoonline.com/article/573925/supply-chain-attacks-increased-over-600-this-year-and-companies-are-falling-behind.html#:\~:text=The number of documented supply,supply chain management company Sonatype. https://blog.google/threat-analysis-group/ https://www.project44.com/blog/supply-chain-cybersecurity#:\~:text=According to the report%2C more,and individuals from cyber-attacks.

Explaining Supply Chain Attacks: Here's What You Need to Know

“Whoever controls the DNS, controls the universe” (image credit: OpenAI Dalle2) spice The following article will provide a very brief overview of what DNS actually is, and what makes it “authoritative”. Then, I will discuss why you may choose to disregard its authority in some cases in order to have a better browsing experience and gain a bunch of superpowers using a service our team has built over the last 2 years. Strap yourself in. What is DNS First things first: what is DNS? DNS (Domain Name System) is a distributed “phone book” that maps (domain) names to IP addresses. When you enter hackernoon.com into your browser, it will query the router on your network for the IP address of hackernoon.com. Your home router doesn’t perform any DNS resolution and forwards the request upstream to whatever DNS server is configured there. This could be your ISP or a 3rd party resolver like Cloudflare 1.1.1.1 or Google 8.8.8.8. These resolvers are typically recursive in nature, meaning they reach out to the root DNS servers for the .com zone, and perform a recursive lookup against the authoritative DNS servers for the DNS zone (hackernoon), and then you finally get the IP(s) that hackernoon.com resolves to. So your DNS query takes the following route (simplified view): Your Device → Router → DNS Provider → Root Name Server → Authoritative Server → IP Address Saul Goodman explains this best. This process is fairly slow and can take ~1s, so typically the response is cached by the DNS provider for some period of time so the next lookup can be served from a cache. Your browser will also cache the response for the duration of the TTL associated with a particular DNS record, so as not to initiate a DNS query every single time. The responses that the authoritative DNS server emits are typically static. This means that any user, anywhere, asking any DNS server “where is hackernoon.com” will get the same response: yegor@Work-Desktop:~$ dig +short hackernoon.com 172.67.69.96 104.26.11.89 104.26.10.89 There are some notable exceptions to this when it comes to CDNs, which will geo-code your source IP, and return variable results based on your location. A user in the UK will see different IPs than a user in the US. This is done for performance reasons, so people in different countries connect to the nearby CDN location in order to reduce latency. That being said, you’re still bound by what the authoritative DNS server returns (or doesn’t). The above flow is true for pretty much every single thing that you do online that communicates with a remote server. Every web resource you download,  every API call you make, and every Git commit you do resolves a domain to an IP address that the authoritative DNS server returned. What if you could disregard what the authoritative server tells you? What advantages (superpowers) does that have? Let’s dive in. What is Control D Control D, as you probably guessed, is a DNS service. It can do all the things your standard DNS provider can (resolve  domains), but it can also do a lot more. Control D is a user-configurable DNS service that offers transparent proxies deployed on top of an anycast network, with exit locations in over 69 countries. Sure, that all sounds impressive, but what does it ACTUALLY do? When you get started with Control D, you will get a set of DNS resolvers that are unique to your account and can enforce your unique configuration. Think of it as your personal authoritative DNS server, for the entire Internet. You can customize your configuration via a simple web interface, as there are no required apps to install to use the service.  Control D supports multiple DNS protocols: - least secure, but most widely supported Legacy IPv4/IPv6 - most secure, supported by modern OSes DNS-Over-HTTPS (DoH) - most secure, supported by modern OSes DNS-Over-TLS (DoT) The first protocol offers the best compatibility and can be used on any Internet connectable device. Legacy DNS is not encrypted, so it can be easily intercepted and manipulated by your ISP, or network administrator. The latter two offer you the best privacy and security, as your DNS queries are encrypted with TLS and cannot be intercepted or manipulated. Think of it as: Legacy = HTTP DoH/DoT = HTTPS What’s the difference between DoH and DoT and which one is better? Honestly, just stick with DoH, as it operates on TCP port 443 and is indistinguishable from normal HTTPS traffic. DoT operates on a special port (853) and is easily detectable (and therefore easily blocked). The advantages of DoT (less overhead and therefore faster resolution time) are debatable, and we have not seen them in practice. Mind you, on some platforms like Android, you have no choice but to use DoT, as DoH is not supported ( ). but will be soon Once you configure one of the DNS resolvers on your device (router, computer, browser, phone) your DNS queries will be steered to the Control D network. By default, if no settings are changed, Control D will behave like a standard DNS resolver, no different than Cloudflare, Google, or your local ISP. Once you start fiddling with the knobs, you can do of neat things, by selectively disregarding authoritative DNS records and substituting your own. all kinds Block a category of sites (ads, or porn for example) Block malicious domains non-malicious domains that point to malicious IPs and Block a specific service (Facebook, or TikTok) Block a specific TLD, FQDN, subdomain, or a wildcard entry (i.e. server-*.domain.com) Spoof a service, TLD, FQDN, subdomain, or all DNS queries to a specific IP address of your choice (think a wildcard-supporting hosts file) Redirect a specific service through a proxy location (BBC iPlayer through London, for example) Redirect a specific TLD, FQDN, subdomain, or wildcard entry through a proxy location (All .ca domains get proxied through Toronto) Redirect resolved DNS queries through a specific proxy location (there are over 100 cities to choose from) all Override blocks enforced by category filters, or services Schedule any of the above behavior to kick in at a certain time of day A whole bunch more TL;DR: Control D allows you to selectively disregard the authoritative DNS records associated with any domain you attempt to resolve (regardless of it actually existing in public DNS), and replace the answers with anything you want. This can prevent the domain from loading by spoofing  it to an IP of your choice (like 127.0.0.1), or to one of over 100 supported by Control D. Then, Control D will transparently proxy SNI (and some non-SNI) enabled traffic through servers in that location/country. exit locations You can also block, spoof, and redirect ALL of your DNS queries by using the catchall "Default Rule". Bad Use Cases You wouldn’t use a hammer to perform dental surgery (hopefully) - every tool has its use case. The same applies here, so before we jump into why you use a service like Control D, let's spend a moment talking about why you . should shouldn't Life Critical Anonymity If you live in a country where freedom of speech is non-existent, or you are a whistleblower, dissident or political activist, you should not use Control D to stay safe online. Despite Control D encrypting your DNS queries, even if you are using the proxy capabilities to spoof your location, the Server Name Indication (SNI) TLS extension is still transmitted in plaintext. This means that on adversarial networks where this information is captured and filtered, Control D will not provide you any security benefits. It will not be able to unblock restricted sites, and your browsing history could still be captured by the network administrator. If this is your particular use case, you are much better off using a (shameless self-plug as Windscribe is our sister company). Do keep in mind that a VPN is not a magical security solution either, despite what you may have heard from your favorite YouTuber while they react to videos of people describing the taste of meat to vegetarians. trusted, no-logging VPN Their scripted “opinions” are worth less than the Zimbabwean dollar.  A VPN is just tools that should be in your toolbelt if you care about these things.  I could rant all day about this, but let’s move on. one of the many BitTorrent Control D will not affect the BitTorrent protocol. Since this is a P2P protocol, which does not rely on DNS, all your torrent activity will be in the clear. Control D offers an optional filter that will block all common torrent indexes and trackers, which will make the use of the BitTorrent protocol difficult on the network where Control D is deployed (and this Filter enabled), but it will not 100% eliminate all torrent activity. If you wish to apply a layer of privacy to your torrent activities, you should use a VPN. Gaming (Improvements) Many people use VPNs for gaming in order to "improve ping" or mask their IP from trolls. The efficacy of this is , but Control D is unlikely to help here. In fact, there is a good chance that if you redirect all your traffic to our proxies, it will break some games entirely. debatable We recommend disabling Control D functionality if you experience problems playing your favorite games. You can do this from the section of the control panel; simply find your game and create a rule. Services BYPASS Actual Use Cases So now that we got that out of the way, what CAN you use Control D for? The following list is not exhaustive but should give you an idea of the flexibility you can achieve. Block Ads, Trackers, and Malware (and many other things) Yes, you can use a browser extension like uBlock that will do a really good job blocking things in your browser (until anyway, when we all eat a Google turd sandwich). January of 2023 However, this will just block ads in your favorite desktop browser. All tracking (and ad delivery) that happens outside of the browser (OS, installed programs, mobile applications, and browsers) will get through. If you set up Control D inside the operating system, or on your router, all DNS queries that any application makes (including the browser) will query your personal resolver, and all your rules apply. This means you can eliminate 99% of all ads and trackers (as well as 14 other categories)  from even loading on your devices. This effectively creates a network/system-wide ad block, which is highly effective. With this in mind, you can block many other categories, including: Malware distributing, typosquatting, and phishing domains (if you accidentally click a shady link) Adult content (handy on networks with kids) Social networks (stop wasting time looking at your "feed") IoT telemetry (prevent your fridge from talking to servers in China) Gambling, Drugs, Dating sites (keep your vices in check) A whole bunch more Help with Dev Efforts You’re probably familiar with the host’s file, which allows you to define domain names that may not exist in public DNS. You can also use it to override the DNS responses that are in public DNS to point to your local dev machine or a remote dev server. With Control D you can have a remote hosts file. This has several advantages: You can enforce the same host’s file on all your dev/test devices, including a mobile (or your smart toaster) Control D supports wildcards entries, so you can redirect all subdomains with a single rule Simulate regional access to your project from over 100 city-level locations Eliminate Your Pi-Hole Pi-Hole is a great tool to control your network, deploy custom blocklists, and have visibility on what is being resolved, but there are downsides: Pi-Hole only works on your home network – Control  D works on all networks, including cellular. You have to manage blocklists yourself – Our bespoke filters are built on top of community-maintained blocklists, with tens of thousands of false positives removed, based on feedback from 45M Windscribe users. We did all the work for you! Lack of advanced blocking – Pi-Hole only blocks domain names. Control D malware filter also blocks domains that resolve to malicious IPs from threat intelligence feeds. Lack of scheduling or flexibility – With Control D, your rules don't have to be static. They can change day to day, and be different on different devices. You have to run it yourself – Control D is deployed on top of a global anycast network, is available everywhere, including cellular networks, and supports Secure DNS protocols out of the box. (Selectively) Spoof Your Location Unlike a VPN, which sends all your activity to a single server in a chosen country, you have a lot more, well, when you use Control D. Since it operates at the DNS layer, you can create all kinds of rules in your configuration to do all kinds of things. control Here is a silly example of 7 different websites seeing you in 7 different places, all at once: You can instruct Control D to resolve different FQDNs through proxies in different countries, with as little as 3 clicks. When your browser asks to resolve a domain that has a rule, Control D will return a proxy IP instead of the true IP of the destination (disregarding the authoritative answer). It will read the SNI, and forward the end-to-end encrypted request to the site you wanted to load. From here, we can get fancier. Instead of creating individual rules, you can create a folder of rules, and assign an action (block, redirect, or bypass) to it. Any domain you add to this folder will inherit and apply the chosen folder rule. Don't want to make your own rules? That's where the Services section comes in. We've created rules for over 200 most common services online, and this list is always growing based on . user requests With a single toggle switch, you can apply a chosen rule to a video streaming service, an audio service, a store, a tool, a social network, or a game.  Each service can be redirected through a unique location (or blocked entirely), so you can appear to be in 69 countries, all at once. Lastly, you can redirect all your activity through the closest , if you don't care about the geolocation of your IP. This will have the best performance and supports IPv6 end-to-end. Control D Primary Location Optionally, you can choose a specific exit location (city), which will double hop your traffic from the closest Control D Primary Location to a Windscribe VPN server in the chosen country. All of this is without having to install a VPN app. You can use all 3 behaviors in parallel to create some very unique browsing profiles. The rule engine works as follows: Custom rules take precedence over everything Service rules are second in line, and are checked if there are no custom rules that match the DNS query Filters (which block things) are 3rd in-line and will match a domain if there is no overriding custom rule or Service rule Last in line is the Default Rule, which will, like the name suggests, match queries that aren't affected by any of the above Parental/Productivity/Impulse Controls The internet is full of distractions and outright harmful content. You may choose to block certain types of content at certain times of day (or all the time) so you can get stuff done, or prevent your kids from wasting time when they should be doing their homework. Control D gives you the tools to block large categories of distractions (social networks and games) so you can concentrate on your school or work. You can also use the same tools to block harmful content like porn, drugs, and malware on your network if you have kids. You can make these rules permanent, or time-based using the Scheduler. Here are some examples: Block social networks and games every Monday to Friday, and allow playtime on the weekends during certain hours Block porn, Reddit, and TikTok Monday to Friday 9 am to 5 pm so you can remain gainfully employed and boost your productivity British TV Fridays: Every Friday your Netflix and Disney+ shows British content Shopping Tuesday Mornings: All popular shopping sites are blocked every day except for 1hr on Tuesday morning. Then you can decide if you REALLY needed those $800 custom sock puppets you found at 3 am after a few drinks, and thought were super cool. Unique Device Configurations You're not limited to a single set of rules. Each account allows you to have up to 10 unique browsing profiles (configurations), and you can then create up to 10 devices and enforce your configurations using unique per-device DNS resolvers. Your personal laptop and phone can have one profile,  your partner's iPad could use another, and your kids' phones can use the rest. Each physical device can be configured to access or block a unique set of filters, services, and custom rules. Conclusion As you can see, there's a you can achieve with Control D, and this article really only scratches the surface. I recommend you (no payment details required!) and play around with it - I think you will be pleasantly surprised. It's time to take back control of your Internet experience. lot get yourself a trial account PS. The service is very much in active development, so if you have an idea or a suggestion, don’t hesitate to send us any and all feedback. We release new stuff weekly.

What Is DNS and Why You Should Control It

Gremlin is a leading software company focusing on in the market. It also has a tool similar to which belongs to Netflix, but is more customized to test the system with random loads or scheduled shutdowns. In the article below we will be testing a simple Kubernetes cluster running on EKS with Chaos Test. chaos-test Chaos Monkey Why is Chaos Test Important? Chaos Engineering is used to improve system resilience. Gremlin’s helps to find weaknesses in the system before problems occur. “Failure as a Service” Overview To successfully experience Chaos Engineering with Gremlin, firstly we have two requirements: a running EKS cluster and two applications deployed to EKS. This tutorial helps to produce the requirements and create a scenario to 'simulate an attack with Gremlin'. Step 1 - Prepare Cloud9 IDE Step 2 - Create an EKS cluster using eksctl Step 3 -  Deploy Kubernetes Dashboard Step 4 - Install Gremlin using Helm Step 5 - Deploy a Microservice Demo Application Step 6 - Run a Shutdown Container Attack using Gremlin Prerequsities 1. An AWS account 2. A Gremlin account which can be registered from here Step 1 - Prepare Cloud9 IDE Firstly, let's start to create the Cloud9 environment. Login to your AWS account and navigate Cloud9 to service page. Click on and type the name with anything, in this example we have chosen . Keep all default settings as they are stated since it is needed only to reach EKS resources. Get Started chaous gremlin Wait for a while for the new console to build. Then close all terminals to open a new one. To start creating the cluster, firstly check whether AWS CLI is installed or not with the command below: Administrator:~/environment $ aws --version
aws-cli/1.17.5 Python/2.7.16 Linux/4.14.158-101.185.amzn1.x86_64 botocore/1.14.5
Administrator:~/environment $ Step 2 - Creating an EKS cluster using eksctl 1. We will use to create our EKS clusters. eksctl curl --silent --location | tar xz -C /tmp


sudo mv -v /tmp/eksctl /usr/ /bin "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_ _amd64.tar.gz" $(uname -s) local 2. Making executable eksctl Administrator:~/environment $ sudo chmod +x /usr/ /bin/eksctl
Administrator:~/environment $ local 3. Confirming whether the command works: eksctl Administrator:~ $ eksctl version
[ℹ]  version.Info{BuiltAt: , GitCommit: , GitTag: }
Administrator:~ $ "" "" "0.13.0" 4. Below we are creating the cluster named with three EC2 nodes. Just a word of warning - EKS can cost a lot so please do not forget to delete your resources after you have done with your failure test. gremlin-eksctl eksctl create cluster --name=gremlin-eksctll --nodes=3 --managed --alb-ingress-access --region= ${AWS_REGION} It might take around 15-30 minutes to get ready which you can cluster on the EKS service page. - Fee of EKS is 0,20$/h and fee of EC2 with a m5.large instance type that EKS runs on is 0.096$/h. Estimation of total cost per day will be around 11-12$. Quit Tip 5. Checking the cluster whether it is working and getting its status. As expected, there is only one cluster created. Administrator:~/environment $ eksctl get clusters
NAME            REGION
gremlin-eksctl  us-east-1
Administrator:~/environment $ 6. Updating the file by giving the cluster name and region via AWS CLI tool. kubeconfig Administrator:~/environment $ sudo aws eks --region us-east-1  update-kubeconfig --name gremlin-eksctl
Updated context arn:aws:eks:us-east-1:312867154612:cluster/gremlin-eksctl /root/.kube/config
Administrator:~/environment $ in 7. On checking we see 3 nodes in the cluster. Two of them are worker-nodes while the third is cluster management master-node. Administrator:~/environment $ kubectl get nodes
NAME                            STATUS   ROLES    AGE   VERSION
ip-192-168-14-59.ec2.internal   Ready       59m   v1.14.7-eks-1861c5
ip-192-168-33-12.ec2.internal   Ready       58m   v1.14.7-eks-1861c5
ip-192-168-49-55.ec2.internal   Ready       58m   v1.14.7-eks-1861c5
Administrator:~/environment $ Step 3 - Deploying Kubernetes Dashboard Next we would deploy the Kubernetes dashboard to Kubernetes cluster by using and . These two tools will help our sample application to be shown in the dashboard. We will start with deploying our Kubernetes dashboard as the first step. Heapster InfluxDB Heapster is a performance monitoring and metrics collection system compatible with Kubernetes (versions 1.0.6 and above). It allows for the collection of not only performance metrics about your workloads, pods, and containers, but also events and other signals generated by your cluster. The great thing about Heapster is that it is fully open source as part of the Kubernetes project, and supports a multitude of backends for persisting the data, including but , , , and . Heapster not limited to Influxdb Elasticsearch Graphite InfluxDB InfluxDB is a time series database designed to handle high volume of writing and query loads. 1. Deploying the Kubernetes Dashboard to your EKS cluster: Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
Administrator:~/environment $ 2. Firstly, access to cluster is given since it is deployed to our private cluster while we need to access it via a proxy. This can be done through as the preferred proxy tool. In your workspace, run the following command: Kube-Proxy Administrator:~/environment $ kubectl proxy --port=8080 --address= -- -filter= &
[1] 336
Administrator:~/environment $ W0125 10:28:19.746961     336 proxy.go:140] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on [::]:8080

Administrator:~/environment $ '0.0.0.0' disable true 3. Deploying : Heapster Administrator:~/environment $  kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
serviceaccount/heapster created
deployment.extensions/heapster created
service/heapster created
Administrator:~/environment $ 4. Deploying : InfluxDB Administrator:~/environment $  kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
deployment.extensions/monitoring-influxdb created
service/monitoring-influxdb created
Administrator:~/environment $ 5. Creating cluster role binding for the Dashboard. Heapster Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
clusterrolebinding.rbac.authorization.k8s.io/heapster created
Administrator:~/environment $ 6. The next step is to create an service account. It will let you connect to the Kubernetes Dashboard with permissions. eks-admin admin To authenticate and use the Kubernetes Dashboard: Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/tammybutow/eks-aws/master/eks-admin-service-account.yaml
serviceaccount/eks-admin created
clusterrolebinding.rbac.authorization.k8s.io/eks-admin created
Administrator:~/environment $ 7. To access the Kubernetes Dashboard: a- In your Cloud9 environment, click to open the Dashboard URL. Tools > Preview > Preview Running Application b- Append the following to the end of the URL: /api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 8. Open a new Terminal tab on Cloud9 and enter the following: aws eks get-token --cluster-name gremlin-eksctl | jq -r '.status.token' 9. Select and then copy the output of the command above and paste it to the text field as shown below: Token Step 4 - Installing Gremlin using Helm 1. Download your Gremlin certificates: Start by signing-in to your Gremlin account. If you don't have one, create an account . here Navigate to and click on your . Click the button to download and save certificates to your local drive. Please note that the downloaded contains both a and a matching . Team Settings Team Download certificate.zip public-key certificate private key Unzip the and save it to your Gremlin folder on your desktop. Rename your certificate as and key files as . certificate.zip gremlin.cert gremlin.key 2. Creating Gremlin Namespace: Administrator:~/environment $ kubectl create namespace gremlin
namespace/gremlin created
Administrator:~/environment $ Create a Kubernetes Secret for your certificate and private key, copy and to Cloud9. A quick tip is to create these by the Vim Editor instead of copying from your local computer. gremlin.cert gremlin.key Administrator:~/environment $ kubectl create secret generic gremlin-team-cert \
> \--namespace=gremlin  \
> \--from-file=/home/ec2-user/environment/gremlin.cert \
> \--from-file=/home/ec2-user/environment/gremlin.key
secret/gremlin-team-cert created
Administrator:~/environment $ Check the files on Dashboard whether they are deployed. 3. Installation with Helm The simplest way of installing the Gremlin client on your Kubernetes cluster is to use Helm. Once Helm is installed and configured, the next steps will be to add the Gremlin and to install the client. repo A. Installing Helm source code and making it executable: Administrator:~/environment $ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7164  100  7164    0     0  45630      0 --:--:-- --:--:-- --:--:-- 45341
Administrator:~/environment $ 
Administrator:~/environment $ chmod +x get_helm.sh
Administrator:~/environment $ 
Administrator:~/environment $ ./get_helm.sh
Downloading https://get.helm.sh/helm-v2.16.1-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/ /bin
helm installed into /usr/ /bin/helm
tiller installed into /usr/ /bin/tiller
Run to configure helm.
Administrator:~/environment $ local local local 'helm init' B. Configuring Helm to access with RBAC Helm relies on a service called which requires special permission on the Kubernetes cluster, for which we will need to build a for using . Next step is to then apply this RBAC to the cluster. Tiller Service Account Tiller C. Creating a new service account: Administrator:~/environment $ cat < ~/environment/rbac.yaml
> ---
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: tiller
>   namespace: kube-system
> ---
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRoleBinding
> metadata:
>   name: tiller
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: cluster-admin
> subjects:
>   - kind: ServiceAccount
>     name: tiller
>     namespace: kube-system
> EoF
Administrator:~/environment $ D. Applying configurations: Administrator:~/environment $ kubectl apply -f ~/environment/rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
Administrator:~/environment $ E. Installing for : Tiller Helm Tiller A companion server component, , that runs on your Kubernetes cluster, listens for commands from , and handles the configuration and deployment of software releases on the cluster. tiller helm Administrator:~/environment $ helm init --service-account tiller
Creating /home/ec2-user/.helm 
Creating /home/ec2-user/.helm/repository 
Creating /home/ec2-user/.helm/repository/cache 
Creating /home/ec2-user/.helm/repository/ Creating /home/ec2-user/.helm/plugins 
Creating /home/ec2-user/.helm/starters 
Creating /home/ec2-user/.helm/cache/archive 
Creating /home/ec2-user/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding repo with URL: http://127.0.0.1:8879/charts has been configured at /home/ec2-user/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/ Administrator:~/environment $ local local $HELM_HOME 'allow unauthenticated users' #securing-your-helm-installation This will install into the cluster and will give access to in your cluster. Please note the security policy alert as shown above, which you can feel free to ignore or follow as per your policy settings. tiller managed resources F. Activating bash-completion for Helm: Administrator:~/environment $ helm completion bash >> ~/.bash_completion
Administrator:~/environment $ . /etc/profile.d/bash_completion.sh
Administrator:~/environment $ . ~/.bash_completion
Administrator:~/environment $ G. To run the Helm install, you will need your Gremlin Team ID. It can be found in the Gremlin app on the page, where you downloaded your certificates earlier. Click on your in the list. The ID you’re looking for can be found under as . Team Settings Team Configuration Team ID H. Export your Team ID as an variable: environment Administrator:~/environment $ GREMLIN_TEAM_ID=13ba2df0-5b0b-572c-a479-8136af79da66
Administrator:~/environment $ export I. Next, export your cluster ID, by giving a name for your Kubernetes cluster. Administrator:~/environment $ GREMLIN_CLUSTER_ID=boraozkanchaos
Administrator:~/environment $ export J. Now add the Gremlin Helm repo, and install Gremlin: Administrator:~/environment $ helm repo add gremlin https://helm.gremlin.com has been added to your repositories
Administrator:~/environment $ "gremlin" Administrator:~/environment $ helm install gremlin/gremlin \
> \--namespace gremlin \
> \--name gremlin \
> \-- gremlin.teamID= \
> \-- gremlin.clusterID= NAME:   gremlin
LAST DEPLOYED: Sat Jan 25 12:37:45 2020
NAMESPACE: gremlin
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME             AGE
gremlin-watcher  0s

==> v1/ClusterRoleBinding
NAME  AGE
chao  0s

==> v1/DaemonSet
NAME     AGE
gremlin  0s

==> v1/Deployment
NAME  AGE
chao  0s

==> v1/Pod(related)
NAME                   AGE
chao-698b9fbfb4-5thjp  0s
gremlin-425wv          0s
gremlin-krdlt          0s
gremlin-l252z          0s

==> v1/ServiceAccount
NAME  AGE
chao  0s


Administrator:~/environment $ set $GREMLIN_TEAM_ID set $GREMLIN_CLUSTER_ID Step 5 - Deploying a Microservice Demo Application The demo environment we are going to deploy on to our EKS cluster is the Hipster Shop: Cloud-Native Microservices Demo Application 1. Clone repo of app source code: Administrator:~/environment $ git https://github.com/GoogleCloudPlatform/microservices-demo.git
Cloning into ...
remote: Enumerating objects: 30, .
remote: Counting objects: 100% (30/30), .
remote: Compressing objects: 100% (21/21), .
remote: Total 2987 (delta 16), reused 16 (delta 7), pack-reused 2957
Receiving objects: 100% (2987/2987), 5.08 MiB | 38.24 MiB/s, .
Resolving deltas: 100% (2039/2039), .
Administrator:~/environment $ clone 'microservices-demo' done done done done done 2. Change directory to the one just created: Administrator:~/environment $ microservices-demo/
Administrator:~/environment/microservices-demo (master) $ cd 3. Deploying the application: Administrator:~/environment/microservices-demo (master) $ kubectl apply -f ./release/kubernetes-manifests.yaml
deployment.apps/emailservice created
service/emailservice created
deployment.apps/checkoutservice created
service/checkoutservice created
deployment.apps/recommendationservice created
service/recommendationservice created
deployment.apps/frontend created
service/frontend created
service/frontend-external created
deployment.apps/paymentservice created
service/paymentservice created
deployment.apps/productcatalogservice created
service/productcatalogservice created
deployment.apps/cartservice created
service/cartservice created
deployment.apps/loadgenerator created
deployment.apps/currencyservice created
service/currencyservice created
deployment.apps/shippingservice created
service/shippingservice created
deployment.apps/redis-cart created
service/redis-cart created
deployment.apps/adservice created
service/adservice created
Administrator:~/environment/microservices-demo (master) $ 4. Wait until pods are in a ready state. Administrator:~/environment/microservices-demo (master) $ kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
adservice-84449b8756-qj4sp               1/1     Running   0          4m57s
cartservice-6cbc9b899c-ww5rf             1/1     Running   0          4m58s
checkoutservice-56b48b77c8-fszpx         1/1     Running   0          4m58s
currencyservice-b9fcb4c98-hx9kq          1/1     Running   0          4m58s
emailservice-797cdcc76d-d257v            1/1     Running   0          4m58s
frontend-785c44fd98-zvgj4                1/1     Running   0          4m58s
loadgenerator-665c4ddb74-xwm8m           1/1     Running   3          4m58s
paymentservice-84d7bf956-fdgxr           1/1     Running   0          4m58s
productcatalogservice-5664f59f54-b5mk5   1/1     Running   0          4m58s
recommendationservice-7f9855d7c6-b2zv6   1/1     Running   0          4m58s
redis-cart-6448dcbdcc-97d55              1/1     Running   0          4m58s
shippingservice-6b6f49747d-svb5f         1/1     Running   0          4m58s
Administrator:~/environment/microservices-demo (master) $ 5. Getting the frontend IP address: Administrator:~/environment/microservices-demo (master) $ kubectl get svc frontend-external -o wide
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)        AGE     SELECTOR
frontend-external   LoadBalancer   10.100.226.157   a0c9bed9a3f7111eab7c912c03cd100e-265884000.us-east-1.elb.amazonaws.com   80:30461/TCP   6m31s   app=frontend
Administrator:~/environment/microservices-demo (master) $ 6. Visit the URL on your browser: Step 6 - Running a Shutdown Container Attack using Gremlin We are going to create our first Chaos Engineering experiment where we would validate the EKS reliability. Our hypothesis is, “After shutting down my container, we will not suffer from downtime and EKS will give us a new one.” cart service Going back to the Gremlin UI, select from the menu on the left and select . We’re going to target a Kubernetes resource, so click on on the upper right. Attacks New Attack Kubernetes Choose : cartservice Choose and : State Shutdown Attacking this pod with our Gremlin UI: We will be shutting down the containers. As a test, we attacked twice and the cartservice pods restarted itself. Which signifies that it is working as expected. Note that it re-generates itself even when you attack to shut down your pods. cartservice When we attacked our containers, the cluster resisted to failure and restarted itself, which symbolizes that our system is now resistant to failure. We have seen what happens when a failure occurs, in this example the failure is shutting down the pods. As a result, we understand that our cluster already has auto-scaling feature. As a reminder; do not forget to delete your and ide. cluster Cloud9 eksctl delete cluster --name=gremlin-eksctl Conclusion Congrats! You’ve installed an AWS EKS cluster, deployed the Kubernetes Dashboard, deployed a microservice demo application, installed the Gremlin agent as a daemon-set, and ran your first Chaos Engineering attack to validate Kubernetes reliability! This article was originally published on https://appfleet.com/blog/chaous-test-to-your-kubernetes-with-gremlin/ .

The Secret To Succesully Simulate An Attack With Gremlin To Test Kubernetes

Introduction So you've decided that with your next project you are going to tread the RESTful path. But to be smart about it, you need an intelligent tool to ensure quality and productivity. With the abundance of options available, let's thumb through some of the developers' favorites. First things first - API definition is a popular choice with graphical visualization. APIs can be defined either in JSON or YAML. Additionally, you can generate a server-side or client-side stub for most of the relevant technologies (from ASP.net core and Kotlin to Nodejs and Python). Swagger Editor is an open-source project, and elegant tool with support for and YAML. What you get: Apicurio simple JSON Importing APIs from source control, file, URL Integration with your GitHub, GitLab or Bitbucket account Previewing live documentation Visual editors for designing APIs on Open API standard (no coding skills required!) Watch it go! - Tests and monitors is an intuitive and well-organized desktop tool. It is one of my personal favorites as it was the first-ever tool that I used to test my own APIs and the learning curve couldn't have been smoother. The product is well-rounded as it covers the API development from to and . Postman creation testing monitoring With Postman, you can: add mock servers, switch between environmental setups, generate documentation, apply custom styling. Medium and large teams will appreciate the personal and team workspaces, and that work similarly to that of any version control system like . You can also add a basic API monitor (runs on a minute, hour or week basis) and get about failures. fork merge features GitHub email notifications is a powerful . Minimalistic in appearance, it allows the creation of projects for HTTP API calls. After you fill in the request and response data, test code is auto-generated. You can also create in JavaScript (or CoffeeScript) that are invoked before each test. Ping API API inspector set-up scripts It has a few basic for parsing, decoding and encoding URIs. In case of failures, a notification is sent via email, Slack or Hipchat. It's especially useful for web developers whose sites need to be responsive around the globe. Tests check APIs responses from 5 locations in 3 continents: in U.S.A., Japan, Germany and Singapore. You can to run tests from 1 minute to a full day. built-in functions schedule is an easy-on-the-eye monitoring tool with and . API Science lets you check , , and of your or third-party APIs and automatically notifies about failures. The dashboard is straightforward to use and intuitive. API Science graphs charts health availability performance You can upgrade your monitoring with to run with tests (e.g. to load or parse data from JSON) or generate a report (not downloadable). A nice touch is the import option from other popular API-related tools: Postman, Swagger, RAML, HAR. Sad news for the Southern Hemisphere, servers that are located in the US, Japan and Ireland. custom JavaScript scripts -  is an advanced platform for for tight integration with your CI/CD pipeline. ReadyAPI is a combination of 3 tools available for separate licence purchase: SoapUI Pro (functional testing), LoadUI Pro (performance testing), ServiceV Pro (service virtualization). They describe themselves as "the Most API Testing Tool on the Market", and they might be right. ReadyAPI functional, security, and load testing Advanced ReadyAPI options are vast and can be intimidating to learn at first. It's not cheap as it's aimed at large companies with large products and complex tests. The projects integrate with several CI/CD tools: Git, Docker, Jenkins, Azure DevOps, TeamCity, and more. Good news for tight budgets: there are available for Windows, Linux and Mac, however, the tests are slower. open-source free versions is a desktop app for debugging your API spec with Git sync option. The "Design" section offers with detailed error messages. The auto-generated preview might remind API-navvies of Swagger Editor. In the "Debug" section you can tinker your API with: Insomnia style validation with cookies, query and header, body of request is filled in editor; Sending request Environments configured as JSONs; Support for multiple : Basic, Digest, Bearer Token, Microsoft NTLM, OAuth, AWS IAM, Hawk, Atlassian ASAP, Netrc File; authorization options Client-side code snippet generator for multiple languages, some with different library implementations. The platform can be extended with made by Insomnia community. The plugins cover JWT and token generators, environmental setups, XML-JSON translators, randomizers and UI themes. With plans, teams can collaborate and host the project in the . installable plugins paid cloud is a free tool dedicated to testing HTTP and REST APIs. The significant functionalities are: Talend API Tester Sending requests and inspecting responses Validating API behaviour Importing Postman, Swagger, OAS, OpenAPI, HAR test collections With API Tester you can create dynamic requests with custom variables, security and authentication and keep them organized as projects or services. You can run assertions against the responses of different formats (JSON, XML, HTML and others). are created in visual editors. Assertions Bringing the API out to the world is an online platform that enables you to , , APIs and expose its functionalities via a portal for your consumers. From all the online tools mentioned in this smackdown, Apigee set itself apart from the others with the most elegant web experience, analytical tools, proxy creator, and . Apigee build manage consume sweet website generator If you need to adopt a SOAP service into a REST one, or you consume API but would like to expose API's backend services via a middleman, you are presented with proxy wizards with lots of policy options (quotas, authentications, data mediation, caching). Apigee offers the consumer, neat, modern and straightforward portals including interactive sites for your APIs. The portal templates, with CSS themes, HTML and user management premade is ready to be customized or published right away. To ensure world-class API performance, their and charts cover: auto-generated documentation analytics tool latency, response time, geographical distribution of API usage, proxy performance, error code analytics. Putting the moving parts together is a sophisticated developer tool for , and . The main use scenarios are: Linx backend integrations automations web services Building, hosting and with one platform, managing multiple APIs Consuming and orchestrating data across many technology end-points, Accelerating API Development, Simplifying existing APIs. Linx is particularly powerful for creating or calling internal and external REST web services. It is on the fastest ways to extend an API that requires (e.g. you need to load or push data into a database, conjure up a rapport, push updates to notification steam, modify or encrypt data at any point, etc). integration with other technologies Alongside REST, Linx offers several other features from database to cryptography and cloud, which creates tons of possibilities for integrating your API with multiple different technologies. The Linx IDE Designer is a great, free option for carrying out tests and service calls. If you want to monitor and automatize the test suits or schedule calls either in , you will need to upgrade to the paid Linx Server. on-premise or cloud Feature smackdown Still at a lost? Use the comparison table below to help yourself decide. Conclusion With REST's and APIs' taking root in the digital world, one can only expect more tools to appear. Now that you had a glimpse of options above, tailored for different needs of various size teams, projects and capabilities. If you are not happy with your current choice, it's easy enough to hop and experiment. Let me know your favorites in the comments below. Happy hunting.

REST API Tools Smackdown

Bypassing SSL Pinning with LLDB on AppStore iOS apps [A Step by Step Guide]

Maintaining the relevance and accuracy of data is the top task for e-commerce aggregator websites. If the aggregator fails to do this , its main advantage disappears – people use aggregators to find the most relevant information in one place. Web scraping techniques are the best way to ensure your aggregated data is up-to-date and accurate. To run scraping, you need specialized software, called a crawler. This bot visits the required websites, parses their data, and uploads this information to the aggregator website. The problem is that very often the owners of the websites you want to scrape are not happy about this. If their data is presented on an aggregator website along with information from competitors, there is no need for the end-user visit  this original website anymore. This  harms sales. So, for this very reason ,website owners try to block crawling bots, which is easily understandable. How websites block scraping bots The companies that operate e-commerce websites try to prevent crawlers from parsing their websites' data. This is usually done by blocking requests from such bots based on an IP. Crawling software often uses so-called data center IPs, which are easy to detect and block. Also, besides just blocking, website owners can trick bots by displaying fake and irrelevant information. For example, they can display prices that are higher or lower than reality, or altered goods descriptions. One of the most common examples of this is in the airplane ticket industry. Here airlines and travel agencies often try to trick aggregators and show them different data based on the aggregator’s IP. Let's look at a real use case by searching for an airplane ticket from Miami to London on a specific day. We've sent one request from an IP in Eastern Europe, with a second one sent through an IP in Asia. Here is the price for the request from the Eastern European-based IP: And here is the search result for the Asian IP: As you can see, the same ticket can display different prices. A difference of $76 is huge for just one ticket. Such situations are horrible for any aggregator website – if the aggregator displays the wrong information, there will be no reason to use it. Also, when the aggregator shows one price, if after clicking on it the user is forwarded to a website with a different price, this harms the project's reputation. The solution: residential proxies To avoid web scraping problems and blocks, you can use residential proxies. Datacenter IPs are assigned by hosting providers. It is not that hard to understand that the specific IP belongs to a particular provider. It can be done based on the ASN number, where this information is stored. There are a lot of tools for automated ASN analysis, which are often integrated with antibot systems. As a result, crawlers are easily detected and blocked, or tricked by fake data. Residential proxies can be helpful here. They use IP addresses assigned to regular users (homeowners) by their ISP. These addresses are marked in corresponding internet registries. If you use a residential proxy like , all requests sent from that particular IP will be indistinguishable from the ones submitted by regular users. In this case, no one tries to block or trick potential customers, especially in the area of e-commerce. Infatica As a result, using rotating proxy tool helps aggregators to ensure the accuracy and consistency of their data, while at the same time avoiding blocks and other problems.

How To Fix Your Aggregator Website Using Residential Proxies

Discover the technologies driving the realtime data ecosystem Realtime data currently enables some of our most beloved apps: Slack, Google Maps, Uber, Lyft… the list goes on. It has become essential for our time-sensitive, reactive lifestyles, where even a few seconds of delay causes us substantial grief. Realtime has become a cornerstone of big data and IoT, where the faster the data is received, the faster it can be processed and made actionable. The difference between your app succeeding or failing will come down to its reliability, functionally, and ultimately its speed. “[Realtime] is about the ability to make better decisions and take meaningful actions at the right time. It’s about detecting fraud while someone is swiping a credit card, or triggering an offer while a shopper is standing on a checkout line, or placing an ad on a website while someone is reading a specific article. It’s about combining and analyzing data so you can take the right action, at the right time, and at the right place.” ― , Mike Barlow Real-Time Big Data Analytics: Emerging Architecture For the past few years, Silicon Valley has capitalized on the rising demand for realtime data by developing both realtime APIs and services. These solutions allow you to easily stream data to multiple clients (devices like phones, laptops, other servers), and are typically more out-of-the-box infrastructure services. Realtime APIs can drop into existing applications and works great with serverless or microservice backends. On the other hand, realtime app services as much more client focused, like delivering realtime updates to an app. This list will outline a few realtime technologies and hopefully give you, the app developer or aspiring entrepreneur, more insight into the technological landscape. Keep in mind, these aren’t listed in any particular order because they all depend on your particular use cases and stack. Apache Kafka One of the most popular distributed streaming platforms, is used for building realtime data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, fast, and runs in production in thousands of companies. It has two main use cases: Kafka Building realtime streaming data pipelines that reliably get data between systems or applications Building realtime streaming applications that transform or react to the streams of data Pubnub is one of the industry leaders in realtime services. They frame themselves as a global (DSN) and realtime infrastructure-as-a-service (IaaS) company, which focuses on streaming updates to web, mobile, and IoT applications. PubNub focuses on data streaming using a publish/subscribe model for devices, which harnesses communication patterns already in the browser. One of their main strengths is the breadth of their SDK and integration libraries, enabling realtime for almost any stack. Pubnub Data Stream Network Fanout/Pushpin Funded by , is a drop-in realtime API that lets you push data to connected devices instantly. Because it serves as a proxy and can work with any backend language, you do not have to substantially reconfigure your current API inner-workings. Fanout is different than most realtime providers because it supports API development, whereas the others do not. Hence, this product is ideal for developers that want to do more advanced application control, like use Fanout with their API management tool or to manage microservices. is the open source version of Fanout. Since Pushpin is a proxy server, the backend can be written in any language, which makes integration very easy. Moreover, it works nicely with any language or framework, including any existing API management services that you use. Seed Sumo Fanout.io Pushpin Pusher Similar to Pubnub, focuses on data streaming for devices that uses a publish/subscribe model. They provide over 30 libraries to help with integration — including developer-friendly features for debugging and access controls. One of the main differences between Pubnub and Pusher is that Pusher uses websockets, whereas Pubnub does not. You should definitely ask your developers which approach would be better for your app. Pusher Realtime Messaging by Realtime.co Realtime Messaging is a cloud based message broker that enables developers to build cross-platform apps that require realtime communication between devices. It offers quick-start guides for JavaScript, Android, and iOS — and SDKs for all major platforms. One of the main benefits is that is handles your auto-scaling for you, so you can grow from 1 device to millions. Ably An alternative to PubNub, ’s platform ensures interoperability between clients using a native realtime protocol, and clients using other realtime or queuing protocols. Ably uses a publish and subscribe service to organize the message traffic within applications into named channels. Clients (laptops, phones, servers) attach to channels to subscribe to messages, and every message published to a unique channel is broadcast by Ably to all subscribers. Ably Socket is an awesome open source project. It is a JavaScript library for realtime web applications. It has two parts: a client-side library that runs in the browser, and a server-side library for Node.js. Systemically, it is similar to Pusher insofar as it relies on websockets (with polling as a fallback). Socket’s getting started guide will show you how to create lots of amazing applications in fewer than 200 lines of code. Socket.io Firebase One of the fastest growing realtime DB’s, the Realtime Database is a cloud-hosted database where data is stored as JSON and synchronized in realtime to every connected client. All clients share one Realtime Database instance and automatically receive updates with the newest data. Firebase a flexible, expression-based rules language (Firebase Realtime Database Security Rules) to define how your data should be structured and when data can be read from or written to. Firebase Streamdata.io a SaaS API proxy tool that converts standard API requests into a streaming API. In other words, it provides a proxy as a service for any HTTP API by polling and acting as a streaming API. More specifically, it is a realtime cache proxy that allows you to poll JSON REST APIs and push updates to clients — keeping a history of change modifications that occur between pollings. This way, Streamdata.io is able to give you the list of modifications which happened since the last data request. Streamdata.io Layer provides developers with APIs, SDKs, and UI toolkits to integrate realtime messaging into your products. They aim to provide developers with the proper tools that matches the rapid development and user experience of leading consumer messengers such as iMessage. Layer is 100% managed infrastructure so developers can easily integrate messaging into their applications. In other words, they handle servers, scaling, performance, and encryption as well. They are HIPPA-complaint, designed for asynchronous, person-to-person messaging or business-to consumer uses cases as well. Here are the many services they offer: Layer Amazon Kinesis makes it easy to collect, process, and analyze so you can get timely insights and react quickly to new information. It allows you to process and analyze data as it arrives and respond in realtime instead of having to wait until all your data is collected before the processing can begin. One of the main benefits is that it is a fully managed platform, so you run streaming applications without requiring any infrastructure management. Amazon Kinesis realtime, streaming data Looking to get started with building realtime APIs or apps? Check out Getting Started with Building Realtime API Infrastructure

Adding Realtime Data Streaming To Your App

Eternal Storage : Building Dapps For the Future

Developers often fear database changes because a mistake by anyone on your team can lead to a major outage and even data loss. The stakes are higher when changes are not backwards compatible, cannot be rolled back, or impact system performance. This can cause a lack of confidence and slow your team velocity. As a result, database changes are a common failure point in agile and DevOps. Databases are often created created manually and too often evolve through manual changes, informal process, and even testing in production. Manual changes often lack documentation and are harder to review, test, and coordinate with software releases. This makes your system more fragile with a higher risk of failure. The solution is to include your database in your source control and CI/CD pipeline. This lets your team document each change, follow the code review process, test it thoroughly before release, make rollbacks easier, and coordinate with software releases. Let’s look at an example of how you can include your database in your CI/CD process and how to push a non-backwards-compatible database change successfully. We'll also look at testing your changes, progressive deployments, dealing with rollbacks, issues to look out for, and a few helpful tools. What is CI/CD? CI/CD is a cornerstone of modern development and DevOps. CI—or Continuous Integration—is the practice of merging all working developer code into a shared repository throughout the day. Its purpose is to prevent integration problems by integrating often and early. Commonly, this integration kicks off an automated build and test. CD—or Continuous Delivery—is the practice of building, testing, and releasing software in short cycles, with the aim of ensuring that a working version of the software can be released at any time. Is Your Database Ready For CI/CD? There are several key requirements to having your database ready for CI/CD. First, the database must be reproducible from scratch using one or more SQL scripts. This means that in addition to a script that creates the initial version of your database, you must also maintain scripts that make all required schema updates to your database. When you create these scripts, you have two options: Create one script per schema object, then update the corresponding script (state based) when making changes to the object. Create one original script that creates the entire database schema. Then, create a series of individual change scripts (migration based) for changes. To learn more, check out on state-based versus migration-based database updates. this excellent article The second requirement for CI/CD is that the database schema (meaning, those scripts we just mentioned), just like your source code, must live in source control. You must treat your database schema changes as a controlled process just as you do with code. Third, always back up before performing any database migrations. If you're working with a live production database, consider a for your migration or upgrade. Postgres follower database Lastly, changes that involve removing a database object, such as deleting a column as shown below, can be more difficult to deal with due to the loss of data. Many organizations develop strategies to deal with this, such as only allowing additive changes (e.g. adding a column), or having a team of DBAs that deals with such changes. Is Your Team Ready for CI/CD? Perhaps the best process for database changes and database CI/CD is ensuring you have a collaborative effort between DevOps and DBAs. Make sure your DBAs are part of the code review cycle; they can help to identify issues that only they may know about. DBAs have knowledge of the databases in each specific environment, including database specific dependencies such as ETL load jobs, database maintenance tasks, and more. Be sure to consult a database SME in setting up your database for CI/CD, and in any migration process, when possible. Be sure to also follow sensible DevOps processes, such as test your changes in a test environment, performing backups, mitigating risks, being prepared for rollbacks, and so on. How Your CI Tool Helps With Migrations When you create or update these scripts, and push them to source control, your CI tool (such as Jenkins or Heroku CI) will pull the changes and then: Rebuild your database to the newest version of the scripts in a test or staging environment. Since the database is being rebuilt, be sure to export the look up/reference data, then import it back to the new schema. Although it is possible to export and import transactional data, transactional data is out of scope for this article. You can if interested. read more about best practices here Run your tests. For testing your database changes, one possible time saver is to have two sets of tests. The first set is a quick test that verifies your build scripts and runs a few basic functional tests (such as referential integrity, stored procedures unit tests, triggers, and so on). The second set includes migration of transactional data (possibly scrubbed production data) to run a more realistic full set of tests. Deploy your database changes to your production environment or another selected environment. (Depending on your migration strategy, the CI tool should also simultaneously deploy and test any code changes dependent on the database change.) Watch Out for These Common Problems In many cases, when you're making a simple schema addition with bidirectionally compatible code, then you can push code and database changes at the same time. This shouldn't be an issue, as rollbacks in our case will be easy and predictable. This is often true when we are dealing with microservices with simple database components. However, in many scenarios, serious problems can happen with this simplistic approach: Production data may be different from test/stage data and cause unforeseen issues.A large number of changes in both code and database schema may be in the pipeline and need to be deployed simultaneously. CI/CD processes may not be consistent through every environment. You may be under a zero-downtime mandate. Even using tools that help you to achieve zero-downtime (such as Heroku preboot) you can end up with two versions of the code running simultaneously. There are several strategies for addressing the above issues. Some popular solutions include: If your changes are backwards-compatible, then use a tick-tock release pattern. This approach involves releasing the new database column then releasing the new code. You can identify problems early in this manner, with minimal production changes. Additionally, the rollback remains small and manageable, and can be accomplished with tools such as Heroku's Postgres rollback, as noted above. If your provider supports it, use a blue/green rollout. In this pattern, an entirely new set of production servers is created side-by-side with the current production servers. Enable database synchronization and use a DNS or a proxyto cut over to the new servers/database. You can rollback by simply changing the proxy back to the original servers. A Simple Migration Example Let’s run through an example based on the the migration scripting option as explained above. Note that some frameworks (Rails, Django, ORM tools, and so on) abstract out or handle schema creation and migration for you. While the details may differ according to the framework you are using, the below example should still help you to understand these core concepts. For example, you may have a schema configuration file to include in your CI/CD process. For our example, we'll use Node.js, Postgres, and GitHub. We'll also use Heroku because it provides convenient tools including with deploy scripts for CI/CD, and easy Postgres rollbacks in case we make a mistake. If you need help deploying Node.js and Postgres on Heroku, . Heroku CI here’s a quick walk-through Here's the pertinent code for our example. We're going to create a simple database with a single table, and a Node.js file that writes to that database table on load. Database creation SQL (we have just one simple table): ( PRIMARY ,
   firstname ( ) ,
   lastname ( ) ,
   enrolled ( ) ,
   created_at ); CREATE TABLE users id integer KEY varchar 40 NOT NULL varchar 40 NOT NULL char 1 NOT NULL date NOT NULL Node.js result = client.query( ,[ , , , , ()]); const await 'INSERT INTO users 
  (id,firstname,lastname,enrolled,created_at) 
  values ($1,$2,$3,$4,$5) ' 1 'Becky' 'Smith' 'y' new Date Once these files are checked into GitHub and our repository is attached to a Heroku app, we can enable the on the Heroku dashboard: Heroku CI tool The real work is done by the and the . Using those, we can tell the Heroku CI tool to run a database migration SQL file any time a new release is created (in other words, a successful compile). Here is the release line we need to include in the Heroku Procfile: Heroku Procfile Heroku release phase release: bash ./release-tasks.sh The content of the release-tasks file includes a list of SQL scripts to run. That list is updated with each release to include the needed schema modifications. For this very simple example, it will point to just one script: psql -h <hostname> -d <database> -U <user> -w -f database/migrate.sql (The database password can be supplied as a Heroku environment variable.) Typically, as we are using the migration-based strategy, we would add additional migration scripts for each set of changes. For a more robust solution, we could use a tool such as Liquibase, or These tools add version control to your database, both generating the necessary change scripts between releases, and giving you the ability to easily roll back changes. For example, Flyaway creates scripts that allow you to migrate from any version of your database (including an empty database) to the latest version of the schema. Alembic Flyway. To kick off the CI tool, we make two changes: drop a required column, and change the JavaScript to no longer reference that column. First, we update the SQL code in Node.js, taking out the column: result = client.query( ,[ , , , ()]); const await 'INSERT INTO users 
  (id,firstname,lastname,created_at) 
  values ($1,$2,$3,$4) ' 2 'Becky' 'Smith' new Date Next, we create a migrate.sql file (referenced in the Procfile above) to alter the table and remove the column: enrolled; ALTER TABLE users DROP COLUMN Now, we commit the code change and SQL file, and watch the CI magic. First, the integration tests run. If you are using a common testing framework, the Heroku CI tool probably works with your test suite. And now the CI tool creates a new release and deploys the app, which kicks off the migrate.sql file. (See the middle of the image below.) We can check to see that the column was removed by inspecting the database through the Heroku CLI tool: It worked! There is no longer a column named 'enrolled'. Our CI tool ran our script and deleted the column. Some tools, like Liquibase, keep a detailed list of database changes. These tools allow you to easily see the last set of changes in cases like the above. Now, any time that code or an updated migrate.sql is committed in the future, the CI tool will kick off the tests. If the tests pass, this creates a new release and pushes it to staging. When there is a new release, the migrate.sql file runs against the staging database. We've taken a simple route here for demonstration purposes, but could have made this process more robust. For instance, when moving a new release to staging, we could wipe out the old version of the database, create a new one from scratch running the original creation script plus all migration scripts, and then populate the database with any reference data all through the Procfile and release phase. Also note that for simplicity sake, we are not running this migration with transactions in progress. In a real-world scenario, to prevent concurrent migrations. Heroku recommends using an advisory lock How To Do Rollbacks Even with the best planning and forethought, there will be times when you need to roll back your database. There are many approaches to rolling back failed deployments. Create a SQL file that rolls back the changes quickly. (For example, while you are in staging, use a compare utility to generate the script.) This file should be part of the deployment package so that you can quickly run the rollback if there is an error. Roll forward (quickly push a new build that fixes the issue). Rely on source control and labels or branches to recreate and deploy the previous version. Restore a full backup of your database. (Use a tool that ships with your database, such as pg_restore in Postgres.) Use a tool provided by your platform, such as and for code. As the name implies, Heroku Postgres Rollback allows you to easily roll back your database to a previous point in time, quickly and confidently moving your database back to a working release. Heroku Postgres Rollback Heroku Release Rollback Be aware that all these solutions come with their own challenges, such as potential loss of new data (restoring a backup or redeploying) and introducing new bugs. Summary Database changes and migrations can be scary, and can cause serious mistrust. However, if you place your database under CI/CD controls, you can not only confidently migrate your changes, but also move towards a better agile and DevOps experience. This is can be as simple as using source control for your database schema, having a good process in place with your DevOps and DBA teams, and using your existing CI tools to test and migrate your databases. Once you establish and train your team on the new process, future changes will be smoother and more automatic than your old manual process.

Fear database changes? Get them under control with CI/CD

has become a very important part of our lives recently and making sense of that data is equally important. There is no point in having data if you cannot track or analyze it, especially if this data has anything to do with finances. Data That’s why we will be building an expense and income tracker chart, with realtime features using . Our interactive dashboard will have a line chart that displays your income and expenses for each day. You’ll be able to add new expenses and income and see the chart update in real time. Pusher The dashboard chart will be powered by + as the backend server and + for the frontend bootstrapped by . Node.js Express Vue vue-chartjs vue-cli Scaffolding the app with vue-cli vue-cli is a simple CLI for scaffolding Vue.js projects. We’ll install vue-cli and then use it to bootstrap the app using the webpack template, with the following commands: npm install -g vue-cli

vue init webpack-simple realtime-chart-pusher Tip: The webpack-simple template is a simple webpack + vue-loader setup for quick prototyping. You can read more about that here . Setting up Node.js Server Next thing to do is set up a server that will help us communicate with Pusher. I’m going to assume that both Node and npm are installed on your system. We will then install the dependencies we will be using for the Node server. npm install body-parser express nodemon pusher Tip: nodemon will watch the files in the directory in which nodemon was started, and if any files change, nodemon will automatically restart your node application. One more thing, we are going to need an entry point/file for our Node server. We can do that by creating a file in the root of the app. server.js Pusher Setup To implement the realtime functionality, we’ll need the power of Pusher. If you haven’t already, and create a new app. When your new app is created, get your app_id, keys and cluster from the Pusher dashboard. sign up for a Pusher account App Setup Now that we have a Pusher account, and have installed the dependencies needed for the Node.js backend, let’s get building. Let’s write code for the file. server.js Let’s have a look at what’s happening here. We require Express, path, body-parser and Pusher and we initialized express() with app. We use body-parser to extract the entire body portion of an incoming request stream and expose it on req.body. Pusher is also initialized with the app credentials and cluster from your dashboard. Make sure to update that, or else the Node server will have no connection to the dashboard. Lastly, the Node server will run on the 5000 port. Next thing to do is define our app’s route and also add mock data for the expenses and income chart. Update your server.js file with the following. First, we have an expensesList object with the data containing expenses and income for particular days. app.get('/finances', (req,res) => {

  res.send(expensesList);

}); This route simply sends the expensesList object as JSON. We use this route to get the data and display on the frontend. The route sure does a lot. It’s a POST route, which means we will be expecting some incoming data (in this case, expense amount and income amount). /expense/add We then push this new income and expense to the existing one, after which we also push the updated expensesList to Pusher. Lastly, we send a JSON as a response to the route, containing the latest income, expense, date and updated expensesList. Your final should look like this: server.js Building the Frontend (Vue + vue-chartjs) Most of the frontend work will be done inside the folder. Navigate to that directory and you should see a file. You can either delete that file or rename to as we will be needing a file inside the components folder. src/components Hello.vue Home.vue Home.vue Before we get started with building the chart and displaying it, there are a couple of things we need to do. Open up the file in the folder and replace with the following code: App.vue src Next, we will install , , (Pusher’s library) and (We’ll use axios to make API requests). and then add them to the app. vue-chartjs momentjs pusher-js Javascript axios Vue.js npm install axios vue-chartjs pusher-js moment Once that’s done, we’ll import axios and register it globally in our app. We can do that by opening the file in the folder. main.js src Next, let’s create a Vue.js component that will help to display our chart. We are going to use this to specify what type of chart we want, configure its appearance and how it behaves. We’ll then import this component into the component and use it there. This is one of the advantages of , it works by importing the base chart class, which we can then extend. Let’s go ahead and create that component. Create a new file called inside the folder, and edit with the code below. Home.vue vue-chartjs LineChart.vue src/components In the code block above, we imported the Line Chart from vue-chartjs and the mixins module. Chart.js ordinarily does not provide an option for an automatic update whenever a dataset changes but that can be done in vue-chartjs with the help of the following mixins: reactiveProp reactiveData These mixins automatically create as a prop or data and add a watcher. If data has changed, the chart will update. Read more . chartData here Also, the function inside the function is responsible for rendering the chart. is an object containing the dataset needed for the chart and we’ll get that by including it as a prop in the template, contains the options object that determines the appearance and configuration of the chart. this.renderChart() mounted this.chartData Home.vue this.options We now have a component, but how can we see our chart and test its realtime functionality? We do that by adding the to our component as well as subscribing to our Pusher channel via pusher-js. LineChart LineChart Home.vue Open up the file and edit with the following: Home.vue fillData This function gets called immediately the app is and it basically makes an API request to the Node backend ( /finances) and retrieves the expensesList. mounted A GET request is made to the route which in turn returns the latest and we then manipulate that data with Javascript’s and assign it to various variables. /finances Node.js expensesList .map addExpenses The code block above simply utilizes a POST method route to to update (Remember route in the Node server sends the updated to the Pusher Dashboard) along with the income, expense and date data. /expense/add expensesList /expense/add expensesList It then uses the data gotten from via to build the Line Chart again and adds the new entry automatically to the Chart. Pusher channel.bind fetchData This function gets called after the Vue instance is and it also listens for changes to the Chart’s dataset via Pusher and automatically updates the Line Chart. created Your final Home.vue file should look like this: One more thing! Before we can run our app, we need to do something called . API proxying allows us to integrate our vue-cli app with a backend server (Node server in our case). This means we can run the dev server and the API backend side-by-side and let the dev server proxy all API requests to the actual backend. API proxying We can enable API proxying by editing the option in . You can edit with the code below. dev.proxyTable config/index.js After that has been done, we are finally ready to see our app and you can run to start the app. npm run dev That’s it! At this point, you should have a realtime dashboard chart that updates in realtime. You can check out the live demo or go to the code for the whole app, which is hosted on . here Github for your perusal Conclusion We’ve seen how to build a basic Line Chart with ChartJS in Vue with the help of vue-chartjs and also added realtime features thanks to Pusher. Then we saw how to use reactiveProps to make ChartJS update its dataset if there’s been a change in the dataset. We also saw how to use Pusher to trigger events on the server and listen for them on the client side using JS. Have you built anything cool with Pusher recently, a chart maybe? Let’s know in the responses below. This post first appeared on the Pusher blog .

Stories, News & Mentions about PROXYTOOL

Proxytool’s Stories on HackerNoon