Gremlin

gremlin

Gremlin is a leading software company focusing on in the market. It also has a tool similar to which belongs to Netflix, but is more customized to test the system with random loads or scheduled shutdowns. In the article below we will be testing a simple Kubernetes cluster running on EKS with Chaos Test. chaos-test Chaos Monkey Why is Chaos Test Important? Chaos Engineering is used to improve system resilience. Gremlin’s helps to find weaknesses in the system before problems occur. “Failure as a Service” Overview To successfully experience Chaos Engineering with Gremlin, firstly we have two requirements: a running EKS cluster and two applications deployed to EKS. This tutorial helps to produce the requirements and create a scenario to 'simulate an attack with Gremlin'. Step 1 - Prepare Cloud9 IDE Step 2 - Create an EKS cluster using eksctl Step 3 -  Deploy Kubernetes Dashboard Step 4 - Install Gremlin using Helm Step 5 - Deploy a Microservice Demo Application Step 6 - Run a Shutdown Container Attack using Gremlin Prerequsities 1. An AWS account 2. A Gremlin account which can be registered from here Step 1 - Prepare Cloud9 IDE Firstly, let's start to create the Cloud9 environment. Login to your AWS account and navigate Cloud9 to service page. Click on and type the name with anything, in this example we have chosen . Keep all default settings as they are stated since it is needed only to reach EKS resources. Get Started chaous gremlin Wait for a while for the new console to build. Then close all terminals to open a new one. To start creating the cluster, firstly check whether AWS CLI is installed or not with the command below: Administrator:~/environment $ aws --version
aws-cli/1.17.5 Python/2.7.16 Linux/4.14.158-101.185.amzn1.x86_64 botocore/1.14.5
Administrator:~/environment $ Step 2 - Creating an EKS cluster using eksctl 1. We will use to create our EKS clusters. eksctl curl --silent --location | tar xz -C /tmp


sudo mv -v /tmp/eksctl /usr/ /bin "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_ _amd64.tar.gz" $(uname -s) local 2. Making executable eksctl Administrator:~/environment $ sudo chmod +x /usr/ /bin/eksctl
Administrator:~/environment $ local 3. Confirming whether the command works: eksctl Administrator:~ $ eksctl version
[ℹ]  version.Info{BuiltAt: , GitCommit: , GitTag: }
Administrator:~ $ "" "" "0.13.0" 4. Below we are creating the cluster named with three EC2 nodes. Just a word of warning - EKS can cost a lot so please do not forget to delete your resources after you have done with your failure test. gremlin-eksctl eksctl create cluster --name=gremlin-eksctll --nodes=3 --managed --alb-ingress-access --region= ${AWS_REGION} It might take around 15-30 minutes to get ready which you can cluster on the EKS service page. - Fee of EKS is 0,20$/h and fee of EC2 with a m5.large instance type that EKS runs on is 0.096$/h. Estimation of total cost per day will be around 11-12$. Quit Tip 5. Checking the cluster whether it is working and getting its status. As expected, there is only one cluster created. Administrator:~/environment $ eksctl get clusters
NAME            REGION
gremlin-eksctl  us-east-1
Administrator:~/environment $ 6. Updating the file by giving the cluster name and region via AWS CLI tool. kubeconfig Administrator:~/environment $ sudo aws eks --region us-east-1  update-kubeconfig --name gremlin-eksctl
Updated context arn:aws:eks:us-east-1:312867154612:cluster/gremlin-eksctl /root/.kube/config
Administrator:~/environment $ in 7. On checking we see 3 nodes in the cluster. Two of them are worker-nodes while the third is cluster management master-node. Administrator:~/environment $ kubectl get nodes
NAME                            STATUS   ROLES    AGE   VERSION
ip-192-168-14-59.ec2.internal   Ready       59m   v1.14.7-eks-1861c5
ip-192-168-33-12.ec2.internal   Ready       58m   v1.14.7-eks-1861c5
ip-192-168-49-55.ec2.internal   Ready       58m   v1.14.7-eks-1861c5
Administrator:~/environment $ Step 3 - Deploying Kubernetes Dashboard Next we would deploy the Kubernetes dashboard to Kubernetes cluster by using and . These two tools will help our sample application to be shown in the dashboard. We will start with deploying our Kubernetes dashboard as the first step. Heapster InfluxDB Heapster is a performance monitoring and metrics collection system compatible with Kubernetes (versions 1.0.6 and above). It allows for the collection of not only performance metrics about your workloads, pods, and containers, but also events and other signals generated by your cluster. The great thing about Heapster is that it is fully open source as part of the Kubernetes project, and supports a multitude of backends for persisting the data, including but , , , and . Heapster not limited to Influxdb Elasticsearch Graphite InfluxDB InfluxDB is a time series database designed to handle high volume of writing and query loads. 1. Deploying the Kubernetes Dashboard to your EKS cluster: Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
Administrator:~/environment $ 2. Firstly, access to cluster is given since it is deployed to our private cluster while we need to access it via a proxy. This can be done through as the preferred proxy tool. In your workspace, run the following command: Kube-Proxy Administrator:~/environment $ kubectl proxy --port=8080 --address= -- -filter= &
[1] 336
Administrator:~/environment $ W0125 10:28:19.746961     336 proxy.go:140] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on [::]:8080

Administrator:~/environment $ '0.0.0.0' disable true 3. Deploying : Heapster Administrator:~/environment $  kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
serviceaccount/heapster created
deployment.extensions/heapster created
service/heapster created
Administrator:~/environment $ 4. Deploying : InfluxDB Administrator:~/environment $  kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
deployment.extensions/monitoring-influxdb created
service/monitoring-influxdb created
Administrator:~/environment $ 5. Creating cluster role binding for the Dashboard. Heapster Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
clusterrolebinding.rbac.authorization.k8s.io/heapster created
Administrator:~/environment $ 6. The next step is to create an service account. It will let you connect to the Kubernetes Dashboard with permissions. eks-admin admin To authenticate and use the Kubernetes Dashboard: Administrator:~/environment $ kubectl apply -f https://raw.githubusercontent.com/tammybutow/eks-aws/master/eks-admin-service-account.yaml
serviceaccount/eks-admin created
clusterrolebinding.rbac.authorization.k8s.io/eks-admin created
Administrator:~/environment $ 7. To access the Kubernetes Dashboard: a- In your Cloud9 environment, click to open the Dashboard URL. Tools > Preview > Preview Running Application b- Append the following to the end of the URL: /api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 8. Open a new Terminal tab on Cloud9 and enter the following: aws eks get-token --cluster-name gremlin-eksctl | jq -r '.status.token' 9. Select and then copy the output of the command above and paste it to the text field as shown below: Token Step 4 - Installing Gremlin using Helm 1. Download your Gremlin certificates: Start by signing-in to your Gremlin account. If you don't have one, create an account . here Navigate to and click on your . Click the button to download and save certificates to your local drive. Please note that the downloaded contains both a and a matching . Team Settings Team Download certificate.zip public-key certificate private key Unzip the and save it to your Gremlin folder on your desktop. Rename your certificate as and key files as . certificate.zip gremlin.cert gremlin.key 2. Creating Gremlin Namespace: Administrator:~/environment $ kubectl create namespace gremlin
namespace/gremlin created
Administrator:~/environment $ Create a Kubernetes Secret for your certificate and private key, copy and to Cloud9. A quick tip is to create these by the Vim Editor instead of copying from your local computer. gremlin.cert gremlin.key Administrator:~/environment $ kubectl create secret generic gremlin-team-cert \
> \--namespace=gremlin  \
> \--from-file=/home/ec2-user/environment/gremlin.cert \
> \--from-file=/home/ec2-user/environment/gremlin.key
secret/gremlin-team-cert created
Administrator:~/environment $ Check the files on Dashboard whether they are deployed. 3. Installation with Helm The simplest way of installing the Gremlin client on your Kubernetes cluster is to use Helm. Once Helm is installed and configured, the next steps will be to add the Gremlin and to install the client. repo A. Installing Helm source code and making it executable: Administrator:~/environment $ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7164  100  7164    0     0  45630      0 --:--:-- --:--:-- --:--:-- 45341
Administrator:~/environment $ 
Administrator:~/environment $ chmod +x get_helm.sh
Administrator:~/environment $ 
Administrator:~/environment $ ./get_helm.sh
Downloading https://get.helm.sh/helm-v2.16.1-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/ /bin
helm installed into /usr/ /bin/helm
tiller installed into /usr/ /bin/tiller
Run to configure helm.
Administrator:~/environment $ local local local 'helm init' B. Configuring Helm to access with RBAC Helm relies on a service called which requires special permission on the Kubernetes cluster, for which we will need to build a for using . Next step is to then apply this RBAC to the cluster. Tiller Service Account Tiller C. Creating a new service account: Administrator:~/environment $ cat < ~/environment/rbac.yaml
> ---
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: tiller
>   namespace: kube-system
> ---
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRoleBinding
> metadata:
>   name: tiller
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: cluster-admin
> subjects:
>   - kind: ServiceAccount
>     name: tiller
>     namespace: kube-system
> EoF
Administrator:~/environment $ D. Applying configurations: Administrator:~/environment $ kubectl apply -f ~/environment/rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
Administrator:~/environment $ E. Installing for : Tiller Helm Tiller A companion server component, , that runs on your Kubernetes cluster, listens for commands from , and handles the configuration and deployment of software releases on the cluster. tiller helm Administrator:~/environment $ helm init --service-account tiller
Creating /home/ec2-user/.helm 
Creating /home/ec2-user/.helm/repository 
Creating /home/ec2-user/.helm/repository/cache 
Creating /home/ec2-user/.helm/repository/ Creating /home/ec2-user/.helm/plugins 
Creating /home/ec2-user/.helm/starters 
Creating /home/ec2-user/.helm/cache/archive 
Creating /home/ec2-user/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding repo with URL: http://127.0.0.1:8879/charts has been configured at /home/ec2-user/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/ Administrator:~/environment $ local local $HELM_HOME 'allow unauthenticated users' #securing-your-helm-installation This will install into the cluster and will give access to in your cluster. Please note the security policy alert as shown above, which you can feel free to ignore or follow as per your policy settings. tiller managed resources F. Activating bash-completion for Helm: Administrator:~/environment $ helm completion bash >> ~/.bash_completion
Administrator:~/environment $ . /etc/profile.d/bash_completion.sh
Administrator:~/environment $ . ~/.bash_completion
Administrator:~/environment $ G. To run the Helm install, you will need your Gremlin Team ID. It can be found in the Gremlin app on the page, where you downloaded your certificates earlier. Click on your in the list. The ID you’re looking for can be found under as . Team Settings Team Configuration Team ID H. Export your Team ID as an variable: environment Administrator:~/environment $ GREMLIN_TEAM_ID=13ba2df0-5b0b-572c-a479-8136af79da66
Administrator:~/environment $ export I. Next, export your cluster ID, by giving a name for your Kubernetes cluster. Administrator:~/environment $ GREMLIN_CLUSTER_ID=boraozkanchaos
Administrator:~/environment $ export J. Now add the Gremlin Helm repo, and install Gremlin: Administrator:~/environment $ helm repo add gremlin https://helm.gremlin.com has been added to your repositories
Administrator:~/environment $ "gremlin" Administrator:~/environment $ helm install gremlin/gremlin \
> \--namespace gremlin \
> \--name gremlin \
> \-- gremlin.teamID= \
> \-- gremlin.clusterID= NAME:   gremlin
LAST DEPLOYED: Sat Jan 25 12:37:45 2020
NAMESPACE: gremlin
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME             AGE
gremlin-watcher  0s

==> v1/ClusterRoleBinding
NAME  AGE
chao  0s

==> v1/DaemonSet
NAME     AGE
gremlin  0s

==> v1/Deployment
NAME  AGE
chao  0s

==> v1/Pod(related)
NAME                   AGE
chao-698b9fbfb4-5thjp  0s
gremlin-425wv          0s
gremlin-krdlt          0s
gremlin-l252z          0s

==> v1/ServiceAccount
NAME  AGE
chao  0s


Administrator:~/environment $ set $GREMLIN_TEAM_ID set $GREMLIN_CLUSTER_ID Step 5 - Deploying a Microservice Demo Application The demo environment we are going to deploy on to our EKS cluster is the Hipster Shop: Cloud-Native Microservices Demo Application 1. Clone repo of app source code: Administrator:~/environment $ git https://github.com/GoogleCloudPlatform/microservices-demo.git
Cloning into ...
remote: Enumerating objects: 30, .
remote: Counting objects: 100% (30/30), .
remote: Compressing objects: 100% (21/21), .
remote: Total 2987 (delta 16), reused 16 (delta 7), pack-reused 2957
Receiving objects: 100% (2987/2987), 5.08 MiB | 38.24 MiB/s, .
Resolving deltas: 100% (2039/2039), .
Administrator:~/environment $ clone 'microservices-demo' done done done done done 2. Change directory to the one just created: Administrator:~/environment $ microservices-demo/
Administrator:~/environment/microservices-demo (master) $ cd 3. Deploying the application: Administrator:~/environment/microservices-demo (master) $ kubectl apply -f ./release/kubernetes-manifests.yaml
deployment.apps/emailservice created
service/emailservice created
deployment.apps/checkoutservice created
service/checkoutservice created
deployment.apps/recommendationservice created
service/recommendationservice created
deployment.apps/frontend created
service/frontend created
service/frontend-external created
deployment.apps/paymentservice created
service/paymentservice created
deployment.apps/productcatalogservice created
service/productcatalogservice created
deployment.apps/cartservice created
service/cartservice created
deployment.apps/loadgenerator created
deployment.apps/currencyservice created
service/currencyservice created
deployment.apps/shippingservice created
service/shippingservice created
deployment.apps/redis-cart created
service/redis-cart created
deployment.apps/adservice created
service/adservice created
Administrator:~/environment/microservices-demo (master) $ 4. Wait until pods are in a ready state. Administrator:~/environment/microservices-demo (master) $ kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
adservice-84449b8756-qj4sp               1/1     Running   0          4m57s
cartservice-6cbc9b899c-ww5rf             1/1     Running   0          4m58s
checkoutservice-56b48b77c8-fszpx         1/1     Running   0          4m58s
currencyservice-b9fcb4c98-hx9kq          1/1     Running   0          4m58s
emailservice-797cdcc76d-d257v            1/1     Running   0          4m58s
frontend-785c44fd98-zvgj4                1/1     Running   0          4m58s
loadgenerator-665c4ddb74-xwm8m           1/1     Running   3          4m58s
paymentservice-84d7bf956-fdgxr           1/1     Running   0          4m58s
productcatalogservice-5664f59f54-b5mk5   1/1     Running   0          4m58s
recommendationservice-7f9855d7c6-b2zv6   1/1     Running   0          4m58s
redis-cart-6448dcbdcc-97d55              1/1     Running   0          4m58s
shippingservice-6b6f49747d-svb5f         1/1     Running   0          4m58s
Administrator:~/environment/microservices-demo (master) $ 5. Getting the frontend IP address: Administrator:~/environment/microservices-demo (master) $ kubectl get svc frontend-external -o wide
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)        AGE     SELECTOR
frontend-external   LoadBalancer   10.100.226.157   a0c9bed9a3f7111eab7c912c03cd100e-265884000.us-east-1.elb.amazonaws.com   80:30461/TCP   6m31s   app=frontend
Administrator:~/environment/microservices-demo (master) $ 6. Visit the URL on your browser: Step 6 - Running a Shutdown Container Attack using Gremlin We are going to create our first Chaos Engineering experiment where we would validate the EKS reliability. Our hypothesis is, “After shutting down my container, we will not suffer from downtime and EKS will give us a new one.” cart service Going back to the Gremlin UI, select from the menu on the left and select . We’re going to target a Kubernetes resource, so click on on the upper right. Attacks New Attack Kubernetes Choose : cartservice Choose and : State Shutdown Attacking this pod with our Gremlin UI: We will be shutting down the containers. As a test, we attacked twice and the cartservice pods restarted itself. Which signifies that it is working as expected. Note that it re-generates itself even when you attack to shut down your pods. cartservice When we attacked our containers, the cluster resisted to failure and restarted itself, which symbolizes that our system is now resistant to failure. We have seen what happens when a failure occurs, in this example the failure is shutting down the pods. As a result, we understand that our cluster already has auto-scaling feature. As a reminder; do not forget to delete your and ide. cluster Cloud9 eksctl delete cluster --name=gremlin-eksctl Conclusion Congrats! You’ve installed an AWS EKS cluster, deployed the Kubernetes Dashboard, deployed a microservice demo application, installed the Gremlin agent as a daemon-set, and ran your first Chaos Engineering attack to validate Kubernetes reliability! This article was originally published on https://appfleet.com/blog/chaous-test-to-your-kubernetes-with-gremlin/ .

The Secret To Succesully Simulate An Attack With Gremlin To Test Kubernetes

In September 2019, Graph Query Language is accepted as a new database query language in a vote by the International SQL Standards Committee. The unification of GQL takes time. In this post, we’ve selected some mainstream graph query languages and compared the CRUD usage in these languages respectively. The Graph Query Languages to Be Compared Gremlin is a graph traversal language developed by Apache TinkerPop and has been adopted by a lot of graph database solutions. It can be either or . Gremlin declarative imperative Gremlin is Groovy-based, but has many language variants that allow developers to in many modern programming languages such as Java, JavaScript, Python, Scala, Clojure and Groovy. write Gremlin queries natively : Janus Graph, InfiniteGraph, Cosmos DB, DataStax Enterprise(5.0+) and Amazon Neptune. Supported graph databases Cypher is a declarative graph query language that allows expressive and efficient data querying in a property graph. Cypher The language was designed with the power and capability of SQL. The keywords of the Cypher language are not case sensitive, but attributes, labels, relationship types and variables are case sensitive. : Neo4j, AgensGraph and RedisGraph Supported graph databases nGQL introduces its own query language, , which is a declarative, textual query language like SQL, but designed for graphs. Nebula Graph nGQL The keywords of the nGQL language are case sensitive and it support statement composition so that there’s no need for statement embedding. : Nebula Graph Support graph databases Terms Comparison Before comparing the three graph query languages, let’s take a look at their terms and concepts first. The table below explains how these languages define nodes and edges: Syntax Comparison - CRUD After understanding the common terms in Gremlin, Cypher, and nGQL, let’s take a look at the general syntax of these graph query languages. This section will walk you through the basic CRUD syntax for Gremlin, Cypher and nGQL respectively. Graph Refer to the following example on how to create a graph space. We omitted Cypher since you don’t need to create a graph space before you can add any data to the graph databases. g = TinkerGraph. ().traversal() CREATE gods # Create a graph that Gremlin can traverse open # Create a graph space in nGQL SPACE Vertex We all know that graph consists of nodes and edges. A node in Cypher is called a vertex in Gremlin and nGQL. And an edge is the connection among two nodes. Refer to the following example on inserting new vertex in these query languages respectively. # vertex Gremlin
g.addV(vertexLabel).property()

# vertex Cypher (:nodeLabel {property})

# vertex nGQL VERTEX tagName (propNameList) vid:(tagKey propValue) Insert in Insert in CREATE Insert in INSERT VALUES Vertex Type The nodes/vertices can have types. They are called labels in Gremlin and Cypher, and tags in nGQL. A vertex type can have multiple properties. For example, vertex type have two properties, i.e. and . Person name age Create Vertex Type Refer to the following example on vertex type creation. We omitted Cypher since you don’t need a label before inserting data. g.addV(vertexLabel).property() CREATE tagName(PropNameList) # Create vertex Gremlin type in # Create vertex nGQL type in Note that Gremlin and nGQL both support . This keyword automatically detects if the corresponding vertex type exists. If it does not exist, a new one is created. Otherwise, no vertex type is created. IF NOT EXISTS Show Vertex Types When the vertex types are created, you can show them with the following queries. They will list all the labels/tags rather than certain labels/tags. # vertex Gremlin
g.V().label().dedup();

# vertex Cypher MATCH (n) labels(n)
# vertex Cypher db.labels();

# vertex nGQL TAGS Show types in Show types in method 1 RETURN DISTINCT Show types in method 2 CALL Show types in SHOW CRUD on Vertices This section introduces the basic CRUD operations on vertices using the three query languages. Insert Vertices # vertex certain Gremlin
g.addV(String vertexLabel).property()

# vertex certain Cypher (node:label) 

# vertex certain nGQL VERTEX <tag_name> (prop_name_list) <vid>:(prop_value_list) Insert of type in Insert of type in CREATE Insert of type in INSERT VALUES Get Vertices # vertices Gremlin
g.V(<vid>)

# vertices Cypher
MATCH (n) condition properties(n)

# vertices nGQL PROP <tag_name> <vid> Fetch in Fetch in WHERE RETURN Fetch in FETCH ON Delete Vertices g.V( ).drop() MATCH ( ) 
DETACH DELETE vertex nGQL
DELETE VERTEX # Delete vertex in Gremlin <vid> # Delete a vertex in Cypher node :label node # Delete in <vid> Update a Vertex’s Property This section shows you how to update properties for vertex. # vertex Gremlin .V(<vid>).property()

# vertex Cypher . = V

# vertex nGQL VERTEX <vid> <update_columns> Update in g Update in SET n prop Update in UPDATE SET Both Cypher and nGQL use keyword to set the vertex type, except that the keyword is added in nGQL to identify the operation. The operation of Gremlin is similar to the above-mentioned fetching vertex, except that it adds operations for changing property. SET UPDATE Edges This section introduces the basic CRUD operations on edges. Edge Types Like vertices, edges can also have types. g.edgeLabel() CREATE EDGE edgeTypeName(propNameList) # Create an edge Gremlin type in # Create an edge nGQL type in CRUD on Edges Insert Edges of Certain Types Inserting an edge is similar to inserting vertices. Cypher uses and and nGQL uses to represent edges respectively. Gremlin uses the keyword to indicate edge direction. -[]-> -> to() Edges are by default directed in the three languages. The chart on the left below is a directed edge while the one on the right is an undirected edge. # edge certain Gremlin
g.addE(String edgeLabel). (v1). (v2).property()

# edge certain Cypher (<node1- >:<label1- >)-
  [(<relationship- >:<relationship-label- >)]
  ->(<node2- >:<label2- >)

# edge certain nGQL EDGE <edge_name> ( <prop_name_list> ) <src_vid> -> <dst_vid>: ( <prop_value_list> ) Insert of type in from to Insert of type in CREATE name name name name name name Insert of type in INSERT VALUES Delete Edges # edge Gremlin
g.E(<eid>). ()

# edge Cypher
MATCH (<node1- >:<label1- >)-[r:relationship-label- ]->() r

# edge nGQL EDGE <edge_type> <src_vid> -> <dst_vid> Delete in drop Delete in name name name DELETE Delete in DELETE Fetch Edges # edges Gremlin
g.E(<eid>)

# edges Cypher
MATCH (n)-[r:label]->() condition properties(r)

# edges nGQL PROP <edge_name> <src_vid> -> <dst_vid> Fetch in Fetch in WHERE RETURN Fetch in FETCH ON Other Operations In addition to the common CRUD on vertices and edges, we will also show you some combined queries in the three graph query languages. Traverse edges # Traverse edges specified vertices Gremlin
g.V(<vid>).outE(<edge>)

# Traverse edges specified vertices Cypher
Match (n)->[r:label]->[]
WHERE id(n) = vid
RETURN r

# Traverse edges specified vertices nGQL
GO FROM <vid> OVER <edge> with in with in with in Traverse edges reversely In reverse traverse, Gremlin uses to indicate reversion, Cypher uses . nGQL uses keyword . in <- REVERSELY # Traverse edges reversely specified vertices Gremlin
g.V(<vid>). (<edge>)

# Traverse edges reversely specified vertices Cypher
MATCH (n)<-[r:label]-()

# Traverse edges reversely specified vertices nGQL
GO FROM <vid>  OVER <edge> REVERSELY with in with with Traverse edges bidirectionally If the edge direction is (either direction is acceptable),  Gremlin uses , Cypher use , and nGQL use keyword . irrelevant bothE() -[]- BIDIRECT g.V( ).bothE( ) MATCH (n)-[r: ]-() GO FROM OVER BIDIRECT # Traverse edges reversely with specified vertices Gremlin <vid> <edge> # Traverse edges reversely with specified vertices Cypher label # Traverse edges reversely with specified vertices nGQL <vid> <edge> Query N hops along specified edge Gremlin and nGQL use and respectively to represent N hops. Cypher uses . times STEP relationship*N # hops along specified edge Gremlin .V(<vid>). ( (<edge>)).times( )

# hops along specified edge Cypher
MATCH ( )-[r: * ]->()
WHERE condition # hops along specified edge nGQL
GO STEPS FROM <vid> OVER <edge> Query N in g repeat out N Query N in n label N RETURN r Query N in N Find paths between two vertices # Find paths two vertices Gremlin
g.V(<vid>).repeat( ()). (<vid>).path()

# Find paths two vertices Cypher
MATCH p =(a)-[.*]->(b) condition p

# Find paths two vertices nGQL
FIND <vid> <vid> * between in out until between in WHERE RETURN between in ALL PATH FROM TO OVER Example queries This section introduces some demonstration queries. Demo model: The Graphs of Gods The examples in this section make extensive use of the toy graph distributed with called as diagrammed below。 Janus Graph The Graphs of Gods This example describes the relationships between the beings and places of the Roman pantheon. Inserting data ==>v[1] ==>v[2] ==>v[31] ==>v[32] ==>e[13][2-father->1] # Inserting vertices # # nGQL nebula> INSERT VERTEX character(name, age, ) VALUES ( ):( , 10000, ), ( ):( , 5000, ); type hash "saturn" "saturn" "titan" hash "jupiter" "jupiter" "god" # # Gremlin gremlin> saturn = g.addV( ).property(T.id, 1).property( , ).property( , 10000).property( , ).next(); "character" 'name' 'saturn' 'age' 'type' 'titan' gremlin> jupiter = g.addV( ).property(T.id, 2).property( , ).property( , 5000).property( , ).next(); "character" 'name' 'jupiter' 'age' 'type' 'god' gremlin> prometheus = g.addV( ).property(T.id, 31).property( , ).property( , 1000).property( , ).next(); "character" 'name' 'prometheus' 'age' 'type' 'god' gremlin> jesus = g.addV( ).property(T.id, 32).property( , ).property( , 5000).property( , ).next(); "character" 'name' 'jesus' 'age' 'type' 'god' # # Cypher cypher> CREATE (src:character {name: , age: 10000, : }) "saturn" type "titan" cypher> CREATE (dst:character {name: , age: 5000, : }) "jupiter" type "god" # Inserting edges # # nGQL nebula> INSERT EDGE father() VALUES ( )-> ( ):(); hash "jupiter" hash "saturn" # # Gremlin gremlin> g.addE( ).from(jupiter).to(saturn).property(T.id, 13); "father" # # Cypher cypher> CREATE (src)-[rel:father]->(dst) Deleting # nGQL nebula> DELETE VERTEX ( ); hash "prometheus" # Gremlin gremlin> g.V(prometheus).drop(); # Cypher cypher> MATCH (n:character {name: }) DETACH DELETE n "prometheus" Updating ==>v[32] # nGQL nebula> UPDATE VERTEX ( ) SET character.type = ; hash "jesus" 'titan' # Gremlin gremlin> g.V(jesus).property( , 6000); 'age' # Cypher cypher> MATCH (n:character {name: }) SET n.type = ; "jesus" 'titan' Fetching/Reading # nGQL # Gremlin
gremlin> g.V(saturn).valueMap();
==>[name:[saturn],type:[titan],age:[10000]]
# Cypher
cypher> MATCH (n:character {name:"saturn"}) RETURN properties(n) nebula> FETCH PROP ON character hash("saturn");
=================================================== | character.name | character.age | character.type |
=================================================== | saturn         | 10000         | titan          |
--------------------------------------------------- ╒════════════════════════════════════════════╕ │"properties(n)"                             │ ╞════════════════════════════════════════════╡ │{"name":"saturn","type":"titan","age":10000}│ └────────────────────────────────────────────┘ Finding the name of Hercules’s Father # nGQL
nebula>  LOOKUP .name == | \
      -> GO $-.VertexID father YIELD $$ .character.name;
=====================
| $$ .character.name |
=====================
| jupiter           | # Gremlin
gremlin> g.V().hasLabel( ).has( , ). ( ). ( );
==>jupiter
# Cypher
cypher> MATCH (src: { :"hercules"})-[:father]->(dst: ) dst.name
      ╒══════════╕
      │"dst.name"│
      ╞══════════╡
      │"jupiter" │
      └──────────┘ ON character WHERE character 'hercules' FROM OVER --------------------- 'character' 'name' 'hercules' out 'father' values 'name' character name character RETURN Finding the name of Hercules’s Grandfather # nGQL
nebula> LOOKUP .name == | \
     -> GO STEPS $-.VertexID father YIELD $$ .character.name;
=====================
| $$ .character.name |
=====================
| saturn            | # Gremlin
gremlin> g.V().hasLabel( ).has( , ). ( ). ( ). ( );
==>saturn
# Cypher
cypher> MATCH (src: { :"hercules"})-[:father* ]->(dst: ) dst.name
      ╒══════════╕
      │"dst.name"│
      ╞══════════╡
      │"saturn"  │
      └──────────┘ ON character WHERE character 'hercules' 2 FROM OVER --------------------- 'character' 'name' 'hercules' out 'father' out 'father' values 'name' character name 2 character RETURN Find the characters with age > 100 # nGQL # Gremlin
gremlin> g.V().hasLabel( ).has( ,gt(100)).values( );
==>saturn
==>jupiter
==>neptune
==>pluto
# Cypher
cypher> MATCH (src:character) WHERE src.age > 100 RETURN src.name nebula> LOOKUP ON character WHERE character.age > 100 YIELD character.name, character.age;
========================================================= | VertexID             | character.name | character.age |
========================================================= | 6761447489613431910  | pluto          | 4000          |
--------------------------------------------------------- | -5860788569139907963 | neptune        | 4500          |
--------------------------------------------------------- | 4863977009196259577  | jupiter        | 5000          |
--------------------------------------------------------- | -4316810810681305233 | saturn         | 10000         |
--------------------------------------------------------- 'character' 'age' 'name' ╒═══════════╕ │"src.name" │ ╞═══════════╡ │  "saturn" │ ├───────────┤ │ "jupiter" │ ├───────────┤ │ "neptune" │ │───────────│ │  "pluto"  │ └───────────┘ Find who are Pluto’s cohabitants, excluding Pluto himself # nGQL
nebula>  GO hash("pluto") lives YIELD lives._dst place | GO $-.place lives REVERSELY \
$$ .character.name != "pluto" YIELD $$ .character.name cohabitants;
===============
| cohabitants |
===============
| cerberus    | # Gremlin
gremlin> g.V(pluto). ( ). ( ). ( (neq(pluto))). ( );
==>cerberus
# Cypher
cypher> MATCH (src: { :"pluto"})-[:lives]->()<-[:lives]-(dst: ) dst.name
      ╒══════════╕
      │"dst.name"│
      ╞══════════╡
      │"cerberus"│
      └──────────┘ FROM OVER AS FROM OVER WHERE AS --------------- out 'lives' in 'lives' where is values 'name' character name character RETURN Pluto’s Brothers # which brother lives which place?
## nGQL
nebula> GO hash("pluto") brother YIELD brother._dst god | \
GO $-.god lives YIELD $^.character.name Brother, $$ . .name Habitations;
=========================
| Brother | Habitations |
=========================
| jupiter | sky         | | neptune | sea         | ## Gremlin
gremlin> g.V(pluto). ( ). ( ). ( ). ( ). ( , ). ( );
==>[god:jupiter, place:sky]
==>[god:neptune, place:sea]
## Cypher
cypher> MATCH (src: { :"pluto"})-[:brother]->(bro: )-[:lives]->(dst) bro.name, dst.name
      ╒═════════════════════════╕
      │"bro.name"    │"dst.name"│
      ╞═════════════════════════╡
      │ "jupiter"    │  "sky"   │
      ├─────────────────────────┤
      │ "neptune"    │ "sea"    │
      └─────────────────────────┘ in FROM OVER AS FROM OVER AS location AS ------------------------- ------------------------- out 'brother' as 'god' out 'lives' as 'place' select 'god' 'place' by 'name' Character name Character RETURN In addition to the basic operations in the three graph query languages, we will work on another piece of comparison of advanced operations in these languages. Stay tuned! Previously published at https://nebula-graph.io/posts/graph-query-language-comparison-cypher-gremlin-ngql/

Gremlin vs Cypher vs nGQL: Graph Query Language Comparison 

The Power of Market Disruption: How to Detect Fraud With Graph Data

They say the devil's in the details, but in the case of , it's more like a mischievous gremlin with a penchant for blockchain security. I plunged headfirst into this tech rabbit hole to witness Mitja Gorochevsky, the wild mind behind and this radical consensus algorithm, unleash his latest creation on a crowd of unsuspecting tech enthusiasts. Acki Nacki GOSH The Scene: A Digital Carnival of Quirk Picture a virtual stage where coding wizards and crypto evangelists collide—that's the kind of energy that crackled through the air as Mitja took the reins. The vibe was part tech conference, part cyberpunk circus, with Acki Nacki taking center stage as a personified defender of integrity. blockchain But even gremlins have their off days. Internet gremlins, that is. The demo hit a few snags thanks to connectivity hiccups, but that didn't dampen the spirit. It just added a layer of unpredictable chaos to the mix, as if Acki Nacki was flexing its trickster muscles. Enter Acki Nacki: The Multi-Headed Hydra of Blockchain Now, hold on to your crypto wallets because Acki Nacki is no ordinary consensus algorithm. Imagine a multi-core, multi-threaded execution environment where anyone can drop a contract and run operations in parallel with lightning-fast, under-1-second finality. And all this on the decentralised supercomputer known as GOSH. "The odds of a successful malicious attack on GOSH (with Acki Nacki) is not just less likely than a successful attack on Bitcoin," Mitja boldly declared, "it's less likely than an asteroid hitting Earth and wiping us all out tomorrow!" Hackathon Hoedown and the GOSH Gold Rush But the real party starts with the "Underworld" hackathon. A siren call to tech rebels, inviting them to bend the boundaries of what's possible. Here are the challenges they laid down: Wallet apps that make your crypto quiver with glee. Visualisation tools to make blockchains look like trippy art installations. Security tooling so tight that even the NSA would be impressed. SDKs, TVMs, ZKPs... basically, if it sounds like alphabet soup, it's fair game. Why go through all this trouble? Because the only way to score GOSH tokens is either through this hackathon or the "Die Last" validator games. It's a tech wild west where your coding skills are your six-shooter. Is anyone paying attention, though, Ben? Well, I'm glad you asked. For me, the big-news-plot-twist of the evening, away from all the tech bluster, is that the United Nations, the global epitome of diplomatic collaboration, has chosen GOSH to host its decentralized autonomous organization (DAO) for the Internet Governance Forum (IGF). This move underscores the growing recognition of GOSH as a reliable and secure platform for handling critical aspects of Internet governance. I can already picture Acki Nacki, the mischievous guardian of blockchain integrity, now donning the mantle of responsibility as it becomes the backbone for this huge UN-backed initiative. Conclusion: A Glimpse into a Decentralized Future, Gremlins and All Did Acki Nacki live up to the hype? The jury's still out, but one thing's for certain: Mitja knows how to throw a tech party that leaves you questioning reality and craving more. In a world of sterile blockchain conferences, this was a jolt of raw, unadulterated innovation. Hold on tight, folks—the GOSH ride is just getting started.

Deep in the Tech Underworld, a Gremlin-Like Consensus Wakes: Dispatches of the Acki Nacki Freakshow

Decoding my dive into the IBM Quantum Challenge 2024 Greetings from the sunny shores of Durban! Having just braved the IBM Quantum Challenge 2024 & still harboring a serious case of quantum fever, I‘m here to spill the Qiskit Kool-Aid on the good, the not-so-bad and the downright mind-blowing. While the focus this year was broader, the last time I took part was IBM Quantum Challenge Africa 2021, which was all about tackling issues that hit close to home. Teaming up with Wits University, the IBM Research Lab in South Africa developed a challenge that targeted real-world African problems with crop logistics in Agriculture, Financial optimization and quantum Drug discovery. It was all about empowering us as African problem-solvers with the skills and knowledge to leverage the power of quantum computing for our communities. 2K24 & I’m levelling up my Quantum IQ: Lets be honest, quantum computing can feel like its spoken in a language only qubits understand. Going into this challenge, I was a solid 5 outta 10 on the quantum knowledge scale. By the end? Lets just say I graduated to a confident 6😉 The challenge wasn’t just informative, it made these mind-bending concepts feel tangible. So, Challenge Accepted (with a few Nitpicks) the whole experience gets a thumbs up from me - a solid 10 outta 10. The content was dense (think brain food buffet) but the learning curve never really felt like a sheer cliff. Our real-time forum for troubleshooting was epic, all of our challenge parties in different corners of the globe, not forgetting our Dev humor keeping us sane. Tools that stole the show this challenge came loaded with some seriously cool tools, and a few that deserve special recognition: Qiskit Code Assistant This code-whisperer was a godsend, suggesting optimizations and catching errors like a quantum ninja, in the Beta version! And being the first ones to try him out! t’was super cool to be able to do that!
Qiskit Serverless This auto backend selection had me deploying quantum tasks like a pro. Plus, the seamless integration and potential for parallel workflows? Quantum Devs, rejoice! A small caveat: more documentation on custom backend integration would be thee cherry on top.
AI Transpiler This one showed promise, but had a few hiccups with a shopping list of errors (think gremlins in the lab) 🥴
Circuit Knitting Toolbox more like untangling the Quantum yarn, this lab showed me how to break down complex circuits into bite-sized pieces.
VQC & Error Mitigation building a Variational Quantum Classifier, this was a such a cool lab. This lab threw me into the heart of practical quantum machine learning.
Scaling VQC to 50+ Qubits had me taking Quantum leaps & bounds, pushing both hardware and my understanding of quantum simulations to the limit. It wasn’t  just about building a bigger classifier, this was a glimpse into the future. These last 10 days have certainly been a game changer for quantum engineers and I have come out with so much new knowledge. Major props to the Team of Developers for the crystal clear instructions & the encouragement. Imagine bouncing ideas off fellow quantum Devs/engineers, it was a perfect blend of theory, practice and future-focused challenges - now that’s how we conquer collaboratively. For my setup, I primarily tested my Labs locally before running them in the Cloud, giving me the flexibility to switch environments. Sure, there were a few code-based hiccups, but nothing a little debugging couldn’t fix. Each lab built upon the previous one, creating a beautiful symphony of learning. Some Advice to Sharpen your Quantum Sword Before diving in, brush up on the basics, online courses are great places to start.
Practice makes perfect, find platforms that offer tutorials and playgrounds to experiment with quantum circuits.
Don’t go it alone, find Quantum forum’s and communities to join & connect with other quantum enthusiasts and get help when you are stuck.
Most Important know your coding environment, decide how you’ will be running your labs, locally or leveraging the cloud. Never be afraid to experiment and seek help, learning curves are steep, but the journey is always worth half the fun. My Quantum Reflections. Its clear IBM is on a mission to democratize quantum computing for developers and the tools I got to explore are a giant leap forward that left me feeling prepped to tackle the quantum frontier. The possibilities for innovation are 🤯 as these tools keep evolving. This was a transformative experience. I left feeling empowered, yet both humbled by the complexity and energized by the possibilities. I’m always incredibly grateful for these opportunities to be at the forefront exploring cutting-edge technology and putting the insights I gain to good use. Decoding my dive into the IBM Quantum Challenge 2024 Decoding my dive into the IBM Quantum Challenge 2024 Greetings from the sunny shores of Durban! Having just braved the IBM Quantum Challenge 2024 & still harboring a serious case of quantum fever, I‘m here to spill the Qiskit Kool-Aid on the good, the not-so-bad and the downright mind-blowing. While the focus this year was broader, the last time I took part was IBM Quantum Challenge Africa 2021, which was all about tackling issues that hit close to home. Teaming up with Wits University, the IBM Research Lab in South Africa developed a challenge that targeted real-world African problems with crop logistics in Agriculture, Financial optimization and quantum Drug discovery. It was all about empowering us as African problem-solvers with the skills and knowledge to leverage the power of quantum computing for our communities. 2K24 & I’m levelling up my Quantum IQ: Lets be honest, quantum computing can feel like its spoken in a language only qubits understand. Going into this challenge, I was a solid 5 outta 10 on the quantum knowledge scale. By the end? Lets just say I graduated to a confident 6😉 The challenge wasn’t just informative, it made these mind-bending concepts feel tangible. 2K24 & I’m levelling up my Quantum IQ: So, Challenge Accepted ( with a few Nitpicks ) the whole experience gets a thumbs up from me - a solid 10 outta 10. The content was dense ( think brain food buffet ) but the learning curve never really felt like a sheer cliff. Our real-time forum for troubleshooting was epic, all of our challenge parties in different corners of the globe, not forgetting our Dev humor keeping us sane. Challenge Accepted ( with a few Nitpicks ) think brain food buffet Tools that stole the show this challenge came loaded with some seriously cool tools, and a few that deserve special recognition: Tools that stole the show Qiskit Code Assistant This code-whisperer was a godsend, suggesting optimizations and catching errors like a quantum ninja, in the Beta version! And being the first ones to try him out! t’was super cool to be able to do that! Qiskit Serverless This auto backend selection had me deploying quantum tasks like a pro. Plus, the seamless integration and potential for parallel workflows? Quantum Devs, rejoice! A small caveat: more documentation on custom backend integration would be thee cherry on top. AI Transpiler This one showed promise, but had a few hiccups with a shopping list of errors (think gremlins in the lab) 🥴 Circuit Knitting Toolbox more like untangling the Quantum yarn, this lab showed me how to break down complex circuits into bite-sized pieces. VQC & Error Mitigation building a Variational Quantum Classifier, this was a such a cool lab. This lab threw me into the heart of practical quantum machine learning. Scaling VQC to 50+ Qubits had me taking Quantum leaps & bounds, pushing both hardware and my understanding of quantum simulations to the limit. It wasn’t  just about building a bigger classifier, this was a glimpse into the future. Qiskit Code Assistant This code-whisperer was a godsend, suggesting optimizations and catching errors like a quantum ninja, in the Beta version! And being the first ones to try him out! t’was super cool to be able to do that! Qiskit Code Assistant Qiskit Serverless This auto backend selection had me deploying quantum tasks like a pro. Plus, the seamless integration and potential for parallel workflows? Quantum Devs, rejoice! A small caveat: more documentation on custom backend integration would be thee cherry on top. Qiskit Serverless AI Transpiler This one showed promise, but had a few hiccups with a shopping list of errors (think gremlins in the lab) 🥴 AI Transpiler Circuit Knitting Toolbox more like untangling the Quantum yarn, this lab showed me how to break down complex circuits into bite-sized pieces. Circuit Knitting Toolbox VQC & Error Mitigation building a Variational Quantum Classifier, this was a such a cool lab. This lab threw me into the heart of practical quantum machine learning. VQC & Error Mitigation Scaling VQC to 50+ Qubits had me taking Quantum leaps & bounds, pushing both hardware and my understanding of quantum simulations to the limit. It wasn’t  just about building a bigger classifier, this was a glimpse into the future. Scaling VQC to 50+ Qubits These last 10 days have certainly been a game changer for quantum engineers and I have come out with so much new knowledge. Major props to the Team of Developers for the crystal clear instructions & the encouragement. Imagine bouncing ideas off fellow quantum Devs/engineers, it was a perfect blend of theory, practice and future-focused challenges - now that’s how we conquer collaboratively. Team of Developers For my setup, I primarily tested my Labs locally before running them in the Cloud, giving me the flexibility to switch environments. Sure, there were a few code-based hiccups, but nothing a little debugging couldn’t fix. Each lab built upon the previous one, creating a beautiful symphony of learning. Some Advice to Sharpen your Quantum Sword Some Advice to Sharpen your Quantum Sword Before diving in, brush up on the basics, online courses are great places to start. Practice makes perfect, find platforms that offer tutorials and playgrounds to experiment with quantum circuits. Don’t go it alone, find Quantum forum’s and communities to join & connect with other quantum enthusiasts and get help when you are stuck. Most Important know your coding environment, decide how you’ will be running your labs, locally or leveraging the cloud. Before diving in, brush up on the basics, online courses are great places to start. Practice makes perfect, find platforms that offer tutorials and playgrounds to experiment with quantum circuits. Don’t go it alone, find Quantum forum’s and communities to join & connect with other quantum enthusiasts and get help when you are stuck. Most Important know your coding environment, decide how you’ will be running your labs, locally or leveraging the cloud. Most Important Never be afraid to experiment and seek help, learning curves are steep, but the journey is always worth half the fun. My Quantum Reflections. Its clear IBM is on a mission to democratize quantum computing for developers and the tools I got to explore are a giant leap forward that left me feeling prepped to tackle the quantum frontier. The possibilities for innovation are 🤯 as these tools keep evolving. This was a transformative experience. I left feeling empowered, yet both humbled by the complexity and energized by the possibilities. I’m always incredibly grateful for these opportunities to be at the forefront exploring cutting-edge technology and putting the insights I gain to good use.

Bits to Qubits: Decoding my dive into the IBM Quantum Challenge 2024

বিট টু কিউবিটস: আইবিএম কোয়ান্টাম চ্যালেঞ্জ 2024-এ আমার ডাইভ ডিকোডিং

Von Bits zu Qubits: Dekodierung meines Eintauchens in die IBM Quantum Challenge 2024

De Bits a Qubits: decodificando mi inmersión en el IBM Quantum Challenge 2024

Bits to Qubits : décoder ma plongée dans l'IBM Quantum Challenge 2024

बिट्स टू क्यूबिट्स: आईबीएम क्वांटम चैलेंज 2024 में मेरे गोते को डिकोड करना

ビットから量子ビットへ: IBM Quantum Challenge 2024 への私の挑戦を解読

비트를 큐비트로: IBM Quantum Challenge 2024에 대한 나의 다이빙 디코딩

Bits para Qubits: decodificando meu mergulho no IBM Quantum Challenge 2024

Биты в кубиты: расшифровка моего погружения в IBM Quantum Challenge 2024

Bitlerden Qubitlere: IBM Quantum Challenge 2024'e yaptığım dalışın kodunu çözüyorum

Bits to Qubits: Giải mã việc tôi tham gia Thử thách lượng tử IBM 2024

从比特到量子比特：解读我对 2024 年 IBM 量子挑战赛的理解

How to Use Knowledge Graphs for Retrieval-Augmented Generation—Without a Graph DB

পুনরুদ্ধার-অগমেন্টেড জেনারেশনের জন্য জ্ঞান গ্রাফগুলি কীভাবে ব্যবহার করবেন—একটি গ্রাফ ডিবি ছাড়াই

So nutzen Sie Wissensgraphen zur abruferweiterten Generierung – ohne Graph-DB

Cómo utilizar gráficos de conocimiento para la generación de recuperación aumentada, sin una base de datos de gráficos

Comment utiliser les Knowledge Graphs pour la génération augmentée par récupération – sans base de données graphique

रिट्रीवल-ऑगमेंटेड जेनरेशन के लिए नॉलेज ग्राफ़ का उपयोग कैसे करें - बिना ग्राफ़ DB के

グラフ DB を使用せずにナレッジ グラフを検索拡張生成に使用する方法

검색 증강 생성을 위해 지식 그래프를 사용하는 방법 - 그래프 DB 없이

Como usar gráficos de conhecimento para geração aumentada de recuperação - sem um banco de dados gráfico

Как использовать графы знаний для генерации с расширенным поиском — без базы данных графов

Almayla Artırılmış Oluşturma için Bilgi Grafikleri Nasıl Kullanılır - Grafik Veritabanı Olmadan

Cách sử dụng Sơ đồ tri thức cho thế hệ tăng cường truy xuất—Không cần cơ sở dữ liệu đồ thị

如何使用知识图谱进行检索增强生成（无需图形数据库）

What the heck is ? That was the first thing I asked myself when I came across it in the Summer of 2023. It was my involvement with that brought it to my attention; they wanted to add Iceberg support to PuppyGraph, but just what the heck is it? PuppyGraph Apache Iceberg This blog is going to be more of a hot take on PuppyGraph to get you thinking about how you might use it in your own projects. I have no affiliation with the company or project other than thinking it was pretty cool. Co-founder recently (Feb 2024) gave a presentation at the that was interesting, and well received, according to my friends that were there. Weimo Liu Chill Data Summit What is PuppyGraph? Simply, PuppyGraph is a cloud-native graph data lakehouse providing a graph analytics engine for your data. They address graph scalability through the auto-sharding of data so the compute and storage are separate, much like the lakehouse design. So, they provide a graph data warehouse, data lake, and multi-data models on a single copy of your data. That means you can do some pretty cool graphing on your data in one of the supported formats. What can it connect to? PuppyGraph has rapidly added support for various platforms, catalogs, and connection engines. Currently, we see: Apache Iceberg Apache Hudi Delta Lake MySQL PostgreSQL DuckDB BigQuery Redshift LanceDB (coming soon) JDBC Catalog Data Lake Catalog Hive Metastore AWS Glue Their SaaS interface also gives you direct access to both a Gremlin and Cypher console to perform graph queries, in addition to a graph notebook, which uses Jupyter. Using PuppyGraph A Docker container is provided to allow you to get started on a local machine. You’ll need a schema defined in JSON format that will define your data layout to PuppyGraph. Once you ingest that and it is verified, then away you go. The integrated graph browser is pretty nifty. You can easily zoom in/out to see the clustering and attributes in addition to queries. Zooming in further, we can see more of the details: Clicking on a node will give us a pop-up of details: This allows you to explore different vertices and edges easily. These static pictures don’t really represent how fast the performance is or how much fun it is to bounce around your data. I should have utilized some genealogical data for fun. Because they are using the Gremlin and Cypher query languages, that means any 3rd party UI tool will also be compatible. A real advantage here is that PuppyGraph works on the data where it lives and isn’t making you copy it elsewhere. Without going into the particulars on a specific platform, this gives you a general idea of what features and functions are available. Summary Certainly, graph databases and their representation don’t apply generically as a structured database does, but we are seeing more and more how these kinds of data representations are being used to model the real world. I didn’t see that this is an open-source project, and I didn’t find it on GitHub. There is no mention of pricing, so I’m not sure where they are going with all of this. The documentation isn’t amazing, but it seems to be enough to get started and try it out. Overall, this is a fun project to play with. I need to percolate on it more to see where I might use it, but I can envision some interesting use cases combining it with other self-contained projects like DuckDB and LanceDB. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton?

What the Heck is PuppyGraph?

Remember the napkin sketch? Four years later, it's launch day, and...well, let's just say Stan's hand is firmly in the air. Straight out of Beyond The Whiteboard, this cautionary tale paints a familiar picture: software development projects gone rogue, deadlines dashed, and dreams deferred. But wait! Enter the Nocode knight in shining armor, promising to slay the dragon of complex coding and deliver your product to market in a blaze of drag-and-drop glory. Should you sheath your coding sword and join the Nocode revolution? Or is this just another tech fad destined to gather dust in the attic of history, like the 4GL of yore? Hold your horses, intrepid entrepreneur! The answer, as always, is nuanced. Let's ditch the black-and-white thinking and explore the battlefield of Nocode vs developers: When Nocode Is Your Hemingway: Concept validation: Got a brilliant idea but a shoestring budget? Nocode lets you build an MVP fast, test the waters with users, and iterate without breaking the bank. You're a solo powerhouse with limited technical expertise. No-code empowers you to create functional products without getting bogged down in coding. Solopreneur hustle: Your product idea doesn't require complex functionalities or integrations. Think landing pages, simple e-commerce stores, or basic CRUD apps. Nocode can get you there efficiently. Simple solutions: But remember, Hemingway didn't just scribble on a napkin: No code platforms have their predefined templates and functionalities. You might find yourself cramped when your vision demands custom solutions. Limited flexibility: As your product grows and evolves, the limitations of Nocode might become apparent. Scaling a complex Nocode app can be challenging. Scaling woes: When disaster hits (and it will), you'll rely on the platform's support, not your own in-house developer hero. The 3 AM gremlin strikes: So when should we call in the coding cavalry? : Your product demands unique features, intricate integrations, or cutting-edge performance. Code gives you the freedom to build exactly what you need. Complex functionalities You're aiming for the big leagues and need a product that can scale seamlessly. Invest in developers for a robust foundation. Long-term vision: You crave complete control over your product's architecture and security. Code is your key to unlocking endless possibilities. Custom control: The ultimate takeaway? It's not an either/or. Think of Nocode and developers as complementary tools in your entrepreneurial toolbox. Use Nocode for rapid prototyping, MVPs, and simple solutions. But a skilled developer is your secret weapon for complex ideas, long-term scalability, and the freedom to craft your vision truly. Remember, the best tool is the one that empowers you to bring your unique product to life. Choose wisely, young entrepreneur, and may your product launch be glorious, not disastrous. P.S. Want to know more about specific Nocode tools or how to find the perfect developer? Hit me up in the comments!

Nocode Vs. Developer: Choosing Between Hemingway-like Coding and Pro-Level Drag-and-Drop

Zero-Day Vulnerabilities | All-Day Stress Encountering a zero-day vulnerability can feel like navigating through a fog of unknowns. This kind of vulnerability, previously undetected and unaddressed, often brings a mix of anxiety and urgency. Questions whirl through your mind: How severe is it? What havoc can it wreak? Who might exploit it? And crucially, is it lurking in our systems? These initial questions are just the tip of the iceberg. As the reality sinks in – you're facing an unfamiliar foe – the real challenge emerges: "How do I deal with something I've never seen before?" The reactions vary – from the near-panic of tearing one's hair out to the extreme thoughts of obliterating everything to prevent a breach, and in some cases, people simply giving up. But here's the silver lining: while a zero-day might be an unknown threat, our response doesn't require unknown measures. We can leverage our existing knowledge and past experiences in security and incident response to tackle these new threats effectively. Understanding Zero-Day Vulnerabilities Imagine the world of cybersecurity as a high-stakes game of hide and seek, and zero-day vulnerabilities are the ultimate hiders – unseen, unknown, and unpredictably dangerous. These digital gremlins are the nightmares of IT professionals, lurking in the shadows of code and systems. They are the loopholes that nobody knew existed, until, of course, someone malicious discovers and exploits them – and that's when the real game begins. But what makes a zero-day so special and feared? It's the element of surprise. These vulnerabilities are like the plot twists in a thriller movie – you never see them coming until they hit you. When a zero-day attack is revealed, it's akin to a magician revealing his trick – there's awe, shock, and a bit of "how did we not see that coming?" To truly grasp the essence of zero-days, let’s step into the history books of cybersecurity. Remember ? This devious little worm, discovered back in 2010, wreaked havoc on Iran's nuclear facilities. Stuxnet It’s the poster child for zero-day exploits – sneaky, sophisticated, and with a punch that left the world reeling. Stuxnet used not one, but several zero-days to infiltrate and sabotage its target. It’s like finding out the quiet kid in class is actually a chess grandmaster – unexpected and a little bit unsettling. In the grand scheme of things, zero-days are a harsh reminder of our digital vulnerability – the unknown threats that lurk in the shadows, waiting for their moment in the spotlight. They're a testament to the fact that in cybersecurity, sometimes the biggest threat is what you don't know you don't know. But fear not, for as daunting as these hidden threats are, they are not invincible. The Power of Existing Knowledge So, a zero-day has just dropped into your lap like an uninvited guest at a party. Now what? It's easy to feel like you're up the creek without a paddle, but hold on – you've got more tools at your disposal than you might think. First off, let’s demystify this beast. Zero-days aren't some alien technology; they are vulnerabilities, plain and simple. Think of them as the hidden doors in a fortress that the architect forgot about. Now, the key lies in using what we already know about fortresses – or in our case, cybersecurity – to find and secure these doors. It’s like realizing you’ve been holding the treasure map upside down all along. Our past experiences with cybersecurity are like gold. Each past threat, each breach, and every little anomaly we've encountered and resolved – they've all been building up our arsenal. We've learned to monitor, to analyze, to patch, and to respond. These aren’t just random skills; they are the very weapons we need to combat zero-days. It's like being a chef in a kitchen – you might not know what surprise ingredient you'll have to cook with next, but you know your way around the kitchen, and that’s what counts. Consider this: the basic principles of securing a network against known vulnerabilities apply just as effectively to unknown ones. Regular system monitoring, rigorous security protocols, and a culture of constant vigilance – these are your shields against the onslaught of zero-day attacks. It’s like the age-old adage in martial arts – the best form of defense is a strong foundation. In the next sections, we'll dive deeper into the specific tools and strategies that make our existing knowledge a formidable force against the enigmatic zero-days. Remember, in the world of cybersecurity, knowledge isn’t just power – it’s your best defense. Strengthening the Process You've identified the zero-day lurking in your system. Now, comes the critical part: beefing up your defense. Think of this as fortifying your digital castle against the siege of unknown threats. It's not about reinventing the wheel; it's about reinforcing the wheels you already have. First and foremost, let's talk about patch management. In the world of cybersecurity, patches are like the armor plating on your castle walls. When a zero-day is discovered, vendors often release patches to fix the vulnerability. It’s crucial to apply these patches promptly. Think of it as a race against time, with hackers on one side and your patching team on the other. Fast, efficient patch management isn’t just good practice; it’s your knight in shining armor in the fight against zero-days. Next up, we have threat intelligence. Knowledge is power, and in cybersecurity, it’s your eyes and ears on the digital battleground. Staying updated with the latest threat intelligence feeds can give you the upper hand. It’s like having scouts reporting back on enemy movements – the more you know about potential threats, the better prepared you are to defend against them. But here's the kicker: the process of strengthening your defenses is never truly complete. Cybersecurity is a dynamic field, constantly evolving. This means your strategies and processes must evolve too. Regular security assessments and audits are crucial. They help you identify potential weaknesses in your defenses – the chinks in your armor, so to speak – before they can be exploited. In essence, strengthening your process against zero-days is about being proactive rather than reactive. It's about building a culture of continuous improvement, where your defenses are always one step ahead of potential threats. In the next section, we'll explore the human factor – how training, awareness, and a robust cybersecurity culture can be your strongest allies in this ongoing war against digital threats. The Human Factor At the heart of every great cybersecurity strategy lies its people. The human element in managing zero-days is akin to the knights in the cybersecurity kingdom. It's about more than just skills and knowledge; it's about cultivating a mindset. Training and awareness are the twin pillars of this approach. Imagine every member of your team as a sentry, vigilant and ready. Regular training ensures that everyone is not just aware of the standard protocols but also the latest threats – including zero-days. It's like each person holds a piece of the puzzle; when put together, you get the full picture of your security posture. Creating a security-minded culture within your organization is akin to building a fortress with a strong foundation. It's not just about the tools and technologies; it's about how people think and act every day. Encouraging open communication about potential threats, fostering an environment where security is everyone’s responsibility, and rewarding proactive security behaviors – these are the building blocks of a resilient cybersecurity culture. Moreover, collaboration extends beyond the walls of your organization. Engaging with the wider cybersecurity community is crucial. Sharing experiences, learning from others’ incidents, and staying informed about emerging threats – this collective wisdom is a treasure trove in the fight against zero-days. Think of it as a council of cybersecurity wizards, each bringing their own magic (aka expertise) to the table. Wrapping This Up In our journey through the mysterious and often intimidating world of zero-day vulnerabilities, we've uncovered a fundamental truth: . Like a well-navigated labyrinth, the path to managing zero-days becomes clearer when we use the light of our existing knowledge, tools, and experiences. these unknown threats are not as invincible as they may seem We've seen how understanding zero-days, leveraging our existing expertise, adapting our tools, and continuously strengthening our processes form the bedrock of an effective defense strategy. But, perhaps most importantly, we've recognized the invaluable role of the human element – the vigilance, the culture, and the community collaboration that turn our collective efforts into an impenetrable shield against these hidden digital threats. As we part ways in this discussion, remember that in the ever-changing landscape of cybersecurity, adaptation and resilience are your truest allies. Zero-day vulnerabilities, daunting as they are, offer us opportunities to learn, grow, and fortify our defenses. So, as you navigate this challenging terrain, carry with you the knowledge that with the right approach, even the unknown can be tamed.

Struck by a Zero-Day: What's Your Next Move?

একটি জিরো-ডে দ্বারা আঘাত: আপনার পরবর্তী পদক্ষেপ কি?

Von einem Zero-Day getroffen: Was ist Ihr nächster Schritt?

Golpeado por un día cero: ¿cuál es su próximo paso?

Frappé par un Zero-Day : quelle est votre prochaine étape ?

एक शून्य दिवस से प्रभावित: आपका अगला कदम क्या है?

ゼロデイに襲われた: 次の行動は何ですか?

제로데이의 충격: 다음 단계는 무엇입니까?

Atingido por um dia zero: qual é o seu próximo passo?

Пораженный нулевым днем: каков ваш следующий шаг?

Sıfır Gün'ün Etkisi: Bir Sonraki Hamleniz Nedir?

Bị mắc kẹt bởi một ngày không: Bước đi tiếp theo của bạn là gì?

遭受零日攻击：您的下一步行动是什么？

Amazon Neptune, the managed graph database service by AWS, makes analytics faster and more agile while introducing a vision aiming to simplify graph databases. It’s not every day that you hear product leads questioning the utility of their own products. Brad Bebee, the general manager of Amazon Neptune, was all serious when he said that most customers don’t actually want a graph database. However, that statement needs contextualization. If Bebee had meant that in the literal sense, the team himself and Amazon Neptune Principal Product Manager Denise Gosnell lead would not have bothered developing and releasing a brand new analytics engine for their customers. We caught up with Bebee and Gosnell to discuss Amazon Neptune new features and the broader vision. We cover where Amazon Neptune fits in the AWS vision of data management, and how the new analytics engine provides a single service for graph workloads, high performance for graph analytic queries and graph algorithms, and vector store and search capabilities for Generative AI applications. We also share insights on the One Graph vision, the road from serverless to One Graph via HPC, as well as vectors and Graph AI. https://www.youtube.com/watch?v=hSq_ux926dY&embeds_widget_referrer=https%3A%2F%2Fmedium.com%2F%40linked_do%2Famazon-neptune-introduces-a-new-analytics-engine-and-the-one-graph-vision-ea2bb8bf4cf7&embeds_referring_euri=https%3A%2F%2Fcdn.embedly.com%2F&embeds_referring_origin=https%3A%2F%2Fcdn.embedly.com&feature=emb_imp_woyt&embedable=true Amazon Neptune and the AWS vision of data management Amazon Neptune is a part of the AWS vision of data management, which Bebee summarized as “an end-to-end architecture where customers don’t have to worry about how their data is stored or managed”. To do that, he went on to add, there are three different pillars. The first is having the most comprehensive set of services to store, analyze, and share data. The second is having solutions that make it easy to connect all data between the services that customers want to use. And the third is having the right governance and policy solutions in place so that teams can use the data effectively and quickly, but within policy and regulatory guidelines. Bebee believes that graphs are awesome, because they allow customers to innovate based on the relationships in their data. In a graph data model, relationships are a first class entity, which means you can ask questions and build applications that explore these relationships and the connections in the data. Bebee referred to some typical graph use cases they see in AWS clients such as Siemens and Wiz: Knowledge Graphs, and graph analytics for Identity, Fraud and Security. For the latter 3, Amazon provides open source with examples leveraging Apache TinkerPop and Gremlin. Jupyter notebooks , one based on the RDF model with SPARQL access and one on property graphs with Gremlin and openCypher access, so we wondered what are the usage patterns the team see for those. Neptune sports two engines “Many customers are using RDF to build knowledge graphs, particularly those who are really thinking deliberately about their information architecture and their information models. But we also see a large number of customers choosing to build knowledge graphs with property graph. I think that’s speaking a lot to the value that customers see by relating the data and that it transcends the choice of the graph model for those kinds of use cases” Bebee said. Introducing Amazon Neptune Analytics Awesome as graphs may be, however, Bebee acknowledged there are challenges as well. When accessing a graph, the way that you need to touch the data is often random. If you think about how one person is connected to another is connected to another, the way that you lay out and store that data internally in the system makes it very difficult to predict how you’re going to access it. This is where the new analytics engine comes in. As Gosnell shared, the goal of the new analytics engine is to enable better data discovery by analyzing large amounts of graph data with billions of connections on the fly. Amazon Neptune Analytics has three main features: a single service for graph workloads, high performance for graph analytic queries and graph algorithms, and vector store and search capabilities for Generative AI applications. A single service for working with graphs can now be used to invoke popular graph algorithms, run low latency queries, and perform vector similarity search all from a single API. This API currently supports openCypher, with SPARQL and Gremlin on the roadmap as well. The choice made regarding prioritization may tell us something about the dynamics Amazon sees in terms of market demand too. Gosnell said that the high-performance graph computing techniques applied in the new analytics engine have proven to be about 100 times faster for loading data. When running graph analytic queries and performing graph algorithms, scans are 20 times faster and columnar scans about 200 times faster. Last but not least, Neptune analytics can store embeddings on nodes. The library can be used to translate natural language questions into openCypher queries, lowering the bar to entry to working with graphs. LangChain Gosnell also shared three use cases for Neptune Analytics. First, ephemeral analytics. These are workflows where customers just need to spin up a graph really quickly, run some analysis, and turn it off. Second, low-latency analytical queries. That involves established machine learning pipelines with feature tables to perform real-time predictions. The new analytics engine enables customers to run highly-concurrent query workloads to augment their existing feature tables with new analytics about their graph structure. That gives their ML models much higher prediction rates and overall higher end-user engagement. The third use case is building GenAI applications. Being able to perform vector similarity search when storing embeddings on Neptune analytics means it’s much easier to translate natural language questions into graph queries, because the way we think about data just so happens to fit really well with how a graph structures it as per Gosnell. The One Graph vision That all deserves some analysis. The first thing to note is that while Neptune did not lack analytics features previously, mostly a function of various graph algorithms being implemented in the database, this is a whole new engine designed from the ground up specifically for analytics. While the performance and use cases sound promising, that also implies data movement. As the Neptune team shared, users can spin up a graph with Neptune Analytics, connect the endpoint to their Neptune cluster, and it’ll automatically ETL data from Neptune into Neptune Analytics. Despite having to move data, that sounds convenient. Especially if loading times are as fast at the Neptune team claims. Bebee noted that the speedup was achieved on two pillars. The first one was moving to more parallelism in loading and changing the way the data is partitioned. The second one was making use of proprietary storage technology that was originally built for other databases in AWS, leveraging a log-based storage mechanism. Potentially the most important thing about the new analytics engine however is the one hiding in plain sight: the notion of using a single API. That API will be used to manage the end-to-end workflow of doing graph analytics. In other words — no provisioning or setting up nodes and clusters and the like. This sounds like the equivalent of Lambda functions for Neptune: zero setup, hit-and-run analytics. The Neptune team referred to this as the One Graph vision. “One of the things we’ve learned is that most customers don’t actually want a graph database. This sounds a little heretical, but what I mean here is that they want graphs and they want to store and query their graphs, but they don’t want to create instances and clusters and have another database management system in their IT infrastructure. We may not have it exactly right yet, but we really think that moving away from the database management system abstraction of consuming graphs and moving more towards a simpler API that gets customers storing and querying their graph data faster feels like the right thing based on what we’ve learned from customers”, said Bebee. https://pod.co/orchestrate-all-the-things-podcast-connecting-the-dots-with-george-anadiotis/amazon-neptune-introduces-a-new-analytics-engine-and-the-one-graph-vision-featuring-brad-beebe-denise-gosnell-amazon-neptune-general-manager-principal-product-manager?embedable=true From serverless to One Graph via HPC Pricing for Neptune Analytics is going to be based on memory-optimized units, i.e. how much compute customers use for Neptune Analytics per hour. The maximum capacity of a new graph in terms of gigabytes of memory can be specified explicitly. It can also be automatically determined when importing data from S3 or Neptune, with an overall max capacity so that customers can control their budget. Also, both teams have members who are familiar with the literature and techniques coming from high performance computing (HPC) processing of large scale graphs. The parallel processing and memory optimization techniques are things that are well-understood in the HPC community, Bebee noted. The difference is that HPC researchers are often solving a very specific graph problem on a very specific graph. For a service that has to solve graph problems for many customers, the main challenge was generalizing the techniques that can work well for HPC. Another common thread with Neo4j is the newly-added vector store and search capabilities. Bebee believes that specialized vector databases may be needed in certain scenarios, but one of the benefits of putting vectors into an existing database is that it makes it a lot easier and faster and to use without having to move data around. Adding vector capabilities to Neptune Analytics means that vectors can be combined with graph searches and graph algorithms, something that the team internally refers to as vector-guided navigation. That is, using the explicit relationships and properties in graphs and combining them at certain points with the statistical capabilities that you get from vector similarity search. Bebee also noted that the other use case for vectors in the graph is vectors that are not necessarily coming from LLMs, but vectors that are coming out of Graph Neural Networks (GNNs). Embeddings coming out of GNNs can be stored in Neptune Analytics and then be used via cosine similarity to do link prediction. Conclusion Overall, the new analytics engine from Amazon Neptune exemplifies : more agile and faster analytics, support for graph-based machine learning, and different flavors of Generative AI capabilities. the direction that graph databases are taking in the last couple of years Neptune, like many other databases, now provides vector capabilities and ways to have a natural language interface through which to interact with the database. Its team argues that vector capabilities coupled with graph analytics, machine learning and knowledge graphs may actually be a more powerful combination than a pure play vector database. One of the things we’ll see in the future, they said, is the One Graph vision start to be realized within the Neptune Analytics platform. In that respect, we may look back and find that the graph API abstraction was the most impactful thing introduced at this time. Also appears . here

Amazon Neptune Launches a New Analytics Engine and the One Graph Vision

Stories, News & Mentions about GREMLIN

Gremlin’s Stories on HackerNoon