As Data Scientists, people tend to think what they do is developing and experimenting with sophisticated and complicated algorithms, and produce state of the art results. This is largely true. It is what a data scientist is mostly proud of and the most innovative and rewarding part. But what people usually don’t see is the sweat they go through to gather, process, and massage the data that leads to the great results. That’s why you can see SQL appears on most of the data scientist position requirements. What is SPARQL? There is another query language that could prove very useful in acquiring data from multiple sources and databases, Wikipedia the biggest among them. The query language is called . According to Wikipedia: SPARQL (pronounced “ ”, a for ) is an — that is, a for — able to retrieve and manipulate data stored in format SPARQL sparkle recursive acronym [2] SPARQL Protocol and RDF Query Language RDF query language semantic query language databases Resource Description Framework (RDF) Well, this is not a very good definition. It hardly tells you what it can do. To translate it into human-readable language: SPARQL is a query language similar to SQL in syntax but works on a knowledge graph database like Wikipedia, that allows you to extract knowledge and information by defining a series of filters and constraints. If this is still too abstract to you, look at the image below: (Awarded Chemistry Nobel Prizes) It is a timeline of awarded chemistry Nobel prizes, generated by the website, using the code below: WikiData Query Service ?item ?itemLabel ? ( (? ) ? ) ?pic {
  ?item p:P166 ?awardStat . ?awardStat ps:P166 wd:Q44585 . ?awardStat pq:P585 ? . SERVICE wikibase:label { bd:serviceParam wikibase: . }
OPTIONAL { ?item wdt:P18 ?pic }
} #Awarded Chemistry Nobel Prizes #defaultView:Timeline SELECT DISTINCT when YEAR when as date WHERE # … with an awarded(P166) statement # … that has the value Nobel Prize in Chemistry (Q35637) when # when did he receive the Nobel prize language "en" Anyone familiar with SQL will find the above code quite intuitive. I’ll use another example to explain basically how to formulate similar queries to achieve the results you interested in. Starting Point: Wikipedia Page SPARQL works on multiple knowledge graph databases. To know what knowledge graph is, let’s start with something everyone is familiar with: Wikipedia. Wikipedia is the go-to place for most people when they want to research a topic/subject. If you go to Python creator Guido van Rossum’s page, you’ll see a detailed page with all kinds of good information. Organized Page: WikiData The problem with this page is it’s not organized. You can search on keywords, but you cannot easily find out the relationship between the information nodes. That’s where the knowledge graph comes into play. The red rectangle on the above page spells “Wikidata Item”, click it will bring you to the knowledge graph view of the same page: Start the Query Here we can see all information about Guido is well organized into categories, each category has multiple items. Use SPARQL, you can easily query this information. To do this, Wikipedia provides another page, a user-friendly query service called : Wikidata Query Service This is where we can experiment with SPARQL. On the WikiData page, we observed that Guido is a programmer (obviously!), now what if we want to know other programmers that have an entry on Wikipedia? Let’s see the SPARQL code: ?person {
?person wdt:P106 wd:Q5482740 .
} SELECT WHERE Here we defined a as the of interest, this is also what will appear as a column in our query results. Then we specify some constraints with . The constraints are need to be . What? You say. Let me explain it in more detail. is a prefix of a ‘predicate’ or ‘attribute’ of the subject while is the prefix of a value(object in SPARQL terms, but that’s not important) of the attribute. means I am gonna specify an attribute of the subject here, and means I will specify what the value of this attribute is. So what is and ? These are just a code for the specific attribute and value. stands for ‘occupation’ and stands for ‘programmer’. This line of code means, I want the subject to have an attribute of ‘occupation’ of ‘programmer’. Not that scary anymore, right? You can find these codes easily on the WikiData page mentioned above. ?person subject WHERE wdt:P106 wd:Q5482740 wdt wd wdt: wd: P106 Q5482740 P106 Q5482740 ?person Run the query and you’ll get the following results: From Code to Name We got a bunch of items with different . If you look closer at the value, they are actually the code for a different person. For example, the first one is , the inventor of WWW. This is not intuitive, we want to be able to directly see the names. To do that, we add a WikiData ‘label service’ that helps us translate the code to name, like so: person wd:value wd:Q80 Tim Berners-Lee ?person ?personLabel {
  ?person wdt:P106 wd:Q5482740 .
  ?person rdfs:label ?personLabel .
  FILTER ( LANGMATCHES ( LANG ( ?personLabel ), ) ) 
  
} SELECT WHERE "fr" Similar syntax, we want the to have a ‘label’ attribute, and we define a value variable to hold these values so we can display them in the query results. Also, we added the into our phrase so it will be displayed. Please be noted that I also added a FILTER below to only display the French language label, otherwise it will show multiple language labels for one person, which is not what we want: person personLabel personLabel SELECT Narrowing Down From the above results, we can see that we have some 790 results. This is way too many. Let’s narrow them down to the ones that are ‘somebody’ in the industry. Someone that has an attribute of ‘notable work’: ?person ?personLabel ?notableworkLabel {
?person wdt:P106 wd:Q5482740 .
  ?person rdfs:label ?personLabel .
  FILTER ( LANGMATCHES ( LANG ( ?personLabel ), ) ) 
  
  ?person wdt:P800 ?notablework .
  ?notablework rdfs:label ?notableworkLabel .
  FILTER ( LANGMATCHES ( LANG ( ?notableworkLabel ), ) ) 
  
} SELECT WHERE "fr" "fr" Again, means ‘notable work’ attribute, everything else is similar. We then get the following results: wdt:P800 Group Multiple Labels Now we have only 175 results, with each person’s notable work also listed. But wait, why we have five Richard Stallman? Turns out, Richard has more than one notable work, thus listed multiple times in the results. Let’s fix this by grouping all the notable work into one attribute: ?person ?personLabel ( ( ?notableworkLabel; separator="; " ) AS ?works )
WHERE {
?person wdt:P106 wd:Q5482740 .
  ?person rdfs:label ?personLabel .
  FILTER ( LANGMATCHES ( LANG ( ?personLabel ), "fr" ) ) 
  
  ?person wdt:P800 ?notablework .
  ?notablework rdfs:label ?notableworkLabel .
  FILTER ( LANGMATCHES ( LANG ( ?notableworkLabel ), "fr" ) ) 
  
}
GROUP BY ?person ?personLabel SELECT GROUP_CONCAT DISTINCT Here ‘ ’ is used. Also, function is used to concatenate multiple into a new column ( : GROUP BY GROUP_CONCAT notableworkLabel works I will not explain how these functions work, just want to quickly show you what SPARQL can do. Please feel free to Google if you want to know more, there are plenty of tutorial articles and videos out there) Faces Now we have a 90 results list, with all the ‘who-is-who’ in the software engineering world. But SPARQL can do more. Let’s add some faces to the names: ?person ?personLabel ( ( ?notableworkLabel; separator="; " ) AS ?works ) ?image
WHERE {
?person wdt:P106 wd:Q5482740 .
  ?person rdfs:label ?personLabel .
  FILTER ( LANGMATCHES ( LANG ( ?personLabel ), "fr" ) ) 
  
  ?person wdt:P800 ?notablework .
  ?notablework rdfs:label ?notableworkLabel .
  FILTER ( LANGMATCHES ( LANG ( ?notableworkLabel ), "fr" ) ) 
  
  OPTIONAL {?person wdt:P18 ?image}
  
}
GROUP BY ?person ?personLabel ?image SELECT GROUP_CONCAT DISTINCT The same pattern, we just added an OPTIONAL keyword prior since we don’t want to exclude someone if he doesn’t have an image in his profile. We also switch the view into ‘Image Grid’ : Where are they? Wow! This is much better. I saw quite some familiar faces! Maybe you wonder where are these guys located? Let’s find out: ?person ?personLabel ( ( ?notableworkLabel; separator="; " ) AS ?works ) ?image ?countryLabel ?cood
WHERE {
  ?person wdt:P106 wd:Q5482740 .
  ?person rdfs:label ?personLabel .
  FILTER ( LANGMATCHES ( LANG ( ?personLabel ), "fr" ) ) 
  
  ?person wdt:P800 ?notablework .
  ?notablework rdfs:label ?notableworkLabel .
  FILTER ( LANGMATCHES ( LANG ( ?notableworkLabel ), "fr" ) ) 
   
  
  OPTIONAL {?person wdt:P18 ?image}
  OPTIONAL {?person wdt:P19 ?country .
           ?country rdfs:label ?countryLabel .
            ?country wdt:P625 ?cood .
            FILTER ( LANGMATCHES ( LANG ( ?countryLabel ), "fr" ) )
           }
}
GROUP BY ?person ?personLabel ?image ?countryLabel ?cood #defaultView:ImageGrid SELECT GROUP_CONCAT DISTINCT You can decipher the code above yourself maybe. It basically says I want this person to have an attribute of , put into a variable , then find out the of the country and put into a variable . With the coordinates, we can activate the ‘map’ view: country country coordinates cood We can see we have a lot of them in the US, some in Europe and others scattered around other parts of the world. More Examples With a few lines of codes, we figured out the big influencers in the software industry, what they are known of, where they are and how they look. As you can see the potential here is limitless. You can click the ‘ ’ button on the WikiData page to find out more fun and interesting examples you can do with it. Example As an assignment for this article, can you figure out how to add the ‘date of birth’ attribute and generate a timeline graph like the one at the beginning of this article? Conclusion In this article, we used WikiData as a knowledge graph example to introduce SPARQL query language. There are other knowledge graphs out there like , etc. This article is by no means a comprehensive tutorial. I just want to introduce the language to more people, so knowledge and information extraction can be done a bit more efficiently. DBpedia Found this article useful? You can find me on Twitter or my blog site . @lymenlee wayofnumbers.com Previously published at https://towardsdatascience.com/how-to-extract-knowledge-from-wikipedia-data-science-style-35f50f095d1a

BUNCH

Google

Twitter

Winner of Noonies 2020 for the award Hacker Hacker Noon Contributor of the Year - SELF-DRIVING CARS.

2020 - Hacker Hacker Noon Contributor of the Year in SELF-DRIVING CARS

Visit My Blog!

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Extract Knowledge from Wikipedia, Data Science Style

Too Long; Didn't Read

Companies Mentioned

Michael Li

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Extract Knowledge from Wikipedia, Data Science Style

Too Long; Didn't Read

Companies Mentioned

Michael Li

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES