Tracking the Worst Sci-Fi Movies With Angular and Slash GraphQL

I originally discovered (MST3K) by mistake. Mystery Science Theater 3000 In order to avoid missing a movie premiere on the HBO network, I set my VCR to record the program. However, when I started to watch the recording, I quickly realized I had recorded " " instead of the HBO premiere production. After recognizing the images of Mike Nelson and the two robots from years of scanning channels, I decided to give my mistaken recording a try. Mystery Science Theater 3000: The Movie After five minutes of watching the critiqued version of "This Island Earth," I knew I was hooked. I was already in tears from gut-busting laughter. These guys were comic geniuses. For those who have no idea what I am talking about, MST3K was an American television comedy series that ultimately ran for 12 seasons. At the heart of the series is a lead character who gets captured by an evil villain and shot into space, then forced to watch really bad movies. Along the way, the main character builds a couple robots to keep him company while watching these bad films. Their silhouettes can be seen in the corner of the screen as the subpar movies unfold. Their quite-comical observations poke fun of the production and make the series a lot of fun to watch. Since I was still interested in doing more with , I thought it would be really cool for us to create a modern-day MST3K wish list. You know, in case the series were to be picked up again. Slash GraphQL Selecting Dgraph's Slash GraphQL A graph database is an ideal solution when the source data handles recommendation and personalization needs. Such functional requirements often place the value of data relationships on the same level as the attributes which are being persisted. In this example, the use of ratings for a given movie title is just as important as the title attributes which will be analyzed and presented, making a graph database the preferred approach. Since September, Dgraph has offered a fully managed backend service, called . Along with a hosted graph database instance, there is also a RESTful interface. This functionality, combined with 10,000 free credits for API use, allows us to rely fully on the Dgraph services. That way, we don’t have to introduce another layer of services between the graph database and web-based client. Slash GraphQL Our technology stack will be quite simple, but effective: Dgraph Slash GraphQL to house the dataDgraph Slash GraphQL to provide a GraphQL API for accessing the dataAngular CLI to create an application for presenting the data With service/storage choice set on using Slash GraphQL, our next step is to figure out how to get data for the project. Getting Data From IMDb (IMDb) has been my primary source of film data for the better part of twenty years. In addition to the details of any movie, there is a crowd-sourced five-star rating system available for each title. At a glance, the consumer can see both the average rating and the number of votes used to reach the current score. These data points will be perfect for our new application. Internet Movie Database For the modern-day MST3K wish list, we’ll use the following criteria to build our list of bad sci-fi movies for consideration: genre must include "Sci-Fi" limited to movie types (exclude shorts, made-for-TV movies, series, etc.) excludes titles with fewer than 500 ratings We will focus on the bottom 125 of movies ranked by imdb.com. IMDb Datasets make subsets of IMDb data available to customers for personal and non-commercial use. On a periodic basis, a series of TSV files are available for download. Upon reviewing the list of files, two of those seem to fit our needs: IMDb Datasets title.basics.tsv.gz — contains basic information for IMDb titlestitle.ratings.tsv.gz — contains the ratings and votes for IMDB titles As one might imagine, these files are quite large — especially when extracted. We need a mechanism to filter these data source files. Filtering the TSV Files Using Java Using , I created a simple class which would accomplish the following steps: IntelliJ IDEA read each line of the title.basics.tsv file determine if the line contains the "Sci-Fi" genre if so, capture the title ID attribute as the key to a Map and place the entire line as the value of the map if any match is found, process the title.ratings.tsv file read each line of the ratings file and capture the title ID if the title ID attribute exists in the Map , add the rating and votes data to the value of the map entry create a new TSV file which contains the Sci-Fi title information, plus the average user rating and number of votes Below is the very simple entry point into the Java program: { String DEFAULT_GENRE = ; String USER_HOME = ; String DELIMITER = ; String TITLE_BASICS_TSV_FILE_LOCATION = ; String TITLE_RATINGS_FILE_LOCATION = ; String DESTINATION_FILE = ; { String genre = DEFAULT_GENRE; (args != && args.length > ) { genre = args[ ]; } Collection data = filterData(TITLE_BASICS_TSV_FILE_LOCATION, genre); (CollectionUtils.isNotEmpty(data)) { writeFile(data, DESTINATION_FILE); } } ... } public class Application private static final "Sci-Fi" private static final "user.home" private static final "\t" private static final "/downloads/title.basics.tsv" private static final "/downloads/title.ratings.tsv" private static final "/downloads/filtered.tsv" IOException public static void main (String[] args) throws if null 0 0 if The main filtering code is shown below: { Map data = HashMap<>(); (BufferedReader br = BufferedReader( FileReader(System.getProperty(USER_HOME) + fileName))) { String string; lineNumber = ; ((string = br.readLine()) != ) { (lineNumber > && StringUtils.contains(string, genre)) { String firstItem = StringUtils.substringBefore(string, DELIMITER); data.put(firstItem, string); } logResults(lineNumber, fileName); lineNumber++; } (MapUtils.isNotEmpty(data)) { appendUserRatings(data, TITLE_RATINGS_FILE_LOCATION); } } data.values(); } Collection IOException private static filterData (String fileName, String genre) throws new try new new long 0 while null if 0 if return The code to process the average rating and total votes TSV file is shown below: { (BufferedReader br = BufferedReader( FileReader(System.getProperty(USER_HOME) + fileName))) { String string; lineNumber = ; ((string = br.readLine()) != ) { (lineNumber > ) { String firstItem = StringUtils.substringBefore(string, DELIMITER); (data.containsKey(firstItem)) { data.put(firstItem, data.get(firstItem) + DELIMITER + StringUtils.substringAfter(string, DELIMITER)); } } logResults(lineNumber, fileName); lineNumber++; } } } IOException private static void appendUserRatings (Map data, String fileName) throws try new new long 0 while null if 0 if Finally, the following helper methods were added: private writeFile(Collection data, fileName) throws IOException { (BufferedWriter bw = BufferedWriter( FileWriter(System.getProperty(USER_HOME) + fileName))) { ( str : data) { bw.write(str); bw.newLine(); } } } private logResults(long lineNumber, fileName) { (lineNumber % == ) { System.out.println( + lineNumber + + fileName + ); } } static void String String try new new for String static void String if 10000 0 "Completed " " " " records" Locating the Bottom 125 With a filtered.tsv file now ready, we can use to narrow the data down to a manageable size of the 125 worst-rated Sci-Fi movies. Based on the Java program’s results, here are our columns: Microsoft Excel idtitleType primaryTitle originalTitle isAdult startYear endYear runtimeMinutes genres averageRating numVotes The following actions were taken in Microsoft Excel: only "movie" value for the titleType column remove any values where isAdult is greater than zero only items which have a value greater than or equal to 500 in the numVotes column We can now sort the list by the averageRating column, where the lowest rating is at the top of the list. Next, copy the top 125 records and drop this data into another sheet. Let’s also remove all but the following columns: id primaryTitle (which will become title) startYear (which will become releaseYear) runtimeMinutes genres (which will become genre) averageRating numVotes (which will become votes) To prepare for use by Dgraph Slash GraphQL, use the CONCAT function in Microsoft Excel to create a new column for each row of data which presents the data in the following JSON format: {id: , title: , releaseYear: ,runtimeMinutes: ,genre: ,averageRating: ,votes: }, "tt5311054" "Browncoats: Independence War" 2015 98 "Action,Sci-Fi,War" 1.1 717 At this point, the source data is ready for use by Dgraph Slash GraphQL. Using Slash GraphQL In the " " article, I walk through the necessary steps to create a free Dgraph Slash GraphQL account, which provides 10,000 free credits. Getting started is as simple as navigating to the following URL: Building an Amazon-Like Recommendation Engine Using Slash GraphQL https://slash.dgraph.io/ Since I still have a significant number of credits available for my account, I decided to create a new backend service called bad-scifi-movies to house the data extracted from IMDb. This action provided me with a GraphQL Endpoint value in the Overview section of the Dgraph user interface, which will be referenced in the Angular client setup. Next, the schema for the new backend service needs to be created. We’ll keep things simple— the Slash GraphQL schema is noted below: type Movie { id: String! @id @search(by: [hash]) title: String! @search(by: [fulltext]) releaseYear: Int! @search runtimeMinutes: Int! genre: String! @search(by: [fulltext]) averageRating: Float! @search votes: Int! @search seen: User } type User { username: String! @id @search(by: [hash]) movies: [Movie] @hasInverse(field: seen) } The Movie object will house all of the data filtered from IMDb. For the sample application, the User object will contain a unique username and a list of really bad sci-fi movies seen by that user. With the schema created, it is time to insert data into Dgraph Slash GraphQL. To insert the Movie data, that JSON-based column in Microsoft Excel needs to be copied. Below, is an abbreviated example of the insert command: mutation AddMovies { addMovie(input: [ { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, ... more data here ... { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : } ]) } id "tt5311054" title "Browncoats: Independence War" releaseYear 2015 runtimeMinutes 98 genre "Action,Sci-Fi,War" averageRating 1.1 votes 717 id "tt2205589" title "Rise of the Black Bat" releaseYear 2012 runtimeMinutes 80 genre "Action,Sci-Fi" averageRating 1.2 votes 690 id "tt1854506" title "Aliens vs. Avatars" releaseYear 2011 runtimeMinutes 80 genre "Horror,Sci-Fi" averageRating 1.5 votes 1584 JSON id "tt0068313" title "Brain of Blood" releaseYear 1971 runtimeMinutes 87 genre "Horror,Sci-Fi" averageRating 2.9 votes 727 id "tt1754438" title "Robotropolis" releaseYear 2011 runtimeMinutes 85 genre "Action,Adventure,Sci-Fi" averageRating 2.9 votes 1180 Please note: at the end of this article, there will be a link to the GitLab repository, which includes the full list of 125 movies. For the purposes of this example, we’ll utilize a single User object: mutation AddUser { addUser(input: [ { : , : [ { : }, { : }, { : }, { : }, { : }, { : }, { : }, { : }, { : } ] } ]) { numUids } } username "johnjvester" movies id "tt0052286" id "tt0077834" id "tt0145529" id "tt0053464" id "tt0060074" id "tt0075343" id "tt0089280" id "tt0059464" id "tt0055562" Once all of the Movie objects have been inserted, the johnjvester User has watched a total of 9 of the 125 really bad sci-fi movies. At this point, the new backend service is available for use at the GraphQL Endpoint noted in the Overview section of the Dgraph interface. Adding the Movie Poster Showing only the raw data for these movies would be okay, but when the application’s user switches from the list-view to the detail-view, we want them to see the movie poster for the title. However, the IMDb extractions do not provide this information. Using Google, I was able to locate the open-movie database (OMDb) API, which just happens to contain a link to the movie poster. Additionally, the OMDb API allows for items to be queried using the same unique key which is used by IMDB. However, an API key would be required. In order to show the movie poster in the Angular application, a free OMDb API key is required: Visit to request an API key. http://www.omdbapi.com/apikey.aspx Select the FREE option and provide an email address. Single-click the Submit button and follow any required follow-up steps. Note the “Here is your key” value provided via email from The OMDb API. Now, when combined with the source data from IMDb, adding the movie poster image is a small API request, which Angular can certainly perform without much effort. In reviewing the Slash GraphQL documentation, I later learned the API call to retrieve the movie poster could have been added to the Slash GraphQL schema using the directive - making it included in the GraphQL query. @custom Creating the Angular Application The Angular CLI is very easy to use. Our sample application will use the base component to serve as the view into the Slash GraphQL data. As one might imagine, this data would be presented in a table format. When the user clicks on a row in the table, a basic modal will be displayed, showing the full details for the title (including the movie poster) via the integration with the OMDb API. Interacting directly with Dgraph Slash GraphQL is handled by a service called graph-ql.service.ts: @Injectable({ : }) { allMovies:string = ; singleUserPrefix:string = ; singleUserSuffix:string = ; (private http: HttpClient) { } baseUrl: string = environment.api; getMovies() { .http.get ( .baseUrl + + .allMovies).pipe( tap(), catchError( { ErrorUtils.errorHandler(err) })); } getUser(username:string) { .http.get ( .baseUrl + + .singleUserPrefix + username + .singleUserSuffix).pipe( tap(), catchError( { ErrorUtils.errorHandler(err) })); } } providedIn 'root' export class GraphQLService '{queryMovie(filter: {}) {votes, title, runtimeMinutes, releaseYear, id, genre, averageRating}}' '{getUser(username:"' '"){username,movies{title,id}}}' constructor return this this '?query=' this => err return return this this '?query=' this this => err return The communication to the OMDb API utilizes the omdb.service.ts: @Injectable({ : }) { (private http: HttpClient) { baseUrl: string = environment.omdbApi + environment.omdbKey; getMoviePoster(id:string) { .http.get ( .baseUrl + + id).pipe( tap(), catchError( { ErrorUtils.errorHandler(err) })); } } providedIn 'root' export class OmdbService constructor return this this '&i=' => err return The Angular environment.ts file includes a few custom attributes as shown below: environment = { : , : , : , : }; export const production false api 'https://some-host-instance.us-west-2.aws.cloud.dgraph.io/graphql' omdbApi 'http://www.omdbapi.com/?apikey=' omdbKey 'omdbApiKeyGoesHere' The API value should be replaced with the GraphQL Endpoint value for the backend service with Dgraph Slash GraphQL. The omdbKey is the unique value received via email from "The OMDb API." When launching the Angular application, the following OnInit method is executed: ngOnInit() { .graphQlService.getMovies() .subscribe( { (data) { queryMovieResponse: QueryMovieResponse = data; .movies = queryMovieResponse.data.queryMovie; .movies.sort( (a.title > b.title) ? : ) } }, (error) => { .error( , error); }).add( { }); } this => data if let this this ( ) => a, b 1 -1 console 'error' => () The GraphQlService is used to make a GraphQL API call to Slash GraphQL. That call, which eliminates the need to massage data in the Angular application, retrieves the list of 125 really bad sci-fi movies. The list is then sorted by title, by default. When users click a movie, a modal opens and the following OnInit is executed: ngOnInit() { ( .movie && .movie.id) { .omdbService.getMoviePoster( .movie.id) .subscribe( { (data && data.Poster) { .posterUrl = data.Poster; .graphQlService.getUser( .username) .subscribe( { (getUserResponse && getUserResponse.data && getUserResponse.data.getUser) { .user = getUserResponse.data.getUser; .hasSeenThisMovie(); } }, (error) => { .error( , error); }).add( { }); } }, (error) => { .error( , error); }).add( { }); } } if this this this this => data if this this this => getUserResponse if this this console 'error' => () console 'error' => () The OmdbService is used to retrieve the URL for the movie poster and the GraphQLService retrieves the list of movies for the user. The user data determines the value of the hasSeenThisMovie boolean attribute. The simple boolean will determine if the button in the template will be active or inactive. Mark as Watched To make things appear a little nicer, try including the following packages in the Angular project: @ng-bootstrap/ng-bootstrap: ^5.3.1 angular-star-rating: ^4.0.0-beta.3 bootstrap: ^4.5.2 css-star-rating: ^1.2.4 Running npm ci (or npm install) made sure all the Node modules were installed. Now we can start the Angular application, using the ng serve command. Using the Angular Application With the Angular application running and the Slash GraphQL running, the following screen should be displayed: Single-clicking an item on the list for a movie not seen by the johnjvester user appears as shown below: Notice the button is . Mark as Watched active Single-clicking an item on the list for a movie in which johnjvester has watched appears as shown below: Notice the button is , since this movie has already been seen. Mark as Watched inactive Conclusion In the example above, we created a fully functional application using an instance of Dgraph Slash GraphQL and the Angular CLI. While the example provided here was on the simple side, the GraphQL features made available by Slash GraphQL allow for diverse and complex graph-database operations. This approach abstracted the design to not require use of any GraphQL libraries, resulting in a really nice GraphQL database backend from the Slash service that can be treated just like any other API. However, if additional needs are required which fall into the graph database realm (as found in my “ ” article), Slash GraphQL is ready to assist with meeting any functional needs. Connecting Angular to the Spring Boot and Slash GraphQL Recommendations Engine This article has demonstrated that it is possible to leverage Slash GraphQL as both a data source and a service layer with your application. This could translate to impressive cost savings over the life of an application. When including the @custom directive, the Angular client no longer has to make a second call for the movie poster, which is handled and included in the payload provided by Slash GraphQL. In fact, the design of the application becomes simplified with the API key moving from Angular into the Slash GraphQL instance - which is a lot easier to secure from anyone with source control access. If you are interested in the full source code for the Angular application, including the Java code referenced above, please visit the following repository on GitLab: https://gitlab.com/johnjvester/slash-graphql-bad-scifi-movies Have a really great day!