Flix-Finder: Building a Movie Recommendation App With Django and Bright Data [Part 3/3]

By the end of the previous blog, Flix-Finder supported movie searches based on rating parameters from various agencies. The functionality of our app met expectations and achieved its intended function. But after a while, Flix-Finder's customers began to gripe that the app did not provide the most recent information about films. Because Flix-Finder employs static data, this is something we anticipate.

Flix-Finder needs some process through which its movie data can be updated to reflect the new movies. And, this is something we would be implementing in this final part!

Despite the fact that there are several ways to implement this functionality, we'll be creating a new endpoint that can be used to update the data. Your data would change instantaneously, much as when you press a button. But there are other strategies as well, such utilising a cronjob. Once the specifics of one strategy are described in this blog, any new technique may be easily created using the same concepts.

Planning it all out…

Before we start to code, let us take a step back and understand all our requirements.

We want to create a new endpoint and hitting that endpoint should update the data Flix-Finder uses. Let us use /update endpoint and any GET request hitting this endpoint should update the data our app uses.

With the interface planned out, let us now think about ‘How to update the data?‘.

The process of updating the data can be broken into three steps:

(1) Re-Run the Collectors on Bright Data

(2) Fetch the Results obtained from running the collector

(3) Overwrite the results to static files

For the first two steps, we can use the APIs provided by Bright Data. Once we have the data, we can use vanilla Python to overwrite the existing static files.

Great! With everything planned out, let us start the implementation.

Adding the new Endpoint to Flix-Finder

Start by creating a new method inside the view.py file of recommender app in Django. We will call this method updateData. This is where all our logic of updating the static data will reside.

# recommender/views.py

def update(request):
  # TODO: Implement this method
  return HttpResponse("Movie Data Updated!")

Let us set up our views as well. We need to add the mapping for the newly created method in views.

# recommender/urls.py

from django.urls import path

from . import views

urlpatterns = [
    path("", views.index, name = "index"),
    path("update", views.updateData, name = "update data")
]

With this, we have successfully created a new endpoint in Flix-Finder. We will be using this endpoint to update the static data. For now, this endpoint does not do anything and simply returns a String message.

➜  ~ curl localhost:8000/update        
Movie Data Updated!                                                            

Generating API Token on Bright Data

Before we can start using Bright Data’s API for data update, we need to generate an API Token.

All communications to Bright Data should contain this token and it is used for authentication purposes.

Steps to generate API Token :

1. Head over to https://brightdata.com/cp/setting.
2. In the API Token section, click on Add Token button.
3. In the dialog box, select the user, the permission and the expiration date.
4. Click on the save button and enter the 2FA code.
5. Copy and securely keep the generated token.

Once the API Token is generated, we are ready to start using the APIs for running the collectors and fetching data from collector runs.

APIs for Initiating Collectors

Head over to Initiate by API section of your collectors.

Here you will find curl requests that you can use to re-run collectors and fetch the results.

API to initiate collector run: Bright Data provides an endpoint to initiate collector run. The endpoint https://api.brightdata.com/dca/trigger_immediate expects a POST request. It also expects a query parameter ‘collector’ that is used to uniquely identify the collector. The header should contain the API token.

Here is the curl request for the same.

curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '{}' "https://api.brightdata.com/dca/trigger_immediate?collector=c_rj3i0wspnhr0ataba" 

The response of this endpoint is a response ID. This can be used to later fetch the results.

API to fetch results from a collector’s run: With the response ID obtained from running a collector, Bright Data provides us with another API to fetch results from the collector run. The endpoint https://api.brightdata.com/dca/get_result expects a GET request. It also expects a query parameter ‘response_id’ that is used to uniquely identify the collector. The header should contain the API token.

Here is how the curl request for this endpoint would look.

curl "https://api.brightdata.com/dca/get_result?response_id=RESPONSE_ID" -H "Authorization: Bearer API_TOKEN"  

Implementing the update endpoint

With the knowledge of endpoints to use and API token handy, we can start implementing the update endpoint.

Let us implement this endpoint according to the subtasks we identified initially.

(1) Re-Run the Collectors on Bright Data

(2) Fetch the Results obtained from running the collector

(3) Overwrite the results to static files

First we need to use the re-run collector API and get a response ID. We will be using Python’s requests library to perform this task.

API_TOKEN = "###########" # Replace by your API Token
response = requests.post(
    "https://api.brightdata.com/dca/trigger_immediate?collector=c_rj3i0wspnhr0ataba",
    headers={"Authorization": f"Bearer {API_TOKEN}", "Content-Type": "application/json"},
    json={}
)
if response.status_code < 200 or response.status_code > 299 or 'response_id' not in response.json():
    return HttpResponseServerError('Something Wrong Happened')
response_id = response.json().get('response_id')

Next step is to use the response ID to fetch the results. We would wait for some time before trying to fetch the results.

time.sleep(30) # Sleeping for 30 seconds
get_results_response = requests.get(
    f"https://api.brightdata.com/dca/get_result?response_id={response_id}",
    headers={"Authorization": f"Bearer {API_TOKEN}"}
)

With data in hand, we can overwrite the static file with this new data.

movie_data = get_results_response.json()
data_file = open(ROTTEN_TOMATO_DATA_FILE_PATH, encoding="utf8", mode="w")
json.dump(movie_data, data_file, indent=4)
data_file.close()

With that the implementation of the update method is completed.

Using the new endpoint

A simple curl request can be made to the endpoint localhost:8000/update to initiate the data updation process. Once the API returns, the data would have been updated.

➜  ~ curl localhost:8000/update        
Movie Data Updated! 

Concluding Remarks

Finally, we have reached the conclusion of this section. In the Flix-Finder, we were able to integrate data updating functionality. As a result, our Flix-Finder will always be available to its users with fresh data!

I hope you liked the three-part series. This Github repository contains the whole collection's source code.

You can follow me on HackerNoon for more such amazing articles. See you in the next article… Till then, Happy Learning! 🙂