From the most popular seats to the most popular viewing times, we wanted to find out more about the movie trends in . So we created — a website to get a glimpse of Singapore’s Movie trends — by scraping data, finding interesting insights, and visualizing them. Singapore PopcornData Ever felt the crippling disappointment of finding out your favorite seat at the theater has been booked? How popular really is your favorite seat? On the website you can see how people watched different movies, at different halls, theaters and timings! Some unique aspects include to show the and to show the . This 2-part article explains how we obtained the data for the website and our analysis of the data. heat maps most popular seats animations order in which seats were bought Scraping the Data To implement our idea, the first and maybe even most crucial step was to collect the data. We decided to scrape the website of , one of the biggest cinema chains in Singapore. Shaw Theatres Starting out with basic knowledge of scraping in python, we initially tried using python’s library to get the site’s HTML and library to parse it but quickly realized that the data we required was not present in the HTML we requested. This was because the website was dynamic — . When we request the HTML directly, the dynamic part of the website is not being rendered, hence the missing data. requests BeautifulSoup it requests the data from an external source using Javascript and renders the HTML dynamically To fix this issue, we used — A web browser automation tool that can first render the website with the dynamic content before getting the HTML. Selenium Issues With Selenium Getting the selenium driver to work and fixing minor issues with it had a big learning curve. After countless StackOverflow searches and ‘giving up’ multiple times we managed to scrape through (pun intended) and get it to work. The main issues we faced were: Scrolling to a specific portion of the screen to click a button so that the data will be found in the HTML Figuring out how to run headless Selenium on the cloud. After deploying the script on Heroku, some of the data was not being scraped when in fact the script was working properly on the local machine. After racking our brains we figured out that some pages loaded by Selenium were defaulting to the mobile version of the page. We fixed it by explicitly mentioning the screen size. With Selenium and BeautifulSoup, we were finally able to get the data for all the available movie sessions for a particular day! Sample movie session data: { : , : , : , : , : , : } "theatre" "Nex" "hall" "nex Hall 5" "movie" "Jumanji: The Next Level" "date" "18 Jan 2020" "time" "1:00 PM+" "session_code" "P00000000000000000200104" Now we needed to collect the seat data for each movie slot to see which seats were occupied and when they were bought. After going through the Network tab of the website in the Developer Tools, we found that the seat data was being requested from Shaw’s API. We were halfway there! The data could be obtained by requesting the URL where the session code was the unique code for each movie session which we had already scraped earlier. https://www.shaw.sg/api/SeatingStatuses?recordcode=<session_code> The data we got was in JSON format and we parsed it and reordered the seats in ascending order of seat buy time to obtain an array of JSON objects where each object contained information about each seat in the movie hall, including seat_number, seat_buy_time, and seat_status. Sample seat data: [
  { : , : , : , : , : },
   ...,
  { : , : , : , : , : }
] "seat_status" "AV" "last_update_time" "2020-01-20 14:34:53.704117" "seat_buy_time" "1900-01-01T00:00:00" "seat_number" "I15" "seat_sold_by" "" "seat_status" "SO" "last_update_time" "2020-01-20 14:34:53.705116" "seat_buy_time" "2020-01-18T13:12:34.193" "seat_number" "F6" "seat_sold_by" "" : Unique identifier for a seat in a hall seat_number : Indicates the availability of a seat ( -seat occupied, -Available) seat_status SO AV : time seat was purchased by the customer seat_buy_time time seat data was last scraped last_update_time: Halls have anywhere between 28 – 502 seats and each seat corresponds to a JSON object in the array. Add this to the fact that there are upwards of 350 movie sessions in a single day, and the amount of data generated is pretty big. Storing data for a single day took about 10MB. The movie session data was combined with the seat data and stored on a MongoDB database. We managed to scrape all the movie data from Shaw for January 2020. A single document in the database { : , : , : , : , : , : :[
   { : , : , : , : , : },
   ...,
   { : , : , : , : , : }
 ]
} "theatre" "Nex" "hall" "nex Hall 5" "movie" "Jumanji: The Next Level" "date" "18 Jan 2020" "time" "1:00 PM+" "session_code" "P00000000000000000200104" "seats" "seat_status" "AV" "last_update_time" "2020-01-20 14:34:53.704117" "seat_buy_time" "1900-01-01T00:00:00" "seat_number" "I15" "seat_sold_by" "" "seat_status" "SO" "last_update_time" "2020-01-20 14:34:53.705116" "seat_buy_time" "2020-01-18T13:12:34.193" "seat_number" "F6" "seat_sold_by" "" To view the full document . click here The complete raw data collected can be downloaded here: Raw Movie Data.zip Time to Get Our Hands Dirty by and pulling out relevant information. Using pandas, we parsed the JSON, cleaned it and made a DataFrame with the data to improve readability and filter it easily. It was now time to get our hands dirty cleaning the data Since the seat data took a lot of memory, we could not include all of it in the DataFrame. Instead, we aggregated the seat data using python to obtain the following: 1. : Total number of seats available for a movie session Total Seats 2. : Number of seats sold for a movie session Sold Seats 3. : 2 dimensional array showing the order in which seats were bought Seat Buy Order [[ , ], [ ], [ , , , ], [ , , ], [ , ], [ , ], [ , ], [ , , ], [ ], [ , , ]] 'A_10' 'A_11' 'A_12' 'B_4' 'B_7' 'B_6' 'B_5' 'C_8' 'C_10' 'C_9' 'B_1' 'B_2' 'C_6' 'C_7' 'C_5' 'C_4' 'B_8' 'B_10' 'B_9' 'D_8' 'A_15' 'A_14' 'A_13' Each element in the array represents the seats bought at the same time and the order of elements represents the order in which the seats were purchased. 4. : Dictionary showing the number of seats that were bought together (in groups of 1, 2, 3 or more) Seat Distribution { : , : , : , : , : } 'Groups of 1' 8 'Groups of 2' 30 'Groups of 3' 9 'Groups of 4' 3 'Groups of 5' 1 5. : Dictionary showing the number of times each seat in a hall was bought over the course of the month Seat Frequency { : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : } 'E_7' 4 'E_6' 5 'E_4' 11 'E_5' 9 'E_2' 2 'E_1' 2 'E_3' 7 'D_7' 15 'D_6' 17 'C_1' 33 'D_2' 15 'D_1' 14 'B_H2' 0 'B_H1' 0 'D_4' 45 'D_5' 36 'D_3' 32 'C_3' 95 'C_4' 94 'A_2' 70 'A_1' 70 'B_2' 50 'B_1' 47 'C_2' 37 'C_6' 53 'C_5' 61 'B_4' 35 'B_3' 40 6. : Two dictionaries, with the first dictionary showing the time left to a movie showing (in days) with the corresponding accumulated number of tickets bought in the second dictionary Rate of Buying {“ ”: [ ..., , , , , ....],
...}
{“ ”: [ , , , , , ...],
...} 1917 4.1084606481481485 2.566423611111111 2.245578703703704 2.0319560185185184 1.9269907407407407 1.8979513888888888 1917 1 3 8 10 11 The cleaned data can be viewed here: Cleaned Movie Data Finally we were done! (with 20% of the work) With our data scraped and cleaned, we could now get to the fun part — to find patterns amongst the popcorn litter. analyzing the data To learn more about our analysis of the data and interesting patterns we found, check out of this article. In Part II, we analyze the data, visualize it, and then build a website for our findings. Part II The code for the scraper can be found in this . github repo Don’t forget to check out our website at http://popcorn-data.herokuapp.com ! This article was written by and Noel Mathew Isaac Vanshiqa Agrawal Previously published at https://medium.com/@noelmathewisaac/popcorn-data-analysing-cinema-seating-patterns-part-1-a0b2a5c2c19a

Heroku

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Use Data Science to Find the Best Seat in the Cinema (Part I)

Too Long; Didn't Read

People Mentioned

Company Mentioned

Noel Mathew Isaac

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Use Data Science to Find the Best Seat in the Cinema (Part I)

Too Long; Didn't Read

People Mentioned

Company Mentioned

Noel Mathew Isaac

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES