I Made a Python Bot That Can Solve Multiple-Choice Question From Any Given Image [incl. Code]

In this post I am going to show you how to build your own answer finding system with Python. Basically, this automation can find the answer of multiple-choice question from the picture. One thing is clear, It is impossible to search questions on the internet during the exam but I can take a picture quickly when the examiner turned his back. That is the first part of the algorithm. Somehow I need to extract the question from the picture. It seems there are a lot of services that can provide text extraction tools but I need some kind of API to solve this problem. Finally, Google's Vision API was the exact tool that I am looking for. The great thing is, the first 1000 API calls is free for each month, which is quite enough for me to test and use the API. Vision AI First, go and create Google Cloud Account then search for in services. Using the Vision AI, you can perform things such as assign labels to an image to organize your images, get the recommended crop vertices, detect famous landscapes or places, extract texts, and few other things. Vision AI Check the to enable and set up the API. After configuration, you have to create JSON file that contains your key downloads to your computer. documentation Run following command to install the client library: pip install google-cloud-vision Then provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. os, io google.cloud vision google.cloud.vision types os.environ[ ] = client = vision.ImageAnnotatorClient() FILE_NAME = io.open(os.path.join(FILE_NAME), ) image_file: content = image_file.read() image = vision.types.Image(content=content) response = client.text_detection(image=image) print(response) texts = response.text_annotations[ ] print(texts.description) import from import from import # JSON file that contains your key 'GOOGLE_APPLICATION_CREDENTIALS' 'your_private_key.json' # Instantiates a client 'your_image_file.jpg' # Loads the image into memory with 'rb' as # Performs text detection on the image file # Extract description 0 When you run the code you will see the response in JSON format that includes specifications of the detected texts. But we only need pure description so I extracted this part from the response. Search Question on Google The next step is to search the question part on Google to get some information. I used regex library to extract question part from the description (response). Then we must slugify the extracted question part to be able to search it. re urllib texts.description: question = re.search( , texts.description).group( ) texts.description: question = re.search( , texts.description).group( ) texts.description: question = re.search( , texts.description).group( ) slugify_keyword = urllib.parse.quote_plus(question) print(slugify_keyword) import import # If ending with question mark if '?' in '([^?]+)' 1 # If ending with colon elif ':' in '([^:]+)' 1 # If ending with newline elif '\n' in '([^\n]+)' 1 # Slugify the match Crawl the Information We are going to use BeautifulSoup to crawl the first 3 results to get some information about the question because the answer probably locates in one of them. Additionally, if you want to crawl particular data from the google's search list don't use inspect element to find attributes of the elements, instead print the whole page to see attributes because it is different from the actual one. We need to crawl first 3 links in search result but these links are really messed up so it is important to get the clean links for crawling. /url?q=https: //en.wikipedia.org/wiki/IAU_definition_of_planet&sa=U&ved=2ahUKEwiSmtrEsaTnAhXtwsQBHduCCO4QFjAAegQIBBAB&usg=AOvVaw0HzMKrBxdHZj5u1Yq1t0en As you see the actual link locates between q= and &sa. By using Regex we can get this particular field or the valid URL. result_urls = [] req = Request( + slugify_keyword, headers={ : }) html = urlopen(req).read() bs = BeautifulSoup(html, ) results = bs.find_all( , class_= ) : result results: link = result.find( )[ ] link: result_urls.append(re.search( , link).group( )) (AttributeError, IndexError) e: : def crawl_result_urls () 'https://google.com/search?q=' 'User-Agent' 'Mozilla/5.0' 'html.parser' 'div' 'ZINbbc' try for in 'a' 'href' # Checking if it is url (in case) if 'url' in 'q=(.*)&sa' 1 except as pass Before we crawl the content of these URLs let me show you the Question Answering System with Python. Question Answering System That is the main part of the algorithm. After crawling the information from first 3 results, program should detect the answer by iterating documents. First I thought it is better to use similarity algorithm to detect the documents which is the most similar to question but I had no any idea how to implement it. After hours of researching I found an article in Medium which explains Question Answering system with Python. Its easy-to-use python package to implement a QA System on your own private data. You can go and check for more explanation from here Let's first install the package: pip install cdqa I am downloading pre-trained models and data manually by using download functions that included in the example code block below: pandas pd ast literal_eval cdqa.utils.filters filter_paragraphs cdqa.utils.download download_model, download_bnpp_data cdqa.pipeline.cdqa_sklearn QAPipeline download_bnpp_data(dir= ) download_model(model= , dir= ) df = pd.read_csv( , converters={ : literal_eval}) df = filter_paragraphs(df) cdqa_pipeline = QAPipeline(reader= ) cdqa_pipeline.fit_retriever(df) query = prediction = cdqa_pipeline.predict(query) print( .format(query)) print( .format(prediction[ ])) print( .format(prediction[ ])) print( .format(prediction[ ])) import as from import from import from import from import # Download data and models './data/bnpp_newsroom_v1.1/' 'bert-squad_1.1' './models' # Loading data and filtering / preprocessing the documents 'data/bnpp_newsroom_v1.1/bnpp_newsroom-v1.1.csv' 'paragraphs' # Loading QAPipeline with CPU version of BERT Reader pretrained on SQuAD 1.1 'models/bert_qa.joblib' # Fitting the retriever to the list of documents in the dataframe # Sending a question to the pipeline and getting prediction 'Since when does the Excellence Program of BNP Paribas exist?' 'query: {}\n' 'answer: {}\n' 0 'title: {}\n' 1 'paragraph: {}\n' 2 he output should like this: It prints the exact answer and paragraph that includes the answer. Basically, When question extracted from picture and sent to the system, the Retriever will select a list of documents from the crawled data that are the most likely to contain the answer. As I stated before, it computes the cosine similarity between the question and each document in the crawled data. After selecting the most probable documents, the system divides each document into paragraphs and send them with the question to the Reader, which is basically a pre-trained Deep Learning model. The model used was the Pytorch version of the well known NLP model BERT. Then, the Reader outputs the most probable answer it can find in each paragraph. After the Reader, there is a final layer in the system that compares the answers by using an internal score function and outputs the most likely one according to the scores which will the answer of our question. Here is the schema of the system mechanism. You have to set your dataframe (CSV) in specific structure so it can be sent to cdQA pipeline. But actually I used PDF converter to create an input dataframe from a directory of PDF files. So, I am going to save all crawled data in pdf file for each result. Hopefully, we will have 3 pdf files in total (can be 1 or 2 as well). Additonaly, we need to name these pdf files that's why I crawled the heading of each page. : req = Request(url, headers={ : }) html = urlopen(req).read() bs = BeautifulSoup(html, ) : title = bs.find(re.compile( )).get_text().strip().replace( , ).lower() filename = + title + os.path.exists(os.path.dirname(filename)): : os.makedirs(os.path.dirname(filename)) OSError exc: exc.errno != errno.EEXIST: open(filename, ) f: line bs.find_all( )[: ]: f.write(line.text + ) AttributeError: urllib.error.HTTPError: df = pdf_converter(directory_path= ) cdqa_pipeline = QAPipeline(reader= ) cdqa_pipeline.fit_retriever(df) query = question + prediction = cdqa_pipeline.predict(query) print( .format(query)) print( .format(prediction[ ])) print( .format(prediction[ ])) print( .format(prediction[ ])) prediction[ ] : def get_result_details (url) try 'User-Agent' 'Mozilla/5.0' 'html.parser' try # Crawl any heading in result to name pdf file '^h[1-6]$' '?' '' # Naming the pdf file "/home/coderasha/autoans/pdfs/" ".pdf" if not try except as # Guard against race condition if raise with 'w' as # Crawl first 5 paragraphs for in 'p' 5 '\n' except pass except pass : def find_answer () '/home/coderasha/autoans/pdfs' 'models/bert_qa.joblib' '?' 'query: {}\n' 'answer: {}\n' 0 'title: {}\n' 1 'paragraph: {}\n' 2 return 0 Well, If I summarize the algorithm it will extract the question form the picture, search it on google, crawl first 3 results, create 3 pdf files from the crawled data and finally find the answer using question answering system. If you want to see how it is working check I made a bot that can solve exam questions from the picture Here is the Full Code: os, io errno urllib urllib.request hashlib re requests time sleep google.cloud vision google.cloud.vision types urllib.request urlopen, Request bs4 BeautifulSoup pandas pd ast literal_eval cdqa.utils.filters filter_paragraphs cdqa.utils.download download_model, download_bnpp_data cdqa.pipeline.cdqa_sklearn QAPipeline cdqa.utils.converters pdf_converter result_urls = [] os.environ[ ] = client = vision.ImageAnnotatorClient() FILE_NAME = io.open(os.path.join(FILE_NAME), ) image_file: content = image_file.read() image = vision.types.Image(content=content) response = client.text_detection(image=image) texts = response.text_annotations[ ] texts.description: question = re.search( , texts.description).group( ) texts.description: question = re.search( , texts.description).group( ) texts.description: question = re.search( , texts.description).group( ) slugify_keyword = urllib.parse.quote_plus(question) req = Request( + slugify_keyword, headers={ : }) html = urlopen(req).read() bs = BeautifulSoup(html, ) results = bs.find_all( , class_= ) : result results: link = result.find( )[ ] print(link) link: result_urls.append(re.search( , link).group( )) (AttributeError, IndexError) e: : req = Request(url, headers={ : }) html = urlopen(req).read() bs = BeautifulSoup(html, ) : title = bs.find(re.compile( )).get_text().strip().replace( , ).lower() filename = + title + os.path.exists(os.path.dirname(filename)): : os.makedirs(os.path.dirname(filename)) OSError exc: exc.errno != errno.EEXIST: open(filename, ) f: line bs.find_all( )[: ]: f.write(line.text + ) AttributeError: urllib.error.HTTPError: df = pdf_converter(directory_path= ) cdqa_pipeline = QAPipeline(reader= ) cdqa_pipeline.fit_retriever(df) query = question + prediction = cdqa_pipeline.predict(query) prediction[ ] crawl_result_urls() url result_urls[: ]: get_result_details(url) sleep( ) answer = find_answer() print( + answer) import import import import import import import from import from import from import from import from import import as from import from import from import from import from import 'GOOGLE_APPLICATION_CREDENTIALS' 'your_private_key.json' 'your_image_file.jpg' with 'rb' as 0 # print(texts.description) if '?' in '([^?]+)' 1 elif ':' in '([^:]+)' 1 elif '\n' in '([^\n]+)' 1 # print(slugify_keyword) : def crawl_result_urls () 'https://google.com/search?q=' 'User-Agent' 'Mozilla/5.0' 'html.parser' 'div' 'ZINbbc' try for in 'a' 'href' if 'url' in 'q=(.*)&sa' 1 except as pass : def get_result_details (url) try 'User-Agent' 'Mozilla/5.0' 'html.parser' try '^h[1-6]$' '?' '' # Set your path to pdf directory "/path/to/pdf_folder/" ".pdf" if not try except as if raise with 'w' as for in 'p' 5 '\n' except pass except pass : def find_answer () # Set your path to pdf directory '/path/to/pdf_folder/' 'models/bert_qa.joblib' '?' # print('query: {}\n'.format(query)) # print('answer: {}\n'.format(prediction[0])) # print('title: {}\n'.format(prediction[1])) # print('paragraph: {}\n'.format(prediction[2])) return 0 for in 3 5 'Answer: ' It can confuse sometimes but generally it is okay I think. At least I can pass the exam with 60% of right answers :D Alright Devs! Please tell me in comments what do you think about it? Actually it is better to iterate through questions at once so I don't need to take picture for each question. But unfortunately I don't have enough time to do it so it is better to keep that for the next time. Check for more cool content. Reverse Python My YouTube Channel References: How to create your own Question-Answering system easily with python Stay Connected!