Imagine having machines that can understand scientific papers and summarize its content. I mean… I’m talking about very sophisticated documents written by some of the smartest people that existed. Assisted by smart machines created in our image, I ask myself what problems can we solve and what diseases can we cure? : The dream of the thinking machines. Figure-1 There’s a huge potential in the domain of vital information extraction and summarization of scientific papers that I believe is under-researched. In this article, I’ll show you how a technique created for biological image segmentation UNet can be combined with Optical Character Recognition to extract relevant parts of a scientific paper. Due to time constraints and project scope, I’ll be covering textual extraction but not summarization. Objective In this project, I aim at extracting certain sections from scientific papers. The intended sections are ones that are commonly present in scientific papers such as Title, Abstract, and Author/s. I will be using UNet to learn where the sections can be found and then pass learned information into an OCR. The complete project can be found . here : Input vs final result. My input is the 1st-page paper converted into an image. Final result is a txt file containing the Title, Author/s, and Abstract section as shown above. Figure-2 How did I do it? I started with data gathering and collecting scientific papers. Since I am only interested in 3 sections namely Title, Abstract, and Authors, I took the first page of the 8+ pages of the collected papers. I then converted the into images since my UNet model only accepts images. I’ve divided my dataset into a training set and a test set by 80/20 respectively. Next, I wrote a python script to mask the images. see figures below. pdf mask.py : on the right is the original first page of a scientific paper, on the left is the result of mask.py for the corresponding paper. Figure-3 The masked image is 1D image where each pixel has values 0 (black) and 1 (white). 0 means that the pixel is not important and 1 means the pixel is important. The white region of the image on the left masks the Title, Abstract and Author/s sections as being important. We pass this mask and the original image to our UNet model for training. UNet is a popular architecture. You can find many implementations of the architecture. I’ve used the one by . if you’ve already cloned my repository you don’t need to clone this repo . You can build the model by changing your working directory to and then running as follows: milesial dagshub Unet-OCR/Pytorch-UNet train.py python train.py --epoch 6 --batch-size 1 --learning-rate 0.000001 You can obviously set your own parameters. Otherwise, you can use my pre-trained model which you’ll get upon running: MODEL.pth dvc pull -r origin Make sure you set your DVC origin to my repository as per my Installation instruction below. My best model was able to get a training loss of 0.395 after 6 epochs and 1365 steps. dagshub Finally, I used my for model prediction. My model generated a mask of important sections of an RGB image. Using a post-processing script , I reconstructed RGB images out of their corresponding masks. Those images were then passed to Tesseract OCR to convert the images into text. testset postprocessing.py Getting started Installation Reproducing my environment and getting myself up and running is quite easy. Start by cloning my repository git clone https://dagshub.com/Eman22S/Unet-OCR-2.0.git 2. Install DVC to track your dataset pip install dvc 3. Install dagshub for data, model, and experiment logging/tracking. pip install dagshub 4. Configure your remote DVC as follows. Dagshub will auto-generate the content in braces for you. dvc remote add origin https://dagshub.com/Eman22S/Unet-OCR-2.0.dvc
dvc remote modify origin — local auth basic
dvc remote modify origin — local user {your-username}
dvc remote modify origin — local password {your-password} If you feel confused about this step don’t worry!! Dagshub provides excellent that even a monkey can easily follow. documentation 5. If you want to pull my Model and dataset run this code. Otherwise, skip. dvc pull -r origin 6. Install tesseract on your machine sudo apt install tesseract-ocr-all -y Running Start by training your model using python train.py --amp Make sure you changed the directory to . Also make sure you’re images and masks are in and respectively. Or run to use my training dataset. installation-path/Unet-OCR-2.0/UNet-OCR/Pytorch-UNet installation-path/Unet-OCR-2.0/UNet-OCR/Pytorch-UNet/data/imgs installation-path/Unet-OCR-2.0/UNet-OCR/Pytorch-UNet/data/masks dvc pull -r origin 2. Run prediction using the following code: python predict.py -i {path to testset} -o {path to save generated file} 3. Convert predicted mask into Image: python postprocess.py -i {path to masked image} -e {path to original image} -o {path to save output} 4. Image to text conversion using tesseract: tesseract {path to the postprocessed image} {path with name of file  } -l  eng What did I use? There are many open source software out there nowadays for machine learning projects. Here’s a list of my favorite ones which I’ve used in this project. Google colab notebook MLflow DVC Dagshub Python ImageIo Pythorch matplotlib Numpy Pillow torch torchvision tqdm If your computer doesn’t have the right resources for training, don’t worry Google got your back! Google colab Notebook provides limited access to Tesla GPU for free. you can obviously upgrade your account to premium but the free version is not too bad. I’ve used the free version for this project. I’ve used daghub’s git tracker to track my model. I also used DVC to track my dataset. For most machine learning projects, no language comes close to Python with its powerful data science tool kit. Pytorch is a Python framework that allows OOP style of Model building. Project Pipeline It’s a bit of a complicated project since I’m using two models so I thought a diagram could help. The diagram below shows an overview of the project pipeline. Results With git’s model tracking capabilities embedded in Dagshub, you can easily view and track the model parameters that gave you the best result. Figure-5: Experiments table My table shows I got better performance with the amp parameter turned false and epoch set to 5. check this to view all my experiments. link I’ve added the prediction done on two models from different branches in the following gif image. That way it’s easier to visualize and compare two different models visually. This is made incredibly easy by the cool comparison feature Dagshub provides. You can compare datasets, code, and pretty much anything. Pretty cool huh? : Comparing results of predictions from different models in different branches. Figure-6 The gif is revealing how the model’s prediction is not good so far. But looking at the experiment table in figure 6. We know this can be fixed with training for a few more epochs. Conclusion The biggest challenge of this project is the frequent data processing that is required. As you can see from the previous section, I have transformed my dataset several times, excluding the initial preprocessing. I’ve used Dagshub to assist me in tracking my dataset changes. Dagshub is an online repository that supports several version control tools namely: git, dvc and mlflow. I’ve mostly taken advantage of the DVC tool. I can’t tell you how many times it came to save my day when I’ve mistakenly overridden images in , , folders by another. testset unetpred postprocessed This is not to undermine the powerful parameters and metrics logger accessible via the Experiment tab in Dagshub. I’ve used it to learn which model parameters gave me the best validation dice score. All the tools I used for this project are free and open source. Now that we have our final result in format under , the next milestone is to build an NLP model for the summarization step. If you’re interested in collaborating on this area or have experience working on text summarization I would be happy to work together. Stay tuned for my next blog! txt ocrresult References https://labs.armut.com/information-extraction-from-the-turkish-identity-card-44a1b4504cf?gi=8a5f76a04f02 https://arxiv.org/abs/1505.04597 https://github.com/tesseract-ocr/tesseract https://dagshub.com/docs/

Extraction of Relevant Text From Scientific Papers Using Machine Learning

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps