4,830 reads

How to Remove all Duplicate Files on your Drive via Python

by Kalebu Jordan October 13th, 2020

Too Long; Didn't Read

How to Remove all Duplicate Files on your Drive via Python using Python programming. Python programming can ultimately save a lot of space on your drive by removing all the duplicates. We are going to use Python's OS deleting tool to remove all the duplicate files on our drive. We will use the Python standard library to use hashlib to hash a string in Python using md5 and hexdigest to generate a binary hash of a given file. The value of the generated hash doesn't change as long as it's the same file.

Company Mentioned

featured image - How to Remove all Duplicate Files on your Drive via Python

Today you're going to learn how to use Python programming in a way that can ultimately save a lot of space on your drive by removing all the duplicates.

Intro

In many situations you may find yourself having duplicates files on your disk and but when it comes to tracking and checking them manually it can tedious.

Heres a solution

Instead of tracking throughout your disk to see if there is a duplicate, you can automate the process using coding, by writing a program to recursively track through the disk and remove all the found duplicates and that's what this article is about.

But How do we do it?

If we were to read the whole file and then compare it to the rest of the files recursively through the given directory it will take a very long time, then how do we do it?

The answer is hashing, with hashing can generate a given string of letters and numbers which act as the identity of a given file and if we find any other file with the same identity we gonna delete it.

There's a variety of hashing algorithms out there such as

md5
sha1
sha224, sha256, sha384 and sha512

Lets do some coding

Hashing in Python is pretty straight forward we are going to use hashlib library which comes by default with Python standard library

Below is an example of how we hash stuff using hashlib, we are going to hash of a string in Python using md5 hashing algorithms

Example of Usage

>>> import hashlib
>>> example_text = "Duplython is amazing".encode('utf-8')
>>> hashlib.md5(example_text).hexdigest()
'73a14f46eadcc04f4e04bec8eb66f2ab'

It’s straight forward, you just need to import hashlib and then use md5 method to create hash and finally use hexdigest to generate string of the hash.

The above example has shown us how to hash a string but as we look in relation to the project we are about to build we are more concerned with files rather than strings, another question arises;

How do we hash files?

Hashing files is similar to hashing string with just minor differences, during the hashing file we first need to open the file in binary and then generate a hash of the file binary value.

Hashing File

Let's say you have simple text document on your project directory with name learn.txt, This is how we will do it.

>>> import hashlib
>>> file = open('learn.txt', 'rb').read()
>>> hashlib.md5(file).hexdigest()
'0534cf6d5816c4f1ace48fff75f616c9'

As you can see above, even If you try to generate the hashes for a second time, The value of the generated hash doesn't change as long as it's the same file.

The challenge arises when we try to read a very large file, It gonna take a while loading it therefore instead of waiting for the whole file to memory we can keep computing the hashes as we read the file.

Computing hashes while reading the file requires us to read the file in blocks of a given size and keep updating the hashes as we keep reading the file until the complete hashing of the whole file.

Doing it this way could save us a lot of waiting time that we could use on waiting for the whole file to be ready.

Example of Usage

>>> import hashlib
>>> block_size = 1024
>>> hash = hashlib.md5()
>>> with open('learn.txt', 'rb') as file:
...     block = file.read(block_size)
...     while len(block)>0:
...             hash.update(block)
...             block = file.read(block_size)
...     print(hash)
... 
0534cf6d5816c4f1ace48fff75f616c9

As you can see hash has not changed, it still the same. Therefore, we are ready to go to building our python tool to do the job.

But wait for hashing is just one step we need a way to actually removes the duplicates, we gonna use built python module OS in deleting duplicates.

We gonna use Python OS remove( ) method to remove the duplicates on our drive.

Let’s try deleting learn.txt with os module

Example of Usage (os module):

>>> import os
>>> os.listdir()
['Desktop-File-cleaner', '.git', 'learn.txt', 'app.py', 'README.md']
>>> os.remove('learn.txt')
>>> os.listdir()
['Desktop-File-cleaner', '.git', 'app.py', 'README.md']

Well, that’s simple you just call remove ( ) with a parameter of the name of the file you wanna remove done. now let’s go build our application.

Building our cleaning Tool

importing necessary libraries

import time
import os
from hashlib import sha256

I'm a huge fan of Object-oriented programming and on this article, we gonna build our tool as a single class, below is just as exoskeleton class for our code.


import time
import os
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
    def main(self)->None:
      self.welcome()

if __name__ == '__main__':
    App = Duplython()
    App.main()

That’s just initial cover for our Python Program of which when we run it will just print the welcoming method to the screen.

$ python3 app.py
******************************************************************
****************        DUPLYTHON      ****************************
********************************************************************

----------------        WELCOME        ----------------------------

Cleaning .................

We now have to create a simple function to generate hash of a file with a given path using the hashing knowledge we have learned above.

import time
import os
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
   def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False
        
    def main(self)->None:
      self.welcome()

if __name__ == '__main__':
    App = Duplython()
    App.main()

Now Let's implement our program Logic

Now after we made a function to generate hash per a given path of the file, let's implement where by we will be comparing those hashes and removing any found duplicate.

I have made a simple function called clean( ) just to that as shown below.

import time
import os
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
   def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False

    def clean(self)->None:
        all_dirs = [path[0] for path in os.walk('.')]
        for path in all_dirs:
            os.chdir(path)
            All_Files =[file for file in os.listdir() if os.path.isfile(file)]
            for file in All_Files:
                filehash = self.generate_hash(file)
                if not filehash in self.File_hashes:
                    if filehash:                       
                        self.File_hashes.append(filehash)
                        #print(file)
                else:
                    byte_saved = os.path.getsize(file); self.count_cleaned+=1
                    self.Total_bytes_saved+=byte_saved
                    os.remove(file); filename = file.split('/')[-1]
                    print(filename, '.. cleaned ')
            os.chdir(self.home_dir)
        
    def main(self)->None:
      self.welcome();self.clean()

if __name__ == '__main__':
    App = Duplython()
    App.main()

Now that our program is nearly complete, we have to add a simple method to print the summary of the cleaning process.

I have implemented the method cleaning_summary () just to do that. Print out the summary of the cleaning process to the screen which completes our python tool as shown below:

import time
import os
import shutil
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')

    def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False

    def clean(self)->None:
        all_dirs = [path[0] for path in os.walk('.')]
        for path in all_dirs:
            os.chdir(path)
            All_Files =[file for file in os.listdir() if os.path.isfile(file)]
            for file in All_Files:
                filehash = self.generate_hash(file)
                if not filehash in self.File_hashes:
                    if filehash:                       
                        self.File_hashes.append(filehash)
                        #print(file)
                else:
                    byte_saved = os.path.getsize(file); self.count_cleaned+=1
                    self.Total_bytes_saved+=byte_saved
                    os.remove(file); filename = file.split('/')[-1]
                    print(filename, '.. cleaned ')
            os.chdir(self.home_dir)
    
    def cleaning_summary(self)->None:
        mb_saved = self.Total_bytes_saved/1048576
        mb_saved = round(mb_saved, 2)
        print('\n\n--------------FINISHED CLEANING ------------')
        print('File cleaned  : ', self.count_cleaned)
        print('Total Space saved : ', mb_saved, 'MB')
        print('-----------------------------------------------')
        
    def main(self)->None:
        self.welcome();self.clean();self.cleaning_summary()

if __name__ == '__main__':
    App = Duplython()
    App.main()

Our app is complete, now to run the application go to the specific folder you want to clean and it will iterate recursively over a given folder to find all the files and remove the duplicate one.

Example output:

$ python3 app.py 
******************************************************************
****************        DUPLYTHON      ****************************
********************************************************************


----------------        WELCOME        ----------------------------

Cleaning .................
0(copy).jpeg .. cleaned 
0 (1)(copy).jpeg .. cleaned 
0 (2)(copy).jpeg .. cleaned 


--------------FINISHED CLEANING ------------
File cleaned  :  3
Total Space saved :  0.38 MB
-----------------------------------------------

I hope you find this post interesting. Now it's time to share it with your fellow friends on Twitter and other dev communities.

This article is also published on kalebujordan.com.