Arslan

[With Just 1 Call] - Use Lazy Collections to Export 100,000 records to a CSV file in Laravel 6.0

What is Lazy Collection in Laravel 6.0 and why it’s introduced?
Lazy Collections concept is introduced by Joseph Silber, it is a class that provides an iterator for the items of array, if you familiar in Laravel 5 we have a Collection class which is used to wrap array items. Now,in Laravel 6.0 we have additional Lazy Collection class here along with the Normal Collection Class.
Lazy collection class is precisely designed for keep memory
usage low by the Laravel applications it uses the power of PHP Generators to allow you to work with very large datasets while keeping memory usage low.
Let’s suppose that our application database table has 100,000 rows and we want to fetch all of the them for a specific requirement
may be exporting to csv or any file.
If we try using Laravel Eloquent all() method mostly the application throws run out of memory exception and this is because when we perform all query it performs 2 operations, first is fetching all records from database and second it loads in memory in form of iterator so it's crystal clear that if we try 100,000 of records then the app will heat up.
To get rid from this problem and allow application to run
smoother by utilizing low memory, Laravel 6.0 has proposed the solution with Lazy Collections.
I've used this feature today to see how we can minimize memory usage by using Lazy Collections. It really worked very efficiently.
Environment setup:
For demonstrating the functionality of Lazy Collections, I
made a new project of Laravel 6.0 using Lazy Collections. You can follow the instructions step by step.
1. If you have not installed "composer" on your
system then go to the "composer's" website

https://getcomposer.org
and install composer.
2. If you have not installed "laravel-installer" on your system then install using following command.  
composer global require laravel/installer
3. Open terminal or command prompt in that directory where you
want to make a new project. Run this command.
laravel new test-lazy-collection
4. Then we switch the directory using this command
cd test-lazy-collection
5. Make a new database in mysql and place the values DB_DATABASE,
DB_USERNAME, DB_PASSWORD in .env file of project. Fill up the data according to the system's MySQL settings
6. Then I made a seeder class using this command.
php artisan make:seeder UsersTableSeeder

The above command generates boiler-plate code for a database
seeder.
 7. Now Open the seeder file that has been created and place the following code in the function run()
public function run()  
	    {  
	        DB::table('users')->truncate();  
	        for($i=0; $i<=100; $i++){  
	            factory(App\User::class, 1000)->create();  
	        }  
	    }  

The run method is executed when you run the database seed
command. First, it will truncate the users table and place 1000 record in each iteration in the database producing 100,000 records in total.

8. Now, you can update the factory in the database/factories directory as
	$factory->define(App\User::class, function (Faker $faker) {  
	    return [  
	        'name' => $faker->name,  
	        'email' => $faker->unique()->safeEmail,  
	        'email_verified_at' => now(),  
	        'password' => bcrypt('123456'), // password  
	        'remember_token' => Str::random(10),  
	    ];  
	});

Faker is a library that generates fake data for a variety of
data types. As above shown $faker will generate name,
email,email_verified_at,password,remember_token

dummy data in the database.
9. Uncomment the following line in database/seeds/DatabaseSeeder.php

	public function run()  
	    {  
	       // $this->call(UsersTableSeeder::class);  
	    } 

This command will add 100,000 record in database.
10. Run the following command in the opened terminal or command
prompt.
php artisan db:seed


The above command seeds the seeder called UsersTableSeeder.
11. Write the following command in the opened terminal/command
prompt to make controller.
php artisan make:controller UsersController
12. Make routes in routes/web.php
Route::get('/write', 'UsersController@write');  
Route::get('/lazyWrite', 'UsersController@lazyWrite');  

write is a normal function and lazyWrite is a function implemented with lazy collection.
13. Add a method
checkDirOrMake()
in UsersController.
	private function checkDirOrMake($dirname){  
	     $filename = $dirname . "/";  
	  
	     if (!file_exists($filename)) {  
	         mkdir($dirname, 0777);  
	     }  
	 }  

checkDirOrMake() function will check the directory is
present or not. If it is not present it will create a new one.
14. Make a function named write() in UsersController and place
below code in it


	public function write(){  
	  
	        $directory = storage_path("uploads");  
	  
	        $this->checkDirOrMake($directory);  
	  
	        $file = fopen($directory."/contacts.csv","w+");  
	  
	        $users = \App\User::all();  
	  
	        foreach($users as $user) {  
	            $user = $user->toArray();  
	            fputcsv($file, $user);  
	        }  
	  
	        fclose($file);  
	    }  

In above function what we used to do before, fetches data
from users table then iterate though foreach, by converting into array write data in csv file.
15. Now make another function named lazyWrite() in UsersController and place below code in it.
public function lazyWrite(){  
  
        $directory = storage_path("uploads");  
  
        $this->checkDirOrMake($directory);  
  
        $file = fopen($directory."/lazy_contacts.csv","w+");  
  
        $users = \App\User::cursor()  
        ->each(function ($user) use ($file) {  
            $user = $user->toArray();  
            fputcsv($file, $user);  
        });  
  
        fclose($file);  
    }  

Now In the above function, Notice instead of using all() method I have used cursor() method, by only changing one function call Laravel will be switching from Laravel collection to lazy collection which will be able to deal with very larger datasets, that will open a file called lazy_contacts.csv write each line and then yield it for the processing, so this is the function which could use to load very large file and it would not let your application down.
16. Run the following command to start the built-in server
php artisan serve
17. Now load the following URL into your browser one by one.
http://localhost:8000/write
  and
http://localhost:8000/lazyWrite
18. You’ll see that page where you ran the first URL will stuck due to high memory consumption while
http://localhost:8000/lazyWrite
will do the job and write the file with 100,000 records as it was intended. Result is shown below.
Lazy collection class surely will help you to increase your Laravel 6.0 speed and performance.

Tags

Topics of interest