What's up Hackers! In this tutorial we are going to build price tracker application which will notify us for discounts. Creating complex projects is the key to learn fast! If you are following me for a while, you already know I like complex stuff, so we are going to use RabbitMQ + Celery + BeautifulSoup + Django for creating this app. Alright! Let's Start! Celery is the best choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. So, we are using Celery to handle the time-consuming tasks by passing them to queue to be executed in the background and always keep the server ready to respond to new requests. Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery. We can install RabbitMQ through Ubuntu’s repositories by following command: sudo apt- install rabbitmq- get server Then enable and start the RabbitMQ service: sudo systemctl enable rabbitmq-server sudo systemctl start rabbitmq-server Well, create a new Django project named pricetracker and app named tracker Install RabbitMQ on Mac Install following dependencies: pip3 install beautifulsoup4 httplib2 Celery Once installation completed, add the CELERY_BROKER_URL configuration to the settings.py file: CELERY_BROKER_URL = 'amqp://localhost' Then, create celery.py inside your project. celery.py os celery Celery os.environ.setdefault( , ) app = Celery( ) app.config_from_object( , namespace= ) app.autodiscover_tasks() import from import 'DJANGO_SETTINGS_MODULE' 'pricetracker.settings' 'pricetracker' 'django.conf:settings' 'CELERY' We are setting the default Django settings module for the 'celery' program and loading task modules from all registered Django app configs. Now inside your import the celery: __init__.py .celery app celery_app __all__ = [ ] from import as 'celery_app' This will make sure our Celery app loaded every time Django starts. Now, Let's create our model models.py django.db models title = models.CharField(max_length= ) url = models.CharField(max_length= ) requested_price = models.IntegerField(default= ) last_price = models.IntegerField(null= , blank= ) discount_price = models.CharField(max_length= , null= , blank= ) date = models.DateTimeField(auto_now_add= ) self.title from import : class Item (models.Model) 200 600 0 True True 100 True True True : def __str__ (self) return We are going to crawl . User will enter URL of specific item and requested price. eBay So, let's create a form for that: forms.py django forms = forms.CharField(max_length= ) requested_price = forms.IntegerField() from import ( . ): class AddNewItemForm forms Form url 600 We will use BeautifulSoup to crawl price and title of item in given URL. After data crawled we have to convert price to float and create new object in database. views.py urllib.request urlopen, Request bs4 BeautifulSoup def crawl_data(url): # User Agent is to prevent Forbidden req = Request(url, headers={ : }) html = urlopen(req).read() bs = BeautifulSoup(html, ) title = bs.find( , id= ).get_text().replace( , ) price = bs.find( , id= ).get_text() clean_price = float(price.strip().replace( , ).replace( , )) { : title, :clean_price } from import from import 403 Error 'User-Agent' 'Mozilla/5.0' 'html.parser' 'h1' "itemTitle" "Details about" "" 'span' "prcIsum" "US" "" "$" "" return 'title' 'last_price' removes spaces at the beginning and at the end of the string and method replaces a specified phrase with another specified phrase. So, we used these methods to get clean price and title of item. strip() replace() Once data crawled successfully, it is time to create new object in database. We will use this function in form submission to crawl title and price of new item. views.py django.shortcuts render, get_object_or_404,HttpResponseRedirect .models Item .forms AddNewItemForm def tracker_view(request): items = Item.objects.order_by( ) form = AddNewItemForm(request.POST) request.method == : form.is_valid(): url = form.cleaned_data.get( ) requested_price = form.cleaned_data.get( ) # crawling the data crawled_data = crawl_data(url) Item.objects.create( url = url, title = crawled_data[ ], requested_price=requested_price, last_price=crawled_data[ ], discount_price= , ) HttpResponseRedirect( ) : form = AddNewItemForm() context = { :items, :form, } render(request, , context) from import from import from import '-id' if 'POST' if 'url' 'requested_price' 'title' 'last_price' 'No Discount Yet' return '' else 'items' 'form' return 'tracker.html' Great! Now, we need to crawl the data for all objects continuously to be aware of discounts. If we do this without celery the server connection will timeout which means that a server is taking too long to reply to a data request and our application will crash. Create in your app and let's handle it with celery tasks. tasks.py time celery shared_task .models Vehicle tracker.views crawl_data @shared_task # something heavy def track_for_discount(): items = Item.objects.all() item items: # crawl item url data = crawl_data(item.url) # check discount data[ ] < item.requested_price: print(f ) # update discount field to notify user item_discount = Item.objects.get(id=item.id) item_discount.discount_price = f item_discount.save() True: track_for_discount() time.sleep( ) import from import from import from import do for in for if 'last_price' 'Discount for {data["title"]}' 'DISCOUNT! The price is {data["last_price"]}' while 15 will create the independent instance of the task for each app, making task reusable. This makes the decorator useful for libraries and reusable apps, since they will not have access to the app of the user. @shared_task @shared_task We are simply crawling data every 15 seconds and comparing last price with requested price. If last price is smaller than requested price then we are updating the discount price field. What if price will increase again? @shared_task def track_for_not_discount(): items = Item.objects.all() item items: data = crawl_data(item.url) data[ ] > item.requested_price: print(f ) item_discount_finished = Item.objects.get(id=item.id) item_discount_finished.discount_price = item_discount_finished.save() for in if "last_price" 'Discount finished for {data["title"]}' 'No Discount Yet' Great! Now, it will possible to track discounts properly. You can add one more function which will detect closer prices and notify user about it. For instance, if item price is 100$ and requested price is 97$. But let's keep it simple for now. Finally, we can create our template. tracker.html {% extends 'base.html' %} {% block content %} {% csrf_token %} {{form.as_p}} Send Title Requested Price Last Price Discount Price Date Created {% for item in items %} {{item.title}} {{item.requested_price}} {{item.last_price}} {{item.discount_price}} {{item.date}} {% endfor %} {% endblock %} < = > form method "POST" < = = > button class "btn btn-primary" type "submit" </ > button </ > form < = > table class "table" < > thead < > tr < = > th scope "col" </ > th < = > th scope "col" </ > th < = > th scope "col" </ > th < = > th scope "col" </ > th < = > th scope "col" </ > th </ > tr </ > thead < > tbody < > tr < > td </ > td < > td </ > td < > td </ > td < > td </ > td < > td </ > td </ > tr </ > tbody </ > table Well, you can improve the project by adding email functionality so Django will send email about discounts. Take a look How to Send Email in a Django App You can clone this project from my GitHub repository below https://github.com/raszidzie/Price-Tracker-Application Mission Accomplished! That's it! Make sure you are following me on social media and see you in next post soon Hackers! Stay Connected!
