Glimpse Into Django Internals for Enterprise Software Developers, BI and Automation

TL;DR: Django ORM and internal utilities have rich functionality that allows you to automate some complex data manipulations, automate processes that rely on multiple models in a way that will not require updates every time the data structure changes collect data from the database quickly without diving into the project very deeply automate some complex data manipulations, automate processes that rely on multiple models in a way that will not require updates every time the data structure changes collect data from the database quickly without diving into the project very deeply In order to do this, we need a few tools like the Django Model `_meta` object and some basic Python code. Reasoning There are many examples when it is necessary to automate some tasks with Django models without knowing the model structure.Let's take a look at one example. I will use GDPR compliance for this. Under GDPR, user may ask for the deletion of their data. In that case, we need to remove all data from the database; however, in a large enterprise project, data may be spread across multiple tables in the DB, multiple models or even multiple databases and projects. During one day, new models can be added by multiple developers, and it is very easy to lose track of these changes. Without the use of some very abstract code and some Django internal utilities, it is impossible to create a piece of code that will handle these cases and will not require updates every time the data structure changes A very simple model structure example from django.conf import settings from django.db import models class Meeting(models.Model): topic = models.CharField(max_length=200) participants = models.ManyToManyField(settings.AUTH_USER_MODEL, through='Participation', blank=True) class Participation(models.Model): meeting = models.ForeignKey('Meeting', on_delete=models.CASCADE, related_name='participations') user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name='participations') class Comment(models.Model): meeting = models.ForeignKey('Meeting', on_delete=models.CASCADE, related_name='comments') user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) description = models.TextField(max_length=3000) attachment = models.FileField(null=True, blank=True) class UserDocumentGroup(models.Model): name = models.CharField(max_length=200) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) class UserUploadedDocument(models.Model): group = models.ForeignKey(UserDocumentGroup, on_delete=models.CASCADE) document = models.FileField() from django.conf import settings from django.db import models class Meeting(models.Model): topic = models.CharField(max_length=200) participants = models.ManyToManyField(settings.AUTH_USER_MODEL, through='Participation', blank=True) class Participation(models.Model): meeting = models.ForeignKey('Meeting', on_delete=models.CASCADE, related_name='participations') user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name='participations') class Comment(models.Model): meeting = models.ForeignKey('Meeting', on_delete=models.CASCADE, related_name='comments') user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) description = models.TextField(max_length=3000) attachment = models.FileField(null=True, blank=True) class UserDocumentGroup(models.Model): name = models.CharField(max_length=200) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) class UserUploadedDocument(models.Model): group = models.ForeignKey(UserDocumentGroup, on_delete=models.CASCADE) document = models.FileField() This is a very simple basic example of Django models that I will be using throughout this article Introduction to Django internals Django have various internal utilities that allow you to access and manipulate the structure of your models without knowing them. Let's explore some of the possibilities before we dive into solving something.For example, it would be good to have the ability to get a list of models from the project and maybe filter them by some arbitrary rules. Luckily, Django has a very simple way to do it. from django.apps import apps all_models = apps.get_models() import pprint pprint.pprint(all_models) # [ , # , # , # , # , # , # , # ] from django.apps import apps all_models = apps.get_models() import pprint pprint.pprint(all_models) # [ , # , # , # , # , # , # , # ] As you can see, there are all models, including internal Django models like ContentType. In order to effectively work with this list, we may need some Django Model `_meta` API.Minor note: Unfortunately, the Django official documentation regarding this topic is fairly poor - https://docs.djangoproject.com/en/5.1/ref/models/meta/ . A good understanding of this API goes beyond the use of Options.get_fields method. So, if you are interested in this topic, I recommend importing a model and playing around with it in a Django shell. It is simple to understand, as long as you play with it a bit, but it takes some time to dive into it. It is also good to check the docs about fields - https://docs.djangoproject.com/en/5.1/ref/models/fields/, and it is good to consult official docs about any API you are going to use and any questions you stumble uponSo far, I will use some utilities and explain how they work along the way.For example, let's try to filter all models that have a foreign key or a one-to-one field toUsermodel Minor note: https://docs.djangoproject.com/en/5.1/ref/models/meta/ Options.get_fields https://docs.djangoproject.com/en/5.1/ref/models/fields/ User def filter_user_related_models(all_models: list[type[models.Model]]) -> list[type[models.Model]]: user_related_models = [] for model in all_models: for field in model._meta.get_fields(include_parents=True, include_hidden=True): if field.one_to_one or field.many_to_one and field.related_model == User: user_related_models.append(model) break return user_related_models pprint.pprint(filter_user_related_models(all_models)) # [ , # , # ] def filter_user_related_models(all_models: list[type[models.Model]]) -> list[type[models.Model]]: user_related_models = [] for model in all_models: for field in model._meta.get_fields(include_parents=True, include_hidden=True): if field.one_to_one or field.many_to_one and field.related_model == User: user_related_models.append(model) break return user_related_models pprint.pprint(filter_user_related_models(all_models)) # [ , # , # ] A few explanations regarding the used code: get_fields method returns a list of all fields of a model one_to_one and many_to_oneare flags that indicate that this field is an FK or O2O field. get_fields method returns a list of all fields of a model get_fields one_to_one and many_to_oneare flags that indicate that this field is an FK or O2O field. one_to_one many_to_one Regarding related fields, I need to stop and explain a little more with examples. Let's play a little with data in the shell. Meeting._meta.get_field("comments").many_to_one # Out[43]: False Comment._meta.get_field("meeting").many_to_one # Out[45]: True Meeting._meta.get_field("comments").one_to_many # Out[44]: True Comment._meta.get_field("meeting").one_to_many # Out[46]: False Meeting._meta.get_field("comments").many_to_one # Out[43]: False Comment._meta.get_field("meeting").many_to_one # Out[45]: True Meeting._meta.get_field("comments").one_to_many # Out[44]: True Comment._meta.get_field("meeting").one_to_many # Out[46]: False So what happened? The answer to this is simple, but nuanced:get_field and get_fields methods give access not only to the fields of a model, but also to relations, including reversed relations. So if Comment has a FK to Meeting it will be accessible through Comment._meta.get_field("meeting") from Comment side and through Meeting._meta.get_field("comments") from Meetingside, but they are two different fields. get_field get_fields Comment Meeting Comment._meta.get_field("meeting") Comment Meeting._meta.get_field("comments") Meeting Comment._meta.get_field("meeting") # Out[47]: Meeting._meta.get_field("comments") # Out[48]: Comment._meta.get_field("meeting") # Out[47]: Meeting._meta.get_field("comments") # Out[48]: These are very important distinctions. If some model has a FK to another model, that means that on the business level of abstraction, it reflects that this model most likely belongs to another model, but not vice versa. For example,Comment is a part of Meeting, but Meeting is not a part of Comment. Another example can be: if Order has a FK to User, then Order is a part of User, but User is not a part of Order. ManyToManyrelations most of the time reflect a different situation. It is most likely a relation between two separate entities that are not parts of each other. Comment Meeting Meeting Comment Order User Order User User Order ManyToMany Back to GDPR example For our GDPR example, all of the above means that we need to look at the FK and O2O relations that go down from User.First, let's collect and, for better understanding, visualise what we are working with. There is a very simple tree library that can help us with this:https://treelib.readthedocs.io/en/latest/First, I will collect all models that are part of theUser. Knowing all the information above, I will write a recursive function that will go over all User model relations, collect related models that are part of User, and add them to the tree. User https://treelib.readthedocs.io/en/latest/ User User User def get_traced_relations( model: type[models.Model], parent: str | None = None, tree: Tree | None = None, ) -> Tree: if not tree: tree = Tree() tree.create_node(identifier=str(model), parent=parent) parent = str(model) fields = model._meta.get_fields(include_parents=True) for field in fields: field_node_id = f"{str(field.related_model)} ({str(field.remote_field)})" if field.is_relation and (field.one_to_many or field.one_to_one): tree.create_node( identifier=field_node_id, parent=parent, data={ "model": field.related_model, "field": field.remote_field, } ) get_traced_relations( model=field.related_model, parent=field_node_id, tree=tree, ) return tree def get_traced_relations( model: type[models.Model], parent: str | None = None, tree: Tree | None = None, ) -> Tree: if not tree: tree = Tree() tree.create_node(identifier=str(model), parent=parent) parent = str(model) fields = model._meta.get_fields(include_parents=True) for field in fields: field_node_id = f"{str(field.related_model)} ({str(field.remote_field)})" if field.is_relation and (field.one_to_many or field.one_to_one): tree.create_node( identifier=field_node_id, parent=parent, data={ "model": field.related_model, "field": field.remote_field, } ) get_traced_relations( model=field.related_model, parent=field_node_id, tree=tree, ) return tree And let's check the result.Treelibhave a useful api that allows us to visualise the tree. Treelib tree = get_traced_relations(User) tree.show() # # ├── (meetings.Comment.user) # ├── (meetings.Participation.user) # └── (meetings.UserDocumentGroup.user) # └── (meetings.UserUploadedDocument.group) tree = get_traced_relations(User) tree.show() # # ├── (meetings.Comment.user) # ├── (meetings.Participation.user) # └── (meetings.UserDocumentGroup.user) # └── (meetings.UserUploadedDocument.group) Treelib library also has a way to make a graphviz diagram from the tree. In this case, it looks like this. Treelib As you see, it even collected theUserUploadedDocument, which is a part of UserDocumentGroup that is a part of User.Now, let's actually collect some data for the user. There are multiple ways to do it, but in order to keep it simple, I will use a very basic approach. To do it, we will need a few utilitiesFirst, we need to have a function that will return a path from the tree root to the node. UserUploadedDocument UserDocumentGroup User def path_to_node(tree: Tree, node: Node) -> list[Node]: path = [node] if (parent := tree.parent(node.identifier)): path.extend(path_to_node(tree, parent)) return path def path_to_node(tree: Tree, node: Node) -> list[Node]: path = [node] if (parent := tree.parent(node.identifier)): path.extend(path_to_node(tree, parent)) return path Second is just to get a lookup for building a queryset. Just a regular lookup that is used in cases likeUserUploadedDocument.objects.filter(group__user=user) UserUploadedDocument.objects.filter(group__user=user) def get_lookup_for_node(path: list[Node]) -> str: return "__".join(node.data["field"].name for node in path if not node.is_root()) def get_lookup_for_node(path: list[Node]) -> str: return "__".join(node.data["field"].name for node in path if not node.is_root()) Now let's see what we have. I exclude root specifically, because root is aUsermodel, so we don't need to query it. User for node in tree.all_nodes(): if not node.is_root(): print( node.data["model"], "lookup", get_lookup_for_node(path_to_node(tree, node)) ) # lookup user # lookup user # lookup user # lookup group__user for node in tree.all_nodes(): if not node.is_root(): print( node.data["model"], "lookup", get_lookup_for_node(path_to_node(tree, node)) ) # lookup user # lookup user # lookup user # lookup group__user Now having all of this, we can actually do the query. But to use this, we actually need to know what we are searching for. For example, we have 2 file fields: attachment in Comment model document in UserUploadedDocumentmodel attachment in Comment model attachment Comment document in UserUploadedDocumentmodel document UserUploadedDocument As we agreed before, because all of these models are related to `User` via FK, we can assume that these files are presumably uploaded by User. So let's create a function that will find every document that was uploaded by the user. First, let’s create some data, for simplicity, i will usemodel-bakery (https://github.com/model-bakers/model_bakery) library model-bakery https://github.com/model-bakers/model_bakery from model_bakery import baker user = baker.make("auth.User") meeting = baker.make("meetings.Meeting") baker.make("meetings.Comment", meeting=meeting, user=user) baker.make("meetings.Comment", meeting=meeting, user=user) document_group = baker.make("meetings.UserDocumentGroup", user=user) baker.make("meetings.UserUploadedDocument", group=document_group) baker.make("meetings.UserUploadedDocument", group=document_group) from model_bakery import baker user = baker.make("auth.User") meeting = baker.make("meetings.Meeting") baker.make("meetings.Comment", meeting=meeting, user=user) baker.make("meetings.Comment", meeting=meeting, user=user) document_group = baker.make("meetings.UserDocumentGroup", user=user) baker.make("meetings.UserUploadedDocument", group=document_group) baker.make("meetings.UserUploadedDocument", group=document_group) In order to find all files uploaded by the user, we can walk over the tree nodes, find models that have a file field in them and query them from the database. from django.db import models def get_all_models_with_file_fields(tree: Tree) -> list[Node]: models_to_search = [] for node in tree.all_nodes(): if node.is_root(): continue for field in node.data["model"]._meta.get_fields(): if isinstance(field, models.FileField): models_to_search.append(node) break return models_to_search get_all_models_with_file_fields(tree) # [Node(tag= (meetings.Comment.user), identifier= (meetings.Comment.user), data={'model': , 'field': }), # Node(tag= (meetings.UserUploadedDocument.group), identifier= (meetings.UserUploadedDocument.group), data={'model': , 'field': })] from django.db import models def get_all_models_with_file_fields(tree: Tree) -> list[Node]: models_to_search = [] for node in tree.all_nodes(): if node.is_root(): continue for field in node.data["model"]._meta.get_fields(): if isinstance(field, models.FileField): models_to_search.append(node) break return models_to_search get_all_models_with_file_fields(tree) # [Node(tag= (meetings.Comment.user), identifier= (meetings.Comment.user), data={'model': , 'field': }), # Node(tag= (meetings.UserUploadedDocument.group), identifier= (meetings.UserUploadedDocument.group), data={'model': , 'field': })] As you can see, we have 2 models that have a file field. Which is exactly correct accordingly to the model’s declaration. Now, let's just collect the needed data def collect_data_for_user(user: User, tree: Tree, nodes: list[Node]) -> dict[type[models.Model], list[models.Model]]: data = {} for node in nodes: model = node.data["model"] data[model] = model.objects.filter( **{get_lookup_for_node(path_to_node(tree, node)): user} ) return data collect_data_for_user(user, tree, desired_nodes) # Out[41]: # {meetings.models.Comment: , ]>, # meetings.models.UserUploadedDocument: , ]>} def collect_data_for_user(user: User, tree: Tree, nodes: list[Node]) -> dict[type[models.Model], list[models.Model]]: data = {} for node in nodes: model = node.data["model"] data[model] = model.objects.filter( **{get_lookup_for_node(path_to_node(tree, node)): user} ) return data collect_data_for_user(user, tree, desired_nodes) # Out[41]: # {meetings.models.Comment: , ]>, # meetings.models.UserUploadedDocument: , ]>} And there you go. Found every instance of every model in the database that has a file field and is related to the user that we specified. Now, in case we are obliged by gdpr to delete user data, we can rely on this script regardless of how many models we have and how many times these models were changed by developers. We don't need to update this script every time we change the model structure, nor don't need to track every minor change in code, etc. However, in case of deletion, precautionary measures must be taken, because every automated tool made in this manner tends to scrap to much data Conclusion Django internal utilities are a very powerful tool for automation. There are many use cases where they can help a lot, such as business intelligence, data analysis and automation of processes. The example above is very basic, yet it can demonstrate the power of these tools.