How to Change Django Code during Seamless Deployment to Production Server
Depending on the environment, working with Django is not always as easy as it is shown in tutorials. It may seem simple – if you need to add a column, add a field to a model, create a migration, run it and the column is ready. And, if you need to delete it, then everything is just as simple: delete the field, create a migration, run it and it is done. However, it does not always work that way in production.
When you have several servers or containers running on Prod, and migrations are published before the code is published, then it becomes important not to break the seamless deployment. That is, you need to maintain the consistency of a database and code, and even more.
In a previous article, I wrote about how to deploy Django model changes to Production. Let's move on to the details.
Deleting Column on Production
Depending on the deployment architecture, to delete a column on production, you can use several methods.
- Performing migrations after deploying the code to all pods in one deployment.
- Performing migrations with the second deployment, immediately after the first one in which the code was deployed.
- Performing migrations with a separate deployment after deploying the code but with a possible delay and even other deployments.
And now, let us look at this in more detail. Let us take the Airport list model as an example.
class Airport(models.Model):
iata = models.CharField(max_length=3, unique=True)
name = models.CharField(max_length=30, unique=True)
If you delete the ‘name’ column from the model and do not create or perform migrations, nothing bad will happen. But if you try to start creating migrations, a migration will be added with the removal of this field:
python manage.py makemigrations
Nevertheless, keeping an extra unused field in a database is not the best idea. Therefore, it seems logical to create a migration and then execute it, thereby removing this field from the table in the database.
As a result, a test on a local machine shows that everything is working. But when we started to deploy to Prod, a bunch of errors appears. So, everything is different on Prod. When pods with new code are deployed, the following most often happens.
- Initialization is performed before each deployment, which includes the execution of migrations:
python manage.py migrate - After that, pods are created, one at a time. Once a pod is created, one of the old pods is destroyed.
Now imagine that the migrations have been completed, but the old code is still working. And it will keep working for a while depending on the number and complexity of pods, sometimes for a few seconds, and sometimes for a few minutes or even tens of minutes.
So, after the migration is completed, until the replacement of the last pod, almost any request to the Airport model will throw an exception. For example:
Airport.objects.first()
will throw an exception like:
ProgrammingError: column avia_airport.name does not exist
LINE 1: SELECT "avia_airport"."id", "avia_airport"."iata", "avia_airport"."name"
Solutions
The first solution suggests that you have two types of migration: the one performed before the code is deployed and the one performed after that. If you have this in CI / CD, then everything is simple. Adding a column, creating a model, and so on must be done before deployment, and deleting must be performed after that.
The second solution is only possible if you are completely sure that you can perform two deployments in a row without a break and during this time, no one will deploy or change anything. Then you create a migration as usual. After that, you first deploy only code, and then you run the migration.
The model class for the previous two methods will look as follows:
class Airport(models.Model):
iata = models.CharField(max_length=3, unique=True)
And the migration will look as follows:
migrations.RemoveField(
model_name='avia_airport',
name='name',
)
And finally, the third solution is for those cases where you cannot guarantee that two versions will be deployed in a row without any interference. In this case, you need to create a "fake" migration. For this, state_operations is used. For example:
migrations.RunSQL(
sql=migrations.RunSQL.noop,
reverse_sql=migrations.RunSQL.noop,
state_operations=[
migrations.RemoveField(
model_name='avia_airport',
name='name',
),
]
)
This way you inform Django that this migration is completed. And when you run the makemigrations command, Django will not add the migration with the 'name' field deletion. However, this migration will not delete this field from the table in the database. So, your first deployment will contain the removal of the field from the model in the code and the ‘fake’ migration.
Then, the second deployment will have to contain the column removal code. But since Django thinks the column was removed in the previous migration, it will have to be deleted using SQL commands.
migrations.RunSQL(
sql="ALTER TABLE public.avia_airport DROP COLUMN name CASCADE",
reverse_sql=migrations.RunSQL.noop
)
If you are not sure what to write in SQL, then create an original migration and run the sqlmigrate command:
python manage.py sqlmigrate avia
You will see the needed SQL command which you can copy.
By the way, I highly recommend specifying reverse_sql in every manual migration. If you have a specific migration rollback script, then reverse_sql should run it. If you do not have it, then use migrations.RunSQL.noop for the RunSQL migration or migrations.RunPython.noop for RunPython migrations – an empty migration. This will help avoid an exception in case you need to roll back the migration.
Conclusion
Always keep in mind that there are several Django instances running on production, which are updated to a new version sequentially. Also, remember that there is a period of time between performing a migration and updating the code in which users can see an error. This can cause reputational and financial damage. Therefore, use multi-step deployments when changing the database.