Member Of Technical Staff - Engineering
Dreams to achieve for hackers in the startup. While you push hard to achieve these, you spend a lot of time fixing the legacy monolith. System with infinite series of quick fixes, repeated codes and millions of lines.
Its okay to start business with battery loaded ERP system like odoo. It helps you build your application with business requirements very quickly. But over the time, its gonna be harder for you to scale the application, as majority of the computing resources will be wasted for unwanted functionalities and supporting these ‘easy to develop’ features. Make sure that you are developing optimized application parallel and have a good migration plan for existing data. If you explore and hack the base/inner system of openerp you will see lot of optimization steps.
Profiling application code helps you to identify time/space issues in the application. You will get answers for typical questions like how long each function/method takes to execute? how good is the application at memory consumption etc. In the web apps world, long running or high memory consuming requests are threat to the application stability(Even in blocking and non blocking requests). If you keep these heavy loading endpoints with you, and don’t invest time to optimize it, and do scale out, you will end up a lot on your cloud provider’s invoices.
There are multiple tools and libraries available for profiling in python world. cProfile, valgrind are freely available. 3rd party web monitoring services like NewRelic, App dynamics, Dynatrace available for certain cost. This article focuses profiling odoo with python line profiler. Even though line profiler adds its own overheads to the code, and you can’t trust the exact timing from line profiler, for identifying relative time consumption, line profiler is a good choice.
Lets see how we can setup odoo 9 for development.You could use the following script to auto configure the environment. Run the script to setup the odoo 9 and PostgreSQL 9.5. If you want to customize username, password etc for your environment, download the script and edit the configurations in the script itself and run it.
The script installs PostgreSQL 9.5 and dependency packages for odoo. Python packages are installed in a virtual environment under user ‘odoo’.
[email protected]:~# wget --quiet http://bit.ly/2cE6pfY -O install_odoo9.sh && bash -u install_odoo9.sh
Lets setup initial odoo database and load some demonstration data. Used following configuration for initial database installation
Odoo admin password: odoo_admin
Odoo database: odoo_test
Odoo username: [email protected]
Odoo password: 123456
Installed below odoo apps with the default installation
Please see the script to populate around `50 thousand` contacts to odoo. Script is based on Odoo API. (It is very slow, Didn’t use any multiprocessing techniques to populate it faster. Took around 2 hours to populate full, while running in background).
Lets log some request and response timing from odoo.
[email protected]:~# systemctl stop odoo-server
[email protected]:~# su odoo
(venv) [email protected]:~$ which python
(venv) [email protected]:~$ python src/odoo.py --config /etc/odoo-server.conf --log-handler openerp.http.rpc.request:DEBUG --logfile /dev/stdout|tee /opt/odoo/logs/odoo-server-request.log
Please go through typical operations of odoo, to generate some random responses and timing in logs.
In worker mode of odoo, server will be running with wsgi standards using sync and non threaded mode. If we have 4 workers and each request taking 1 second to generate output, our system’s concurrency will be 4 requests per second.
Lets find out the requests taking long time. As a base measure, check the responses taking more than 500 ms to get generated. We can use any of the three methods for that.
According to the previous command, we are logging requests and response timing to stdout along to `/opt/odoo/logs/odoo-server-request.log`.
Lets use the odoo log line parser. To generate the log files, we need to pass the log handler argument
when running odoo. This handler will add request and response timing to log file. Please see the attached file for parser.
You can run the following to generate the log slow responding odoo requests.
[email protected]:~# wget --quiet http://bit.ly/2ceKpfa -O odoo_request_log_parser.py && python odoo_request_log_parser.py -l '/opt/odoo/logs/odoo*.log'
I played with `Unlimited` listing of `0.1 million` contacts multiple times. The unlimited listing in odoo is a dangerous feature, if the number of records is very high for a table, the ‘All’ listing will be a big blocking operation for web workers. Processing these records consumes huge memory as well.
Lets check the functions listed here from log parser. `poll` is used for long polling and in production ERP, we can configure it with `gevent` async port. So its not a big issue for us. `search_read: res.partner` has an apdex score of 0.240 and average execution time of 14 seconds. OMG!!. its not acceptable at all. `get_session_info` has apdex score of 0.5 and taking 520 ms to generate.
We could do line profile of these functions to identify the bottlenecks.
Cpython as interpreted language, gives advantage on profiling code line by line. We can identify the time difference between each line and number of times the line got called. Narrowing performance bottlenecks allows us for straight forward optimization of code. Though there are multiple profilers for cpython, line profiler is simple to use and direct us to focusable areas.
Activate virtual environment. Install
See how we can check the line by line consumption in our log parser script.
Edit the file and add decorator `@profile` to the main function generate_fn_table. And run command along with -lv options.
(venv) [email protected]:~$ kernprof -lv -o /tmp/lp.prof odoo_log_parser.py -l '/opt/odoo/logs/*.log'
PS: We will be ignoring profile outputs in the file /tmp/lp.prof, the verbose mode `-v` make sure we get profile report when we terminate the application using Using CTRL + C (Sending the SIGINT signal).
To enable profiling in Odoo
We identified the endpoint methods to debug. Which has low apdex scores. Lets go through this method first.
Make sure you have created your own git profiling branch. As this involves editing code parts at some places.
(venv) [email protected]:~$ kernprof -lv -o /tmp/odoo.lprof ~/src/odoo.py --config /etc/odoo-server.conf --logfile /dev/stdout
Perform the slow loading operations in Odoo web interface multiple time. Press CTRL +C to terminate the application. You will get the profile report.
Profiling gives us bottle necks on codes. We need to do different aproaches to speed up the code execution. Some techniques we could use are
Create your free account to unlock your custom reading experience.