Building Your Own ESP8266/ESP32 Over-The-Air Firmware Updater [A How-To Guide] Part II

This is a continuation of Part 1 of our initial simple OTA solution for the ESP8266 or ESP32. In this next part we’ll look into how we can build a more comprehensive OTA solution that does the following:

Build the ESP8266 into something that is actually half useful; a humidity, temperature and moisture plant sensor.
‘Dial home’ to check whether there are new firmware updates available.
Update the module depending to the latest version number as defined by our database.

I have found this a very useful thing to learn while I’ve been trying to reduce the size of my projects through the use of surface mounted ICs. While they can be incredibly small, it’s pretty much impossible to reprogram them after soldering them to a PCB!

Setting the scene

Part 1 showed a relatively useless example where we uploaded the ‘Blink’ sketch over-the-air. For this guide we’ll turn our project into a plant monitor that can sense the humidity of the soil, in addition to sensing the ambient temperature and humidity.

For the guide we’ll be using the following:

ESP8266/ESP12E (ESP32 will work as well)
DHT11
Capacitive soil moisture sensor

We’ll build a quick and dirty prototype where we connect all sensors to the ESP8266.

Database setup

We want to check the ESP8266 to periodically ping our database and ask whether there is a new firmware update available. If the answer is yes we perform an automatic update.

First we set up our database and associated table. For this we use MySQL and the follow SQL:

mysql> CREATE DATABASE OTA;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE TABLE Ping (
    -> mac_id CHAR(17) PRIMARY KEY NOT NULL,
    -> available_firmware_version INT(2) NOT NULL,
    -> last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
    -> );
Query OK, 0 rows affected (0.01 sec)

Now that we’ve got a table we can use to query whether our ESP8266 needs an update we can go ahead and insert a row for our device. We know from the previous lesson that out MAC address is CC:50:E3:DC:90:2A.

mysql> INSERT INTO Ping (mac_id, available_firmware_version) VALUES ('CC:50:E3:DC:90:2A',2);
Query OK, 1 row affected (0.01 sec)

Now let’s set up our NodeJS app.

NodeJS module

We want to build a module that pings our database with the MAC address and let’s us know whether an update is available. We then want to update the module and update the database to reflect the new firmware on the device.

Note that the below code is a continuation of what we’ve already built in Part 1.

Firstly my database handler:

var mysql = require('mysql')
var config = require('../js/config')

var connection = mysql.createConnection({
	host: 'localhost',
	user: 'root',
	password: config.db_password,
	database: 'OTA'
})

var ping = function(mac_id, callback){

	var sql = "SELECT * FROM Ping WHERE mac_id = ?"

	connection.query(sql, mac_id, function(err, rows, fields) {
		if (!err) {
			if (rows.length === 0) {
				callback(null, null)
			} else {
				callback(null, rows)
			}
		} else {
			console.log(err)
			callback(err, null)
		}
	})

}

module.exports = {
	ping: ping
}

And we add the new POST and update logic to our NodeJS app:

var express = require('express')
var path = require('path')
var router = express.Router()
var md5 = require('md5-file')
var db = require('../js/databaseHandler')

router.get('/update', function(req, res, next) {

	var updateVersion = req.headers['x-esp8266-version']

	var filePath = path.join(__dirname, '../updates/plant_sensor_v'+updateVersion+'.bin')

	var options = {
		headers: {
			"x-MD5":  md5.sync(filePath)
		}
	}

	res.sendFile(filePath, function (err) {
		if (err) {
			next(err)
		} else {
			console.log('Sent:', filePath)
		}
	})

})

router.post('/ping', function(req, res, next) {

	var mac_id = req.body.mac_id

	console.log(req.body)
	console.log(mac_id)

	db.ping(mac_id,function(err, results){
		(results) ? res.send({'mac_id':results[0].mac_id,'available_firmware_version':results[0].available_firmware_version}) : res.send({'error':err}) 
	})

})

module.exports = router

ESP8266 sketch

Now we build out our sketch and generate two different version numbers. The sketch works as follows:

We connect to Wifi.
We send our MAC address to the server which pings back the latest firmware version.
We check whether our firmware needs an update and update if necessary.
We start the sensor measurements.

And finally we upload our binaries to the server, and make sure we have different version numbers for each build.

#define firmware_version 1 // For the first sketch
#define firmware_version 2 // For the second sketch

The complete code:

#include <ESP8266httpUpdate.h>
#include <ESP8266HTTPClient.h>
#include <ArduinoJson.h>
#include "DHT.h"

// Set out Wifi auth constants.
const char* ssid =     "Yolo";
const char* password = "password";

// Define our constants.
#define DHTPIN 5
#define DHTTYPE DHT11
#define CSMS A0

// Set your firmware version here. Your other sketch should have a different version number.
#define firmware_version 1

// Initialise the DHT11 sensor.
DHT dht(DHTPIN, DHTTYPE);

// Declare DHT output value variables.
int output_value;
int output_raw;

void setup() {

  // Initialise Serial connection.
  Serial.begin(74880);
  Serial.setDebugOutput(true);

  // Start HTTPClient.
  HTTPClient http;

  // Start Wifi.
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(1000);
  }

  // Record our MAC address.
  String mac = "mac_id=" + String(WiFi.macAddress());

  // Allocate the JSON document.
  const size_t capacity = JSON_OBJECT_SIZE(2) + 60;
  DynamicJsonDocument doc(capacity);

  // Set up our HTTP request.
  http.begin("http://test.derkzomer.com/ping");      //Specify request destination
  http.addHeader("Content-Type", "application/x-www-form-urlencoded");  //Specify content-type header

  // Send the request and get the response payload.
  int httpCode = http.POST(mac);
  String payload = http.getString();

  //Close HTTP connection.
  http.end();

  // Parse the received JSON object.
  deserializeJson(doc, payload);
  const char* macId = doc["macId"];
  int available_firmware_version = doc["available_firmware_version"];
  String fwv = String(available_firmware_version);

  // Check whether your firmware is outdated.
  if (available_firmware_version > firmware_version) {

    Serial.println("Your firmware version is V"+String(firmware_version)+", the latest available firmware version is V"+available_firmware_version)+".";
    Serial.println("Installing the new update now...");
  
    t_httpUpdate_return ret = ESPhttpUpdate.update("http://test.derkzomer.com/update",fwv);
    
    switch(ret) {
        case HTTP_UPDATE_FAILED:
            Serial.printf("[update] Update failed (%d): %s", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
            break;
        case HTTP_UPDATE_NO_UPDATES:
            Serial.println("[update] Update no Update.");
            break;
        case HTTP_UPDATE_OK:
            Serial.println("[update] Update ok."); // may not be called since we reboot the ESP
            break;
    }

  } else {
    Serial.println("Your firmware version is V"+String(firmware_version)+", the latest available firmware version is V"+available_firmware_version)+".";
    Serial.println("You have the latest version.");
  }

  // Start the DHT11 sensor.
  dht.begin();

}

void loop() {

  // Measure and map the raw moisture output from the moisture sensor.
  output_raw = analogRead(CSMS);
  output_value = map(output_raw, 725, 330, 0, 100);

  // Print the mapped output from the moisture sensor.
  Serial.print("Moisture : ");
  Serial.print(output_value);
  Serial.println("%");

  // Measure the humidity and temperature from the DHT11.
  float h = dht.readHumidity();
  float t = dht.readTemperature();
  float f = dht.readTemperature(true);

  // Check whether the DHT11 sensor is working.
  if (isnan(h) || isnan(t) || isnan(f)) {
    Serial.println(F("Failed to read from DHT sensor!"));
    delay(2000);
    return;
  }

  // Index the DHT11 values.
  float hif = dht.computeHeatIndex(f, h);
  float hic = dht.computeHeatIndex(t, h, false);

  // Print the DHT11 humidity and temperature values.
  Serial.print(F("Humidity: "));
  Serial.print(h);
  Serial.print(F("%  Temperature: "));
  Serial.print(t);

  // Wait five seconds before the next measurement.
  delay(5000);

}

Let’s now move the binaries over to our server under the ‘/uploads/’ folder and test it out. Let’s upload the initial V1 sketch to our ESP8266 via serial and see what happens.

The above shows you the serial output of how the ESP8266 checks for new updates by pinging the server, it gets told there is a new firmware update available, and then flashes the module. It then restarts, pings the server again, but now it skips the firmware update because you already have the latest version and starts its sensor measuring activities.

So now you’ve built a complete version controlled OTA solution that allows you to install new firmware wirelessly onto your ESP8266!

Next I’m thinking of reducing the size of my project from around 170 square cm to about 25 square cm, or about a ~7x reduction in size, through the use of a custom built PCB and smaller IC’s.

So for all of that, and more, please stay tuned for a new article.

Previously published at https://medium.com/@derk_zomer/esp8266-ota-solution-part-2-28bbd9b82429