Get on the Good Foot with MicroPython, Part 2

Photo by Jeremy Hockin of this excruciating tutorial, I taught the reader how to begin with on an ESP32-based development board. We: In the first part MicroPython Flashed the board Frolicked in the REPL Configured WiFi Uploaded scripts Build a circuit with a DS18B20 1-Wire temperature sensor Used MicroPython to read the temperature In part of the tutorial, we’ll take the data we gather with the sensor and publish it over . this MQTT If you’re unfamiliar with the concept, I’ll try to explain MQTT in a nutshell. MQTT in a Nutshell is a machine-to-machine for publishing and subscribing to messages. Importantly, MQTT imposes no constraints upon the nor of those messages. MQTT protocol content structure , you have a single MQTT and one-or-many MQTT . A client may publish messages, subscribe to messages, or both. A client needn’t be an IoT device, a web app, a desktop or mobile app, a microservice, or anything in particular, as long as it speaks MQTT. In a typical setup broker clients All clients connect to the broker. The broker is responsible for receiving published messages and (possibly) delivering them to interested clients. Each message has a “topic”. As is vital to the publish/subscribe pattern, a message’s publisher if anyone is listening. Interested clients will to this topic. doesn’t necessarily care subscribe A MQTT Example You have an MQTT client — perhaps a device with a temperature sensor — called which wants to publish temperature data. It may publish on a topic such as , and the message would be the data, e.g., . bob bob/sensor/temperature 68.75 Another MQTT client, , may want to listen for temperature data so we can display it as a time series graph on a dashboard; would tell the broker it wishes to subscribe to the topic. Finally, when publishes on this topic, the broker notifies , and receives the message. can then do whatever it needs with its data. ray ray bob/sensor/temperature bob ray ray ray Wildcards Subscriptions support . If client had another sensor which reports the relative humidity, it may publish this data under the topic . Client could use a such as , which would receive messages published on . Or perhaps a such as , which would subscribe to topic beginning with . wildcards bob bob/sensor/humidity ray single-level wildcard bob/sensor/+ bob/sensor/humidity and bob/sensor/temperature multi-level wildcard bob/# any bob/ A “topic per client” is merely a convention for sake of our example. MQTT enforces no such constraint. There’s certainly than just the above — but that’s the nutshell, and I’m calling it good. more to it Photo by / Caleb Martin Unsplash Why MQTT? It’s just as important to understand you’d want to use a technology over another (or none at all). why MQTT’s designers had resource-constrained devices (such as sensors) in mind; it’s a “thin” protocol, and easier to implement compared to, say, HTTP. As such, you’ll find that MQTT is a core technology behind many cloud-based “IoT platforms”, including the offerings of , , , , and many others. IBM Amazon Microsoft Adafruit You directly access many of these services via APIs, but it will necessarily consume more of your devices’ resources to do so. can RESTful Using HTTP(S) instead of MQTT makes sense if you need to make a , or if a model is more natural than MQTT’s publish/subscribe model in your problem domain. Even then, protocols such as will demand fewer resources. remote procedure call request/response CoAP Now that we understand what MQTT is all (or more accurately, “partly”) about, let’s use it to spread the word about our ambient temperatures. Boot Script and Temperature Module We’ll use the code from the last tutorial to begin with. For reference, I’ll show them below. You should have two (2) files, the first being our startup script, : boot.py def connect(): import network sta_if = network.WLAN(network.STA_IF) if not sta_if.isconnected(): print('connecting to network...') sta_if.active(True) sta_if.connect(' ', ' ') while not sta_if.isconnected(): pass print('network config:', sta_if.ifconfig()) def no_debug(): import esp # you can run this from the REPL as well esp.osdebug(None) no_debug()connect() And the second is , an abstraction around the temperature sensor: temperature.py import timefrom machine import Pinfrom onewire import OneWirefrom ds18x20 import DS18X20 class TemperatureSensor: """ Represents a Temperature sensor """ def __init__(self, pin): """ Finds address of single DS18B20 on bus specified by `pin` :param pin: 1-Wire bus pin :type pin: int """ self.ds = DS18X20(OneWire(Pin(pin))) addrs = self.ds.scan() if not addrs: raise Exception('no DS18B20 found at bus on pin %d' % pin) # save what should be the only address found self.addr = addrs.pop() def read_temp(self, fahrenheit=True): """ Reads temperature from a single DS18X20 :param fahrenheit: Whether or not to return value in Fahrenheit :type fahrenheit: bool :return: Temperature :rtype: float """ self.ds.convert_temp() time.sleep_ms(750) temp = self.ds.read_temp(self.addr) if fahrenheit: return self.c_to_f(temp) return temp @staticmethod def c_to_f(c): """ Converts Celsius to Fahrenheit :param c: Temperature in Celsius :type c: float :return: Temperature in Fahrenheit :rtype: float """ return (c * 1.8) + 32 Upload both of these files via : ampy $ ampy --port /dev/tty.SLAB_USBtoUART put boot.py && \ ampy --port /dev/tty.SLAB_USBtoUART put temperature.py (Replace with your device path or COM port.) /dev/tty.SLAB_USBtoUART In the first part of this tutorial, I told you to download (or clone) the project. This is not necessary! Read on. micropython-lib Install the MQTT Modules via upip Since your device should be online, we can use from the REPL. is a stripped-down package manager for MicroPython. It's built-in to the ESP32 port of MicroPython; you already have it. It downloads packages from , just like . upip upip PyPi pip Open your REPL, and execute: import upipupip.install('micropython-umqtt.robust') Sample output: Installing to: /lib/Warning: pypi.python.org SSL certificate is not validatedInstalling micropython-umqtt.robust 1.0 from https://pypi.python.org/packages/31/02/7268a19a5054cff8ff4cbbb126f00f098848dbe8f402caf083295a3a6a11/micropython-umqtt.robust-1.0.tar.gz Take note: if your device isn’t online, won’t work from the device’s REPL. upip You also need to grab its dependency, : micropython-umqtt.simple upip.install('micropython-umqtt.robust') Sample output: Installing to: /lib/Installing micropython-umqtt.simple 1.3.4 from https://pypi.python.org/packages/bd/cf/697e3418b2f44222b3e848078b1e33ee76aedca9b6c2430ca1b1aec1ce1d/micropython-umqtt.simple-1.3.4.tar.gz is a barebones MQTT client. depends on ; it’s an MQTT client which will automatically reconnect to the broker if a disconnection occurs. umqtt.simple umqtt.robust umqtt.simple To verify that this installed properly, you can execute from your REPL: from umqtt.robust import MQTTClient No errors? You’re good. Get a MQTT Client App Before we begin the next section, you might want another application handy — a standalone MQTT client. You could try: (GUI; Windows/Mac) MQTT.fx (GUI; Windows/Mac/Linux) MQTTBox from is available via package manager (CLI; Linux/Mac) mosquitto-clients Mosquitto Various free clients on app stores (iOS/Android) can also connect to an MQTT broker (Web; Windows/Mac/Linux) Node-RED Using one isn’t necessary, but will aid experimentation. strictly Experimenting with in the REPL umqtt If you’ve been reading closely, you’ll understand that we need an MQTT (“server”); a MQTT client with no broker is useless. broker It just so happens that MQTT brokers exist; by the project is one such broker. As a member of the public, you can use it! Just be aware: . Don’t publish anything you wouldn’t want to know about. public [test.mosquitto.org](http://test.mosquitto.org/) Mosquitto any data or information you publish on a public MQTT broker is _also_public everyone We’ll use this public broker for the purposes of the tutorial, but if you have a different one you wish to use, you go ahead and do that. Now, let’s try to use our MQTT lib to publish a message on the broker. Create a Unique “Client ID” One caveat to note about MQTT: each MQTT client connected to a broker must have a unique identifier: a . You’ll need to pick a phrase or generate something. I’ll just generate one on the command line: client ID $ python3 -c 'from uuid import uuid4; print(uuid4())'52dc166c-2de7-43c1-88ff-f80211c7a8f6 Copy the resulting value to your clipboard; you’ll need it in a minute. Connect to the REPL Open up a serial connection to your ESP32. I’m going to use here, which Python 3 bundles: miniterm $ python3 -m serial.tools.miniterm --raw /dev/tty.SLAB_USBtoUART 115200 The flag avoids problems with special characters such as and . --raw BS DEL Connect to the Broker As in the first tutorial, I’ll omit the prompt ( ) when working with the REPL. >>> We should now be able to import : MQTTClient from umqtt.simple import MQTTClient The constructor accepts a client ID and a DNS or IP address of a MQTT broker. We’ll use our pseudorandom client ID from above, and for the server, then call : MQTTClient test.mosquitto.org connect() client = MQTTClient('52dc166c-2de7-43c1-88ff-f80211c7a8f6', 'test.mosquitto.org')client.connect() The output of this command, if all went well, should be ; will raise an exception it the connection failed. 0 connect() Connect a Second Client At this point, I’m going to fire up ; I’ll use it to subscribe to the messages which the ESP32 publishes. MQTT.fx I enter server in the server input field, and leave the port field , which is the default (insecure) MQTT port. I then click “Connect,” and wait for negotiation. Here’s a screenshot of my connected client: test.mosquitto.org 1883 MQTT.fx connected to . test.mosquitto.org I’ll come back to MQTT.fx after we learn to publish from the REPL. Publish an MQTT Message Assuming the ESP32 is now connected to the broker, you can publish messages. First, I’ll emit a temperature in Fahrenheit, with the topic : boneskull/test/temperature/fahrenheit client.publish('boneskull/test/temperature/fahrenheit', 72) …but MicroPython complained: Traceback (most recent call last): File " ", line 1, in File "umqtt/simple.py", line 112, in publishTypeError: object of type 'int' has no len() What’s the problem here? Let me explain: An MQTT message payload could be MQTT has no notion of “data types”. It doesn’t know what a “number” or “integer” is. Your payload will always consist of . literally any data. raw bytes There’s no direct mapping of an integer to “bytes,” as there isn’t to encode this number as binary data. We don’t know if this is a or _unsigned_integer, how many bits we should use, etc. just one way signed The problem could have been obvious (and we could have RTFM), but MicroPython shies away from overly “friendly” APIs due to resource constraints, so it’s not obvious what’s happening here. The easiest solution? Publish a instead: str client.publish('boneskull/test/temperature/fahrenheit', '72') If this worked, there should be no output from the statement. Hooray? I’m not convinced — are you? This just squirted the temperature into the ether! We should where these messages are going. I can do that in my MQTT.fx client by to the topic. This is how: see subscribing Subscribing to a topic in MQTT.fx Click on the “Subscribe” tab Enter in the input field boneskull/test/temperature/fahrenheit Click “Subscribe” button to the right of input field After you’ve done this, MQTT.fx will contact the broker, and if successful, you will see the subscription appear beneath the input field: An active subscription in MQTT.fx Next time we (or any client attached to the broker) publishes on this topic, we will see it in the lower-right area of this window, where it is grey and empty. Return to your serial terminal, and run the last command again (you can just hit “up-arrow” then “enter”): client.publish('boneskull/test/temperature/fahrenheit', '72') Switch back to MQTT.fx. It may take a few seconds depending on how busy the broker is, but the message should now appear to the right, along with its payload: A received message in MQTT.fx Excellent work! Now we can use everything we’ve learned, and periodically publish temperature data. Let’s cook up a little module to do that. real A Module to Publish Temperature I’ve written up a little module which uses and (from our first tutorial) to publish temperature data. MQTTClient TemperatureSensor Create : temperature_client.py import time from umqtt.robust import MQTTClient from temperature import TemperatureSensor class TemperatureClient: """ Represents an MQTT client which publishes temperature data on an interval """ def __init__(self, client_id, server, pin, fahrenheit=True, topic=None, **kwargs): """ Instantiates a TemperatureSensor and MQTTClient; connects to the MQTT broker. Arguments `server` and `client_id` are required. :param client_id: Unique MQTT client ID :type client_id: str :param server: MQTT broker domain name / IP :type server: str :param pin: 1-Wire bus pin :type pin: int :param fahrenheit: Whether or not to publish temperature in Fahrenheit :type fahrenheit: bool :param topic: Topic to publish temperature on :type topic: str :param kwargs: Arguments for MQTTClient constructor """ self.sensor = TemperatureSensor(pin) self.client = MQTTClient(client_id, server, **kwargs) if not topic: self.topic = 'devices/%s/temperature/degrees' % \ self.client.client_id else: self.topic = topic self.fahrenheit = bool(fahrenheit) self.client.connect() def publishTemperature(self): """ Reads the current temperature and publishes it on the configured topic. """ t = self.sensor.read_temp(self.fahrenheit) self.client.publish(self.topic, str(t)) def start(self, interval=60): """ Begins to publish temperature data on an interval (in seconds). This function will not exit! Consider using deep sleep instead. :param interval: How often to publish temperature data (60s default) :type interval: int """ while True: self.publishTemperature() time.sleep(interval) Upload this to your board: $ ampy --port /dev/tty.SLAB_USBtoUART put temperature_client.py Your standalone MQTT client app should still be online. Let’s send a message in the REPL, then view the result in the standalone client (please create your own client ID below): from temperature_client import TemperatureClienttc = TemperatureClient('boneskull-test-1516667340', 'test.mosquitto.org', 12, topic='boneskull/test/temperature')tc.start(10) # publish temperature every 10s A word of warning: once you execute the above, the REPL will “hang,” since the method is just . start() busy-waiting Even though this is a busy-wait, does not mean that "nothing happens"; the tick rate in the is 10ms; any sleep time (necessarily using or ) less than or equal to 10ms will preempt other tasks! time.sleep() underlying operating system time.sleep_ms() time.sleep_us() Tab back to MQTT.fx: Real temperature data in MQTT.fx! This will loop indefinitely, so when ready, push the “reset” button on your dev board to get back to the REPL (you need to quit your serial terminal beforehand). don’t Important to note: the “time and date” you see in the payload detail does not mean “when the originating client sent the message.” Rather, it means “when the receiving client received the message.” MQTT messages do not contain a “sent on” timestamp unless you add one yourself! (To do this, you’d need to ask an server or an external module, which is beyond our scope.) NTP RTC We’re successfully published a number! That is great news, except, that number could refer to anything. It’d be helpful to include the unit — either Fahrenheit or Celsius — in the payload. I’ll show you how. Working with JSON As I’ve , MQTT payloads contain anything. That means if you want to send some structured data, are responsible for serialization and deserialization. beaten to death you is a common data interchange format for which MicroPython contains built-in support (unlike, say, that vile Arduino API). It’s trivial to “stringify” a and publish the result. JSON dict To work with JSON — just like in Real Python — we will need to import another module in : temperature_client.py import json Then, add the data to the payload within the method: publishTemperature def publishTemperature(self): """ Reads the current temperature and publishes a JSON payload on the configured topic, e.g., `{"unit": "F", "degrees": 72.5}` """ t = self.sensor.read_temp(self.fahrenheit) payload = dict(degrees=t) if self.fahrenheit: payload['unit'] = 'F' else: payload['unit'] = 'C' self.client.publish(self.topic, json.dumps(payload)) Notice that we didn’t need to coerce the temperature (“degrees”) into a for purposes of publishing, because JSON is a itself—the recipient of this payload will decode the JSON into a numeric value. str str Disconnect from the REPL (that’s if you happen to be using ), and upload to the ESP32 again, then reconnect to the REPL. We don’t need to begin an infinite loop to test it, since we can just call directly: Ctrl-] miniterm temperate_client.py publishTemperature() from temperature_client import TemperatureClienttc = TemperatureClient('boneskull-test-1516667340', 'test.mosquitto.org', 12, topic='boneskull/test/temperature')tc.publishTemperature() The above will send a single message. On the receiving end: Pretty-printed JSON in MQTT.fx If you resize your MQTT.fx window to be tall enough, you’ll see the “Payload decoded by” dropdown in the lower-right. You can see the pretty-printed payload appears as we ‘spected. MQTT.fx also includes Base64 and hex decoders, but the default is “plain text”. I think you have the basics down. But maybe you aren’t going to run your own private MQTT broker. Let’s take this one step further and interface with an IoT platform. Use an ESP32 with MicroPython on IBM Cloud Watson IoT Platform is a service in IBM Cloud (formerly Bluemix). I’ve written a MicroPython module to interface with it, and we’ll use that to save some time. Watson IoT Platform Quickstart You can experiment with this platform without needing to sign up for an account. Visit the page Quickstart Tick “I Accept” after carefully reading the entire terms of use. Enter a unique device identifier in the input box. I’m calling mine “boneskull-esp32-test”. Click “Go”. Watson IoT Platform’s Quickstart Page Keep this browser window open; you’re now ready to send data, and see the result in real-time. Let’s get to it. Upload the module micropython-watson-iot is the module I referenced earlier. Its README contains installation instructions using , but essentially it’s the same as before, via the REPL: micropython-watson-iot upip import upipupip.install('micropython-watson-iot') To verify installation, run: from watson_iot import Device Assuming that didn’t throw an exception, we can use it like so: d = Device(device_id='boneskull-esp32-test')d.connect()d.publishEvent('temperature', {'degrees': 68.5, 'unit': 'F'}) You should see it reflected in your browser. In fact, if you do something like this… import timed.publishEvent('temperature', {'degrees': 68.5, 'unit': 'F'})time.sleep(5)d.publishEvent('temperature', {'degrees': 69.5, 'unit': 'F'})time.sleep(5)d.publishEvent('temperature', {'degrees': 67.5, 'unit': 'F'})time.sleep(5)d.publishEvent('temperature', {'degrees': 66.5, 'unit': 'F'}) …you should see a nifty line graph: Real-time graph of our temperature data You’re welcome to play with this in more depth; Watson IoT Platform has a free tier. To sign up, you need to: 1. (no credit card needed)2. using the “free plan” from the catalog3. Click “Launch” to explore the platform.4. Also, check out . Register with IBM Cloud Create a Watson IoT Platform service instance the docs The library offers a few “quality of life” benefits—as IoT platforms typically do—when compared to a vanilla MQTT client and/or broker: micropython-watson-iot Messages contain metadata such as “published on” time, handled by the cloud platform You can group devices via logical “device types” Structured data can be automatically encoded/decoded to/from JSON (it does this by default) Create your own custom encoders and decoders (e.g., numeric, Base64) Create custom “command handlers,” which cause the device to react upon reception of a “command”-style MQTT message. For example, you could send a command to blink an onboard LED or reboot the device. I’ve committed a few ; you can use adapt these patterns to your own code. micropython-watson-iot examples There’s really more going on here than just MQTT — dashboards and gateways and all sorts of hoodoo that I am not going to go into. But now it’s easy to use with MicroPython on an ESP32, thanks to ME. a lot Ahem… Recap, Obligatory Link Dump, & Goodbyes In this tutorial, we’ve learned: What MQTT is (and what it’s for) How to talk to an MQTT broker using MicroPython and an ESP32 How to publish structured data Install MicroPython libraries from PyPi via upip How to subscribe to simple topics via a standalone MQTT client How to publish data to Watson IoT Platform via its site, using Quickstart micropython-watson-iot for more info on usage and discussion of its limitations. Check out the README of [micropython-watson-iot](https://github.com/boneskull/micropython-watson-iot/blob/master/README.md) I’ve posted the complete example files for your convenience. in this Gist Thanks for reading! Extra thanks for , too. doing This article originally appeared January 25, 2018 on boneskull.com .