Collecting and Analyzing IoT Data in Near Real-Time with AWS IoT, LoRa, and LoRaWAN

Gary Stafford · Collecting and Analyzing IoT Data in Near Real-Time with AWS IoT, LoRa, and LoRaWAN

Introduction

In a recent post published on ITNEXT, LoRa and LoRaWAN for IoT: Getting Started with LoRa and LoRaWAN Protocols for Low Power, Wide Area Networking of IoT, we explored the use of the LoRa (Long Range) and LoRaWAN protocols to transmit and receive sensor data, over a substantial distance, between an IoT device, containing several embedded sensors, and an IoT gateway. In this post, we will extend that architecture to the Cloud, using AWS IoT, a broad and deep set of IoT services, from the edge to the Cloud. We will securely collect, transmit, and analyze IoT data using the AWS cloud platform.

LoRa and LoRaWAN

According to the LoRa Alliance, Low-Power, Wide-Area Networks (LPWAN) are projected to support a major portion of the billions of devices forecasted for the Internet of Things (IoT). LoRaWAN is designed from the bottom up to optimize LPWANs for battery lifetime, capacity, range, and cost. LoRa and LoRaWAN permit long-range connectivity for IoT devices in different types of industries. According to Wikipedia, LoRaWAN defines the communication protocol and system architecture for the network, while the LoRa physical layer enables the long-range communication link.

AWS IoT

AWS describes AWS IoT as a set of managed services that enable ‘internet-connected devices to connect to the AWS Cloud and lets applications in the cloud interact with internet-connected devices.’ AWS IoT services span three categories: Device Software, Connectivity and Control, and Analytics.

In this post, we will focus on three AWS IOT services, one from each category, including AWS IoT Device SDKs, AWS IoT Core, and AWS IoT Analytics. According to AWS, the AWS IoT Device SDKs include open-source libraries and developer and porting guides with samples to help you build innovative IoT products or solutions on your choice of hardware platforms. AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other devices. AWS IoT Core can process and route messages to AWS endpoints and other devices reliably and securely. Finally, AWS IoT Analytics is a fully-managed IoT analytics service, designed specifically for IoT, which collects, pre-processes, enriches, stores, and analyzes IoT device data at scale.

To learn more about AWS IoT, specifically the AWS IoT services we will be exploring within this post, I recommend reading my recent post published on Towards Data Science, Getting Started with IoT Analytics on AWS.

Hardware Selection

In this post, we will use the following hardware.

IoT Device with Embedded Sensors

An Arduino single-board microcontroller will serve as our IoT device. The 3.3V AI-enabled Arduino Nano 33 BLE Sense board (Amazon: USD 36.00), released in August 2019, comes with the powerful nRF52840 processor from Nordic Semiconductors, a 32-bit ARM Cortex-M4 CPU running at 64 MHz, 1MB of CPU Flash Memory, 256KB of SRAM, and a NINA-B306 stand-alone Bluetooth 5 low energy (BLE) module.

The Sense contains an impressive array of embedded sensors:

9-axis Inertial Sensor (LSM9DS1): 3D digital linear acceleration sensor, a 3D digital
angular rate sensor, and a 3D digital magnetic sensor
Humidity and Temperature Sensor (HTS221): Capacitive digital sensor for relative humidity and temperature
Barometric Sensor (LPS22HB): MEMS nano pressure sensor: 260–1260 hectopascal (hPa) absolute digital output barometer
Microphone (MP34DT05): MEMS audio sensor omnidirectional digital microphone
Gesture, Proximity, Light Color, and Light Intensity Sensor (APDS9960): Advanced Gesture detection, Proximity detection, Digital Ambient Light Sense (ALS), and Color Sense (RGBC).

The Arduino Sense is an excellent, low-cost single-board microcontroller for learning about the collection and transmission of IoT sensor data.

IoT Gateway

An IoT Gateway, according to TechTarget, is a physical device or software program that serves as the connection point between the Cloud and controllers, sensors, and intelligent devices. All data moving to the Cloud, or vice versa, goes through the gateway, which can be either a dedicated hardware appliance or software program.

LoRa Gateways, to paraphrase The Things Network, form the bridge between devices and the Cloud. Devices use low power networks like LoRaWAN to connect to the Gateway, while the Gateway uses high bandwidth networks like WiFi, Ethernet, or Cellular to connect to the Cloud.

A third-generation Raspberry Pi 3 Model B+ single-board computer (SBC) will serve as our LoRa IoT Gateway. This Raspberry Pi model features a 1.4GHz Cortex-A53 (ARMv8) 64-bit quad-core processor System on a Chip (SoC), 1GB LPDDR2 SDRAM, dual-band wireless LAN, Bluetooth 4.2 BLE, and Gigabit Ethernet (Amazon: USD 42.99).

LoRa Transceiver Modules

To transmit the IoT sensor data between the IoT device, containing the embedded sensors, and the IoT gateway, I have used the REYAX RYLR896 LoRa transceiver module (Amazon: USD 19.50 x 2). The transceiver modules are commonly referred to as a universal asynchronous receiver-transmitter (UART). A UART is a computer hardware device for asynchronous serial communication in which the data format and transmission speeds are configurable.

According to the manufacturer, REYAX, the RYLR896 contains the Semtech SX1276 long-range, low power transceiver. The RYLR896 module provides ultra-long range spread spectrum communication and high interference immunity while minimizing current consumption. Each RYLR896 module contains a small, PCB integrated, helical antenna. This transceiver operates at both the 868 and 915 MHz frequency ranges. In this demonstration, we will be transmitting at 915 MHz for North America.

The Arduino Sense (IoT device) transmits data, using one of the RYLR896 modules (shown below front). The Raspberry Pi (IoT Gateway), connected to the other RYLR896 module (shown below rear), receives the data.

LoRaWAN Security

The RYLR896 is capable of AES 128-bit data encryption. Using the Advanced Encryption Standard (AES), we will encrypt the data sent from the IoT device to the IoT gateway, using a 32 hex digit password (128 bits / 4 bits/hex digit).

Provisioning AWS Resources

To start, we will create the necessary AWS IoT and associated resources on the AWS cloud platform. Once these resources are in place, we can then proceed to configure the IoT device and IoT gateway to securely transmit the sensor data to the Cloud.

All the source code for this post is on GitHub. Use the following command to git clone a local copy of the project.

git clone --branch master --single-branch --depth 1 --no-tags \
    https://github.com/garystafford/iot-aws-lora-demo.git

AWS CloudFormation

The CloudFormation template, iot-analytics.yaml, will create an AWS IoT CloudFormation stack containing the following resources.

AWS IoT Thing
AWS IoT Thing Policy
AWS IoT Core Topic Rule
AWS IoT Analytics Channel, Pipeline, Data store, and Data set
AWS Lambda and Lambda Permission
Amazon S3 Bucket
Amazon SageMaker Notebook Instance
AWS IAM Roles

Please be aware of the costs involved with the AWS resources used in the CloudFormation template before continuing. To create the AWS CloudFormation stack from the included CloudFormation template, execute the following AWS CLI command.

      
        aws cloudformation create-stack \
      
          –stack-name lora-iot-demo \
      
          –template-body file://cloudformation/iot-analytics.yaml \
      
          –parameters ParameterKey=ProjectName,ParameterValue=lora-iot-demo \
      
                       ParameterKey=IoTTopicName,ParameterValue=lora-iot-demo \
      
          –capabilities CAPABILITY_NAMED_IAM

view raw lora-aws-iot-create-stack.sh hosted with ❤ by GitHub

The resulting CloudFormation stack should contain 16 AWS resources.

Additional Resources

Unfortunately, AWS CloudFormation cannot create all the AWS IoT resources we require for this demonstration. To complete the AWS provisioning process, execute the following series of AWS CLI commands, aws_cli_commands.md. These commands will create the remaining resources, including an AWS IoT Thing Type, Thing Group, Thing Billing Group, and an X.509 Certificate.

      
        # LoRaWAN / AWS IoT Demo
      
        # Author: Gary Stafford
      
        # Run AWS CLI commands after CloudFormation stack completes successfully
      
        # variables
      
        thingName=lora-iot-gateway-01
      
        thingPolicy=LoRaDevicePolicy
      
        thingType=LoRaIoTGateway
      
        thingGroup=LoRaIoTGateways
      
        thingBillingGroup=LoRaIoTGateways
      
        mkdir ${thingName}
      
        aws iot create-keys-and-certificate \
      
            –certificate-pem-outfile "${thingName}/${thingName}.cert.pem" \
      
            –public-key-outfile "${thingName}/${thingName}.public.key" \
      
            –private-key-outfile "${thingName}/${thingName}.private.key" \
      
            –set-as-active
      
        # assuming you only have one certificate registered
      
        certificate=$(aws iot list-certificates | jq '.[][] | .certificateArn')
      
        ## alternately, for a specific certificate if you have more than one
      
        # aws iot list-certificates
      
        ## then change the value below
      
        # certificate=arn:aws:iot:us-east-1:123456789012:cert/<certificate>
      
        aws iot attach-policy \
      
            –policy-name $thingPolicy \
      
            –target $certificate
      
        aws iot attach-thing-principal \
      
            –thing-name $thingName \
      
            –principal $certificate
      
        aws iot create-thing-type \
      
            –thing-type-name $thingType \
      
            –thing-type-properties "thingTypeDescription=LoRaWAN IoT Gateway"
      
        aws iot create-thing-group \
      
            –thing-group-name $thingGroup \
      
            –thing-group-properties "thingGroupDescription=\"LoRaWAN IoT Gateway Thing Group\", attributePayload={attributes={Manufacturer=RaspberryPiFoundation}}"
      
        aws iot add-thing-to-thing-group \
      
            –thing-name $thingName \
      
            –thing-group-name $thingGroup
      
        aws iot create-billing-group \
      
            –billing-group-name $thingBillingGroup \
      
            –billing-group-properties "billingGroupDescription=\"Gateway Billing Group\""
      
        aws iot add-thing-to-billing-group \
      
            –thing-name $thingName \
      
            –billing-group-name $thingBillingGroup
      
        aws iot update-thing \
      
            –thing-name $thingName \
      
            –thing-type-name $thingType \
      
            –attribute-payload "{\"attributes\": {\"GatewayMfr\":\"RaspberryPiFoundation\", \"LoRaMfr\":\"REYAX\", \"LoRaModel\":\"RYLR896\"}}"
      
        aws iot describe-thing \
      
            –thing-name $thingName

view raw aws_cli_commands.sh hosted with ❤ by GitHub

IoT Device Configuration

With the AWS resources deployed, we can configure the IoT device and IoT Gateway.

Arduino Sketch

For those not familiar with Arduino, a sketch is the name that Arduino uses for a program. It is the unit of code that is uploaded into non-volatile flash memory and runs on an Arduino board. The Arduino language is a set of C and C++ functions. All standard C and C++ constructs supported by the avr-g++ compiler should work in Arduino.

For this post, the sketch, lora_iot_demo_aws.ino, contains the code necessary to collect and securely transmit the environmental sensor data, including temperature, relative humidity, barometric pressure, Red, Green, and Blue (RGB) color, and ambient light intensity, using the LoRaWAN protocol.

	/*
	Description: Transmits Arduino Nano 33 BLE Sense sensor telemetry over LoRaWAN,
	including temperature, humidity, barometric pressure, and color,
	using REYAX RYLR896 transceiver modules
	http://reyax.com/wp-content/uploads/2020/01/Lora-AT-Command-RYLR40x_RYLR89x_EN.pdf
	Author: Gary Stafford
	*/

	#include <Arduino_HTS221.h>
	#include <Arduino_LPS22HB.h>
	#include <Arduino_APDS9960.h>

	const int UPDATE_FREQUENCY = 5000; // update frequency in ms
	const float CALIBRATION_FACTOR = –4.0; // temperature calibration factor (Celsius)
	const int ADDRESS = 116;
	const int NETWORK_ID = 6;
	const String PASSWORD = "92A0ECEC9000DA0DCF0CAAB0ABA2E0EF";
	const String DELIMITER = "\|";

	String uid = "";

	void setup()
	{
	Serial.begin(9600);

	Serial1.begin(115200); // default baud rate of module is 115200
	delay(1000); // wait for LoRa module to be ready

	// get unique transceiver id to identify iot device on network
	Serial1.print((String)"AT+UID?\r\n");
	uid = Serial1.readString();
	uid.replace("+UID=", ""); // trim off '+UID=' at start of line
	uid.replace("\r\n", ""); // trim off CR/LF at end of line

	// needs all need to be same for receiver and transmitter
	Serial1.print((String)"AT+ADDRESS=" + ADDRESS + "\r\n");
	delay(200);
	Serial1.print((String)"AT+NETWORKID=" + NETWORK_ID + "\r\n");
	delay(200);
	Serial1.print("AT+CPIN=" + PASSWORD + "\r\n");
	delay(200);
	Serial1.print("AT+CPIN?\r\n"); // confirm password is set

	if (!HTS.begin())
	{ // initialize HTS221 sensor
	Serial.println("Failed to initialize humidity temperature sensor!");
	while (1);
	}

	if (!BARO.begin())
	{ // initialize LPS22HB sensor
	Serial.println("Failed to initialize pressure sensor!");
	while (1);
	}

	// avoid bad readings to start bug
	// https://forum.arduino.cc/index.php?topic=660360.0
	BARO.readPressure();
	delay(1000);

	if (!APDS.begin())
	{ // initialize APDS9960 sensor
	Serial.println("Failed to initialize color sensor!");
	while (1);
	}
	}

	void loop()
	{
	updateReadings();
	delay(UPDATE_FREQUENCY);
	}

	void updateReadings()
	{
	float temperature = getTemperature(CALIBRATION_FACTOR);
	float humidity = getHumidity();
	float pressure = getPressure();
	int colors[4];
	getColor(colors);

	String payload = buildPayload(temperature, humidity, pressure, colors);
	Serial.println("Payload: " + payload); // display the payload for debugging

	Serial1.print(payload); // send the payload over LoRaWAN WiFi

	displayResults(temperature, humidity, pressure, colors); // display the results for debugging
	}

	float getTemperature(float calibration)
	{
	return HTS.readTemperature() + calibration;
	}

	float getHumidity()
	{
	return HTS.readHumidity();
	}

	float getPressure()
	{
	return BARO.readPressure();
	}

	void getColor(int c[])
	{
	// check if a color reading is available
	while (!APDS.colorAvailable())
	{
	delay(5);
	}

	int r, g, b, a;
	APDS.readColor(r, g, b, a);
	c[0] = r;
	c[1] = g;
	c[2] = b;
	c[3] = a;
	}

	// display for debugging purposes
	void displayResults(float t, float h, float p, int c[])
	{
	Serial.println((String)"UID: " + uid);
	Serial.print("Temperature: ");
	Serial.println(t);
	Serial.print("Humidity: ");
	Serial.println(h);
	Serial.print("Pressure: ");
	Serial.println(p);
	Serial.print("Color (r, g, b, a): ");
	Serial.print(c[0]);
	Serial.print(", ");
	Serial.print(c[1]);
	Serial.print(", ");
	Serial.print(c[2]);
	Serial.print(", ");
	Serial.println(c[3]);
	Serial.println("———-");
	}

	String buildPayload(float t, float h, float p, int c[])
	{
	String readings = "";
	readings += uid;
	readings += DELIMITER;
	readings += t;
	readings += DELIMITER;
	readings += h;
	readings += DELIMITER;
	readings += p;
	readings += DELIMITER;
	readings += c[0];
	readings += DELIMITER;
	readings += c[1];
	readings += DELIMITER;
	readings += c[2];
	readings += DELIMITER;
	readings += c[3];

	String payload = "";
	payload += "AT+SEND=";
	payload += ADDRESS;
	payload += ",";
	payload += readings.length();
	payload += ",";
	payload += readings;
	payload += "\r\n";

	return payload;
	}

view raw lora_iot_demo_aws.ino hosted with ❤ by GitHub

AT Commands

Communications with the RYLR896’s long-range modem is done using AT commands. AT commands are instructions used to control a modem. AT is the abbreviation of ATtention. Every command line starts with AT. That is why modem commands are called AT commands, according to Developer’s Home. A complete list of AT commands can be downloaded as a PDF from the RYLR896 product page.

To efficiently transmit the environmental sensor data from the IoT sensor to the IoT gateway, the sketch concatenates the sensor ID and the sensor values together in a single string. The string will be incorporated into an AT command, sent to the RYLR896 LoRa transceiver module. To make it easier to parse the sensor data on the IoT gateway, we will delimit the sensor values with a pipe (|), as opposed to a comma. According to REYAX, the maximum length of the LoRa payload is approximately 330 bytes.

Below, we see an example of an AT command used to send the sensor data from the IoT sensor and the corresponding unencrypted data received by the IoT gateway. Both contain the LoRa transmitter Address ID, payload length (62 bytes in the example), and the payload. The data received by the IoT gateway also has the Received signal strength indicator (RSSI), and Signal-to-noise ratio (SNR).

Receiving Data on IoT Gateway

The Raspberry Pi will act as a LoRa IoT gateway, receiving the environmental sensor data from the IoT device, the Arduino, and sending the data to AWS. The Raspberry Pi runs a Python script, rasppi_lora_receiver_aws.py, which will receive the data from the Arduino Sense, decrypt the data, parse the sensor values, and serialize the data to a JSON payload, and finally, transmit the payload in an MQTT-protocol message to AWS. The script uses the pyserial, the Python Serial Port Extension, which encapsulates the access for the serial port for communication with the RYLR896 module. The script uses the AWS IoT Device SDK for Python v2 to communicate with AWS.

	import json
	import logging
	import sys
	import threading
	import time
	from argparse import ArgumentParser

	import serial
	from awscrt import io, mqtt, auth, http, exceptions
	from awsiot import mqtt_connection_builder

	# LoRaWAN IoT Sensor Demo
	# Using REYAX RYLR896 transceiver modules
	# http://reyax.com/wp-content/uploads/2020/01/Lora-AT-Command-RYLR40x_RYLR89x_EN.pdf
	# Author: Gary Stafford
	# Requirements: python3 -m pip install –user -r requirements.txt
	# Usage:
	# sh ./rasppi_lora_receiver_aws.sh \
	# b1d0fncxn1hf3m-ats.iot.us-east-1.amazonaws.com


	# constants
	ADDRESS = 116
	NETWORK_ID = 6
	PASSWORD = "92A0ECEC9000DA0DCF0CAAB0ABA2E0EF"

	# global variables
	count = 0 # from args
	received_count = 0
	received_all_event = threading.Event()


	def main():
	# get args
	logging.basicConfig(filename='output.log',
	filemode='w', level=logging.DEBUG)
	args = get_args() # get args
	payload = ""
	lora_payload = {}

	# set log level
	io.init_logging(getattr(io.LogLevel, args.verbosity), 'stderr')

	# spin up resources
	event_loop_group = io.EventLoopGroup(1)
	host_resolver = io.DefaultHostResolver(event_loop_group)
	client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)

	# set MQTT connection
	mqtt_connection = set_mqtt_connection(args, client_bootstrap)

	logging.debug("Connecting to {} with client ID '{}'…".format(
	args.endpoint, args.client_id))

	connect_future = mqtt_connection.connect()

	# future.result() waits until a result is available
	connect_future.result()

	logging.debug("Connecting to REYAX RYLR896 transceiver module…")
	serial_conn = serial.Serial(
	port=args.tty,
	baudrate=int(args.baud_rate),
	timeout=5,
	parity=serial.PARITY_NONE,
	stopbits=serial.STOPBITS_ONE,
	bytesize=serial.EIGHTBITS
	)

	if serial_conn.isOpen():
	logging.debug("Connected!")
	set_lora_config(serial_conn)
	check_lora_config(serial_conn)

	while True:
	# read data from serial port
	serial_payload = serial_conn.readline()
	logging.debug(serial_payload)

	if len(serial_payload) >= 1:
	payload = serial_payload.decode(encoding="utf-8")
	payload = payload[:–2]
	try:
	data = parse_payload(payload)
	lora_payload = {
	"ts": time.time(),
	"data": {
	"device_id": str(data[0]),
	"gateway_id": str(args.gateway_id),
	"temperature": float(data[1]),
	"humidity": float(data[2]),
	"pressure": float(data[3]),
	"color": {
	"red": float(data[4]),
	"green": float(data[5]),
	"blue": float(data[6]),
	"ambient": float(data[7])
	}
	}
	}
	logging.debug(lora_payload)
	except IndexError:
	logging.error("IndexError: {}".format(payload))
	except ValueError:
	logging.error("ValueError: {}".format(payload))

	# publish mqtt message
	message_json = json.dumps(
	lora_payload,
	sort_keys=True,
	indent=None,
	separators=(',', ':'))

	try:
	mqtt_connection.publish(
	topic=args.topic,
	payload=message_json,
	qos=mqtt.QoS.AT_LEAST_ONCE)
	except mqtt.SubscribeError as err:
	logging.error(".SubscribeError: {}".format(err))
	except exceptions.AwsCrtError as err:
	logging.error("AwsCrtError: {}".format(err))


	def set_mqtt_connection(args, client_bootstrap):
	if args.use_websocket:
	proxy_options = None
	if args.proxy_host:
	proxy_options = http.HttpProxyOptions(
	host_name=args.proxy_host, port=args.proxy_port)

	credentials_provider = auth.AwsCredentialsProvider.new_default_chain(
	client_bootstrap)
	mqtt_connection = mqtt_connection_builder.websockets_with_default_aws_signing(
	endpoint=args.endpoint,
	client_bootstrap=client_bootstrap,
	region=args.signing_region,
	credentials_provider=credentials_provider,
	websocket_proxy_options=proxy_options,
	ca_filepath=args.root_ca,
	on_connection_interrupted=on_connection_interrupted,
	on_connection_resumed=on_connection_resumed,
	client_id=args.client_id,
	clean_session=False,
	keep_alive_secs=6)

	else:
	mqtt_connection = mqtt_connection_builder.mtls_from_path(
	endpoint=args.endpoint,
	cert_filepath=args.cert,
	pri_key_filepath=args.key,
	client_bootstrap=client_bootstrap,
	ca_filepath=args.root_ca,
	on_connection_interrupted=on_connection_interrupted,
	on_connection_resumed=on_connection_resumed,
	client_id=args.client_id,
	clean_session=False,
	keep_alive_secs=6)

	return mqtt_connection


	def get_args():
	parser = ArgumentParser(
	description="Send and receive messages through and MQTT connection.")
	parser.add_argument("–tty", required=True,
	help="serial tty", default="/dev/ttyAMA0")
	parser.add_argument("–baud-rate", required=True,
	help="serial baud rate", default=1152000)
	parser.add_argument('–endpoint', required=True, help="Your AWS IoT custom endpoint, not including a port. " +
	"Ex: \"abcd123456wxyz-ats.iot.us-east-1.amazonaws.com\"")
	parser.add_argument('–cert', help="File path to your client certificate, in PEM format.")
	parser.add_argument('–key', help="File path to your private key, in PEM format.")
	parser.add_argument('–root-ca', help="File path to root certificate authority, in PEM format. " +
	"Necessary if MQTT server uses a certificate that's not already in " +
	"your trust store.")
	parser.add_argument('–client-id', default='samples-client-id',
	help="Client ID for MQTT connection.")
	parser.add_argument('–topic', default="samples/test",
	help="Topic to subscribe to, and publish messages to.")
	parser.add_argument('–message', default="Hello World!", help="Message to publish. " +
	"Specify empty string to publish nothing.")
	parser.add_argument('–count', default=0, type=int, help="Number of messages to publish/receive before exiting. " +
	"Specify 0 to run forever.")
	parser.add_argument('–use-websocket', default=False, action='store_true',
	help="To use a websocket instead of raw mqtt. If you specify this option you must "
	"specify a region for signing, you can also enable proxy mode.")
	parser.add_argument('–signing-region', default='us-east-1',
	help="If you specify –use-web-socket, this is the region that will be used for computing "
	"the Sigv4 signature")
	parser.add_argument('–proxy-host', help="Hostname for proxy to connect to. Note: if you use this feature, " +
	"you will likely need to set –root-ca to the ca for your proxy.")
	parser.add_argument('–proxy-port', type=int, default=8080,
	help="Port for proxy to connect to.")
	parser.add_argument('–verbosity', choices=[x.name for x in io.LogLevel], default=io.LogLevel.NoLogs.name,
	help='Logging level')
	parser.add_argument("–gateway-id", help="IoT Gateway serial number")
	args = parser.parse_args()
	return args


	def parse_payload(payload):
	# input: +RCV=116,29,0447383033363932003C0034\|23.94\|37.71\|99.89\|16\|38\|53\|80,-61,56
	# output: [0447383033363932003C0034, 23.94, 37.71, 99.89, 16.0, 38.0, 53.0, 80.0]

	payload = payload.split(",")
	payload = payload[2].split("\|")
	payload = [i for i in payload]
	return payload


	def set_lora_config(serial_conn):
	# configures the REYAX RYLR896 transceiver module

	serial_conn.write(str.encode("AT+ADDRESS=" + str(ADDRESS) + "\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Address set? {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+NETWORKID=" + str(NETWORK_ID) + "\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Network Id set? {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+CPIN=" + PASSWORD + "\r\n"))
	time.sleep(1)
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("AES-128 password set? {}".format(serial_payload.decode(encoding="utf-8")))


	def check_lora_config(serial_conn):
	serial_conn.write(str.encode("AT?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Module responding? {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+ADDRESS?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Address: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+NETWORKID?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Network id: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+IPR?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("UART baud rate: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+BAND?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("RF frequency: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+CRFOP?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("RF output power: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+MODE?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("Work mode: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+PARAMETER?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("RF parameters: {}".format(serial_payload.decode(encoding="utf-8")))

	serial_conn.write(str.encode("AT+CPIN?\r\n"))
	serial_payload = (serial_conn.readline())[:–2]
	logging.debug("AES128 password of the network: {}".format(serial_payload.decode(encoding="utf-8")))


	# Callback when connection is accidentally lost.
	def on_connection_interrupted(connection, error, **kwargs):
	logging.error("Connection interrupted. error: {}".format(error))


	# Callback when an interrupted connection is re-established.
	def on_connection_resumed(connection, return_code, session_present, **kwargs):
	logging.warning("Connection resumed. return_code: {} session_present: {}".format(
	return_code, session_present))

	if return_code == mqtt.ConnectReturnCode.ACCEPTED and not session_present:
	logging.warning("Session did not persist. Resubscribing to existing topics…")
	resubscribe_future, _ = connection.resubscribe_existing_topics()

	# Cannot synchronously wait for resubscribe result because we're on the connection's event-loop thread,
	# evaluate result with a callback instead.
	resubscribe_future.add_done_callback(on_resubscribe_complete)


	def on_resubscribe_complete(resubscribe_future):
	resubscribe_results = resubscribe_future.result()
	logging.warning("Resubscribe results: {}".format(resubscribe_results))

	for topic, qos in resubscribe_results['topics']:
	if qos is None:
	sys.exit("Server rejected resubscribe to topic: {}".format(topic))


	# Callback when the subscribed topic receives a message
	def on_message_received(topic, payload, **kwargs):
	logging.debug("Received message from topic '{}': {}".format(topic, payload))
	global received_count
	received_count += 1
	if received_count == count:
	received_all_event.set()


	if __name__ == "__main__":
	sys.exit(main())

view raw rasppi_lora_receiver_aws.py hosted with ❤ by GitHub

Running the IoT Gateway Python Script

To run the Python script on the Raspberry Pi, we will use a helper shell script, rasppi_lora_receiver_aws.sh. The shell script helps construct the arguments required to execute the Python script.

      
        #!/bin/bash
      
        # LoRaWAN / AWS IoT Demo
      
        # Author: Gary A. Stafford
      
        # Revised: 2021-03-24
      
        # Start IoT data collector script and tails output
      
        # Usage:
      
        # sh ./rasppi_lora_receiver_aws.sh a1b2c3d4e5678f-ats.iot.us-east-1.amazonaws.com
      
        if [[ $# -ne 1 ]]; then
      
          echo "Script requires 1 parameter!"
      
          exit 1
      
        fi
      
        # input parameters
      
        ENDPOINT=$1  # e.g. a1b2c3d4e5678f-ats.iot.us-east-1.amazonaws.com
      
        DEVICE="lora-iot-gateway-01" # matches CloudFormation thing name
      
        CERTIFICATE="${DEVICE}-cert.pem"  # e.g. lora-iot-gateway-01-cert.pem
      
        KEY="${DEVICE}-private.key"  # e.g. lora-iot-gateway-01-private.key
      
        GATEWAY_ID=$(< /proc/cpuinfo grep Serial | grep -oh "[a-z0-9]*$") # e.g. 00000000f62051ce
      
        # output for debugging
      
        echo "DEVICE: ${DEVICE}"
      
        echo "ENDPOINT: ${ENDPOINT}"
      
        echo "CERTIFICATE: ${CERTIFICATE}"
      
        echo "KEY: ${KEY}"
      
        echo "GATEWAY_ID: ${GATEWAY_ID}"
      
        # call the python script
      
        nohup python3 rasppi_lora_receiver_aws.py \
      
          –endpoint "${ENDPOINT}" \
      
          –cert "${DEVICE}-creds/${CERTIFICATE}" \
      
          –key "${DEVICE}-creds/${KEY}" \
      
          –root-ca "${DEVICE}-creds/AmazonRootCA1.pem" \
      
          –client-id "${DEVICE}" \
      
          –topic "lora-iot-demo" \
      
          –gateway-id "${GATEWAY_ID}" \
      
          –verbosity "Info" \
      
          –tty "/dev/ttyAMA0" \
      
          –baud-rate 115200 \
      
          >collector.log 2>&1 </dev/null &
      
        sleep 2
      
        # tail the log (Control-C to exit)
      
        tail -f collector.log

view raw rasppi_lora_receiver_aws.sh hosted with ❤ by GitHub

To run the helper script, we execute the following command, substituting the input parameter, the AWS IoT endpoint, with your endpoint.

sh ./rasppi_lora_receiver_aws.sh \
  a1b2c3d4e5678f-ats.iot.us-east-1.amazonaws.com

You should see the console output, similar to the following.

The script starts by configuring the RYLR896 module and outputting that configuration to a log file, output.log. If successful, we should see the following debug information logged.

DEBUG:root:Connecting to a1b2c3d4e5f6-ats.iot.us-east-1.amazonaws.com with client ID 'lora-iot-gateway-01'
DEBUG:root:Connecting to REYAX RYLR896 transceiver module
DEBUG:root:Connected!
DEBUG:root:Address set? +OK
DEBUG:root:Network Id set? +OK
DEBUG:root:AES-128 password set? +OK
DEBUG:root:Module responding? +OK
DEBUG:root:Address: +ADDRESS=116
DEBUG:root:Network id: +NETWORKID=6
DEBUG:root:UART baud rate: +IPR=115200
DEBUG:root:RF frequency: +BAND=915000000
DEBUG:root:RF output power: +CRFOP=15
DEBUG:root:Work mode: +MODE=0
DEBUG:root:RF parameters: +PARAMETER=12,7,1,4
DEBUG:root:AES128 password of the network: +CPIN=92A0ECEC9000DA0DCF0CAAB0ABA2E0EF

That sensor data is also written to the log file for debugging purposes. This first line in the log (shown below) is the raw decrypted data received from the IoT device via LoRaWAN. The second line is the JSON-serialized payload, sent securely to AWS, using the MQTT protocol.

DEBUG:root:b'+RCV=116,59,0447383033363932003C0034|23.46|41.89|99.38|230|692|833|1116,-48,39\r\n'

DEBUG:root:{'ts': 1598305503.7041512, 'data': {'humidity': 41.89, 'temperature': 23.46, 'device_id': '0447383033363932003C0034', 'gateway_id': '00000000f62051ce', 'pressure': 99.38, 'color': {'red': 230.0, 'blue': 833.0, 'ambient': 1116.0, 'green': 692.0}}}

DEBUG:root:b'+RCV=116,59,0447383033363932003C0034|23.46|41.63|99.38|236|696|837|1127,-49,35\r\n'

DEBUG:root:{'ts': 1598305513.7918658, 'data': {'humidity': 41.63, 'temperature': 23.46, 'device_id': '0447383033363932003C0034', 'gateway_id': '00000000f62051ce', 'pressure': 99.38, 'color': {'red': 236.0, 'blue': 837.0, 'ambient': 1127.0, 'green': 696.0}}}

DEBUG:root:b'+RCV=116,59,0447383033363932003C0034|23.44|41.57|99.38|232|686|830|1113,-48,32\r\n'

DEBUG:root:{'ts': 1598305523.8556132, 'data': {'humidity': 41.57, 'temperature': 23.44, 'device_id': '0447383033363932003C0034', 'gateway_id': '00000000f62051ce', 'pressure': 99.38, 'color': {'red': 232.0, 'blue': 830.0, 'ambient': 1113.0, 'green': 686.0}}}

DEBUG:root:b'+RCV=116,59,0447383033363932003C0034|23.51|41.44|99.38|205|658|802|1040,-48,36\r\n'

DEBUG:root:{'ts': 1598305528.8890748, 'data': {'humidity': 41.44, 'temperature': 23.51, 'device_id': '0447383033363932003C0034', 'gateway_id': '00000000f62051ce', 'pressure': 99.38, 'color': {'red': 205.0, 'blue': 802.0, 'ambient': 1040.0, 'green': 658.0}}}

AWS IoT Core

The Raspberry Pi-based IoT gateway will be registered with AWS IoT Core. IoT Core allows users to connect devices quickly and securely to AWS.

Things

According to AWS, IoT Core can reliably scale to billions of devices and trillions of messages. Registered devices are referred to as things in AWS IoT Core. A thing is a representation of a specific device or logical entity. Information about a thing is stored in the registry as JSON data.

Below, we see an example of the Thing created by CloudFormation. The Thing, lora-iot-gateway-01, represents the physical IoT gateway. We have assigned the IoT gateway a Thing Type, LoRaIoTGateway, a Thing Group, LoRaIoTGateways, and a Thing Billing Group, IoTGateways.

In a real IoT environment, containing hundreds, thousands, even millions of IoT devices, gateways, and sensors, these classification mechanisms, Thing Type, Thing Group, and Thing Billing Group, will help to organize IoT assets.

Device Gateway and Message Broker

IoT Core provides a Device Gateway, which manages all active device connections. The Gateway currently supports MQTT, WebSockets, and HTTP 1.1 protocols. Behind the Message Gateway is a high-throughput pub/sub Message Broker, which securely transmits messages to and from all IoT devices and applications with low latency. Below, we see a typical AWS IoT Core architecture containing multiple Topics, Rules, and Actions.

AWS IoT Security

AWS IoT Core provides mutual authentication and encryption, ensuring all data is exchanged between AWS and the devices are secure by default. In the demonstration, all data is sent securely using Transport Layer Security (TLS) 1.2 with X.509 digital certificates on port 443. Below, we see an example of an X.509 certificate assigned to the Thing, lora-iot-gateway-01, which represents the physical IoT gateway. The X.509 certificate and the private key, generated using the AWS CLI, previously, are installed on the IoT gateway.

Authorization of the device to access any resource on AWS is controlled by AWS IoT Core Policies. These policies are similar to AWS IAM Policies. Below, we see an example of an AWS IoT Core Policy, LoRaDevicePolicy, which is assigned to the IoT gateway.

AWS IoT Core Rules

Once an MQTT message is received from the IoT gateway (a thing), we use AWS IoT Rules to send message data to an AWS IoT Analytics Channel. Rules give your devices the ability to interact with AWS services. Rules are analyzed, and Actions are performed based on the MQTT topic stream. Below, we see an example rule that forwards our messages to an IoT Analytics Channel.

Rule query statements are written in standard Structured Query Language (SQL). The datasource for the Rule query is an IoT Topic.

      
        SELECT
      
            data.device_id,
      
            data.gateway_id,
      
            data.temperature,
      
            data.humidity,
      
            data.pressure,
      
            data.color.red,
      
            data.color.green,
      
            data.color.blue,
      
            data.color.ambient,
      
            ts,
      
            Clientid () AS device,
      
            parse_time ("yyyy-MM-dd'T'HH:mm:ss.SSSZ", timestamp(), "UTC") AS msg_received
      
        FROM
      
            "${IoTTopicName}"

view raw iot_rule.sql hosted with ❤ by GitHub

AWS IoT Analytics

AWS IoT Analytics is composed of five primary components: Channels, Pipelines, Data stores, Data sets, and Notebooks. These components enable you to collect, prepare, store, analyze, and visualize your IoT data.

Below, we see a typical AWS IoT Analytics architecture. IoT messages are received from AWS IoT Core, thought a Rule Action. Amazon QuickSight provides business intelligence, visualization. Amazon QuickSight ML Insights adds anomaly detection and forecasting.

IoT Analytics Channel

An AWS IoT Analytics Channel pulls messages or data into IoT Analytics from other AWS sources, such as Amazon S3, Amazon Kinesis, or Amazon IoT Core. Channels store data for IoT Analytics Pipelines. Both Channels and Data store support storing data in your own Amazon S3 bucket or an IoT Analytics service-managed S3 bucket. In the demonstration, we are using a service managed S3 bucket.

When creating a Channel, you also decide how long to retain the data. For the demonstration, we have set the data retention period for 21 days. Channels are generally not used for long term storage of data. Typically, you would only retain data in the Channel for the period you need to analyze. For long term storage of IoT message data, I recommend using an AWS IoT Core Rule to send a copy of the raw IoT data to Amazon S3, using a service such as Amazon Kinesis Data Firehose.

IoT Analytics Pipeline

An AWS IoT Analytics Pipeline consumes messages from one or more Channels. Pipelines transform, filter, and enrich the messages before storing them in IoT Analytics Data stores. A Pipeline is composed of an ordered list of activities. Logically, you must specify both a Channel (source) and a Datastore (destination) activity. Optionally, you may choose as many as 23 additional activities in the pipelineActivities array.

In our demonstration’s Pipeline, iot_analytics_pipeline, we have specified three additional activities, including DeviceRegistryEnrich, Filter, and Lambda. Other activity types include Math, SelectAttributes, RemoveAttributes, and AddAttributes.

The Filter activity ensures the sensor values are not Null or otherwise erroneous; if true, the message is dropped. The Lambda Pipeline activity executes an AWS Lambda function to transform the messages in the pipeline. Messages are sent in an event object to the Lambda. The message is modified, and the event object is returned to the activity.

The Python-based Lambda function easily handles typical IoT data transformation tasks, including converting the temperature from Celsius to Fahrenheit, pressure from kilopascals (kPa) to inches of Mercury (inHg), and 12-bit RGBA values to 8-bit color values (0–255). The Lambda function also rounds down all the values to between 0 and 2 decimal places of precision.

      
        def lambda_handler(event, context):
      
            for e in event:
      
                e['temperature'] = round((e['temperature'] * 1.8) + 32, 2)
      
                e['humidity'] = round(e['humidity'], 2)
      
                e['pressure'] = round((e['pressure'] / 3.3864), 2)
      
                e['red'] = int(round(e['red'] / (4097 / 255), 0))
      
                e['green'] = int(round(e['green'] / (4097 / 255), 0))
      
                e['blue'] = int(round(e['blue'] / (4097 / 255), 0))
      
                e['ambient'] = int(round(e['ambient'] / (4097 / 255), 0))
      
            return event

view raw PipelineLambdaFunction.py hosted with ❤ by GitHub

The demonstration’s Pipeline also enriches the IoT data with metadata from the IoT device’s AWS IoT Core Registry. The metadata includes additional information about the device that generated the IoT data, including the custom attributes such as LoRa transceiver manufacturer and model, and the IoT gateway manufacturer.

A notable feature of Pipelines is the ability to reprocess messages. If you make changes to the Pipeline, which often happens during the data preparation stage, you can reprocess any or all the IoT data in the associated Channel, and overwrite the IoT data in the Data set.

IoT Analytics Data store

An AWS IoT Analytics Data store stores prepared data from an AWS IoT Analytics Pipeline, in a fully-managed database. Both Channels and Data store support storing IoT data in your own Amazon S3 bucket or an IoT Analytics managed S3 bucket. In the demonstration, we are using a service-managed S3 bucket to store the IoT data in our Data store, iot_analytics_data_store.

IoT Analytics Data set

An AWS IoT Analytics Data set automatically provides regular, up-to-date insights for data analysts by querying a Data store using standard SQL. Periodic updates are implemented using a cron expression. For the demonstration, we are updating our Data set, iot_analytics_data_set, at a 15-minute interval. The time interval can be increased or reduced, depending on the desired ‘near real-time’ nature of the IoT data being analyzed.

Below, we see messages in the Result preview pane of the Data set. Note the SQL query used to obtain the messages, which queries the Data store. The Data store, as you will recall, contains the transformed messages from the Pipeline.

IoT Analytics Data sets also support sending content results, which are materialized views of your IoT Analytics data, to an Amazon S3 bucket.

The CloudFormation stack created an encrypted Amazon S3 Bucket. This bucket receives a copy of the messages from the IoT Analytics Data set whenever the cron expression runs the scheduled update.

IoT Analytics Notebook

An AWS IoT Analytics Notebook allows users to perform statistical analysis and machine learning on IoT Analytics Data sets using Jupyter Notebooks. The IoT Analytics Notebook service includes a set of notebook templates that contain AWS-authored machine learning models and visualizations. Notebook Instances can be linked to a GitHub or other source code repository. Notebooks created with IoT Analytics Notebook can also be accessed directly through Amazon SageMaker. For the demonstration, the Notebooks Instance is cloned from our project’s GitHub repository.

The repository contains a sample Jupyter Notebook, LoRa_IoT_Analytics_Demo.ipynb, based on the conda_python3 kernel. This preinstalled environment includes the default Anaconda installation and Python 3.

The Notebook uses pandas, matplotlib, and plotly to manipulate and visualize the sample IoT data stored in the IoT Analytics Data set.

The Notebook can be modified, and the changes pushed back to GitHub. You could easily fork the demonstration’s GitHub repository and modify the CloudFormation template to point to your source code repository.

Amazon QuickSight

Amazon QuickSight provides business intelligence (BI) and visualization. Amazon QuickSight ML Insights adds anomaly detection and forecasting. We can use Amazon QuickSight to visualize the IoT message data, stored in the IoT Analytics Data set.

Amazon QuickSight has both a Standard and an Enterprise Edition. AWS provides a detailed product comparison of each edition. For the post, I am demonstrating the Enterprise Edition, which includes additional features, such as ML Insights, hourly refreshes of SPICE (super-fast, parallel, in-memory, calculation engine), and theme customization.

Please be aware of the costs of Amazon QuickSight if you choose to follow along with this part of the demo. Although there is an Amazon QuickSight API, Amazon QuickSight is not automatically enabled or configured with CloudFormation or using the AWS CLI in this demonstration.

QuickSight Data Sets

Amazon QuickSight has a wide variety of data source options for creating Amazon QuickSight Data sets, including the ones shown below. Do not confuse Amazon QuickSight Data sets with IoT Analytics Data sets; they are two different service features.

For the demonstration, we will create an Amazon QuickSight Data set that will use our IoT Analytics Data set, iot_analytics_data_set.

Amazon QuickSight gives you the ability to view and modify QuickSight Data sets before visualizing. QuickSight even provides a wide variety of functions, enabling us to perform dynamic calculations on the field values. For this demonstration, we will leave the data unchanged since all transformations were already completed in the IoT Analytics Pipeline.

QuickSight Analysis

Using the QuickSight Data set, built from the IoT Analytics Data set as a data source, we create a QuickSight Analysis. The QuickSight Analysis console is shown below. An Analysis is primarily a collection of Visuals (aka Visual types). QuickSight provides several Visual types. Each visual is associated with a Data set. Data for the QuickSight Analysis or each visual within the Analysis can be filtered. For the demo, I have created a simple QuickSight Analysis, including a few typical QuickSight visuals.

QuickSight Dashboards

To share a QuickSight Analysis, we can create a QuickSight Dashboard. Below, we see a few views of the QuickSight Analysis, shown above, as a Dashboard. Although viewers of the Dashboard cannot edit the visuals, they can apply filtering and interactively drill-down into data in the Visuals.

Amazon QuickSight ML Insights

According to Amazon, ML Insights leverages AWS’s machine learning (ML) and natural language capabilities to gain deeper insights from data. QuickSight’s ML-powered Anomaly Detection continuously analyze data to discover anomalies and variations inside of the aggregates, giving you the insights to act when business changes occur. QuickSight’s ML-powered Forecasting can be used to predict your business metrics accurately, and perform interactive what-if analysis with point-and-click simplicity. QuickSight’s built-in algorithms make it easy for anyone to use ML that learns from your data patterns to provide you with accurate predictions based on historical trends.

Below, we see the ML Insights tab (left) in the demonstration’s QuickSight Analysis. Individually detected anomalies can be added to the QuickSight Analysis, like Visuals, and configured to tune the detection parameters. Observe the temperature, humidity, and barometric pressure anomalies, identified by ML Insights, based on their Anomaly Score, which is higher or lower, given a minimum delta of five percent. These anomalies accurately reflected an actual failure of the IoT device, caused by overheated during testing, which resulted in abnormal sensor readings.

Receiving the Messages on AWS

To confirm the IoT gateway is sending messages, we can use a packet analyzer, like tcpdump, on the IoT gateway. Running tcpdump on the IoT gateway, below, we see outbound encrypted MQTT messages being sent to AWS on port 443.

To confirm those messages are being received from the IoT gateway on AWS, we can use the AWS IoT Core Test feature and subscribe to the lora-iot-demo topic. We should see messages flowing in from the IoT gateway at approximately 5-second intervals.

The JSON payload structure of the incoming MQTT messages will look similar to the below example. The device_id is the unique id of the IoT device that transmitted the message using LoRaWAN. The gateway_id is the unique id of the IoT gateway that received the message using LoRaWAN and sent it to AWS. A single IoT gateway would usually manage messages from multiple IoT devices, each with a unique id.

      
        {
      
            "data": {
      
                "color": {
      
                    "ambient": 1057,
      
                    "blue": 650,
      
                    "green": 667,
      
                    "red": 281
      
                },
      
                "device_id": "0447383033363932003C0034",
      
                "gateway_id": "00000000f62051ce",
      
                "humidity": 45.73,
      
                "pressure": 98.65,
      
                "temperature": 23.6
      
            },
      
            "ts": 1598543131.9861386
      
        }

view raw sample_message_original.json hosted with ❤ by GitHub

The SQL query used by the AWS IoT Rule described earlier, transforms and flattens the nested JSON payload structure, before passing it to the AWS IoT Analytics Channel, as shown below.

      
        {
      
            "ambient": 1057,
      
            "blue": 650,
      
            "green": 667,
      
            "red": 281,
      
            "device_id": "0447383033363932003C0034",
      
            "gateway_id": "00000000f62051ce",
      
            "humidity": 45.73,
      
            "pressure": 98.65,
      
            "temperature": 23.6,
      
            "ts": 1598543131.9861386,
      
            "msg_received": "2020-08-27T11:45:32.074+0000",
      
            "device": "lora-iot-gateway-01"
      
        }

view raw sample_message_rule_result.json hosted with ❤ by GitHub

We can measure the near real-time nature of the IoT data using the ts and msg_received data fields. The ts data field is date and time when the sensor reading occurred on the IoT device, while the msg_received data field is the date and time when the message was received on AWS. The delta between the two values is a measure of how near real-time the sensor readings are being streamed to the AWS IoT Analytics Channel. In the below example, the difference between ts (2020–08–27T11:45:31.986) and msg_received (2020–08–27T11:45:32.074) is 88 ms.

Final IoT Data Message Structure

Once the message payload passes through the AWS IoT Analytics Pipeline and lands in the AWS IoT Analytics Data set, its final data structure looks as follows. Note that the device’s attribute metadata has been added from the AWS IoT Core device registry. Regrettably, the metadata is not well-formatted JSON and will require additional transformation to be usable.

      
        {
      
        	"device_id": "0447383033363932003C0034",
      
        	"gateway_id": "00000000f62051ce",
      
        	"temperature": 74.48,
      
        	"humidity": 45.73,
      
        	"pressure": 29.13,
      
        	"red": 17,
      
        	"green": 42,
      
        	"blue": 40,
      
        	"ambient": 66,
      
        	"ts": 1598543131.9861386,
      
        	"device": "lora-iot-gateway-01",
      
        	"msg_received": "2020-08-27T15:45:32.024+0000",
      
        	"metadata": {
      
        		"defaultclientid": "lora-iot-gateway-01",
      
        		"thingname": "lora-iot-gateway-01",
      
        		"thingid": "017db4b8-7fca-4617-aa58-7125dd94ab36",
      
        		"thingarn": "arn:aws:iot:us-east-1:123456789012:thing/lora-iot-gateway-01",
      
        		"thingtypename": "LoRaIoTGateway",
      
        		"attributes": {
      
        			"loramfr": "REYAX",
      
        			"gatewaymfr": "RaspberryPiFoundation",
      
        			"loramodel": "RYLR896"
      
        		},
      
        		"version": "2",
      
        		"billinggroupname": "LoRaIoTGateways"
      
        	},
      
        	"__dt": "2020-08-27 00:00:00.000"
      
        }

view raw sample_message_final.json hosted with ❤ by GitHub

A set of sample messages is included in the GitHub project’s sample_messages directory.

Conclusion

In this post, we explored the use of the LoRa and LoRaWAN protocols to transmit environmental sensor data from an IoT device to an IoT gateway. Given its low energy consumption, long-distance transmission capabilities, and well-developed protocols, LoRaWAN is an ideal long-range wireless protocol for IoT devices. We then demonstrated how to use AWS IoT Device SDKs, AWS IoT Core, and AWS IoT Analytics to securely collect, analyze, and visualize streaming messages from the IoT device, in near real-time.

This blog represents my own viewpoints and not of my employer, Amazon Web Services.

AWS, AWS IoT, AWS IoT Analytics, IIoT, IoT, LoRa, LoRaWAN

This entry was posted on August 26, 2020, 12:00 pm and is filed under AWS, Bash Scripting, IoT, Python, Raspberry Pi, Serverless, Software Development. You can follow any responses to this entry through RSS 2.0. You can leave a response, or trackback from your own site.

Collecting and Analyzing IoT Data in Near Real-Time with AWS IoT, LoRa, and LoRaWAN - Cas Hoefman

Programmatic Ponderings