IoT Messages and
Storage
Troy Merritt 2025
Unit Co-ordinator Week 6
1
Recap: IoT Architecture
2
Webservers from and IoT Perspective
3
Messaging
4
What are Middleware/Carrier Languages?
Messages between IoT devices that carry data.
These messages are not instructions that a
compiler will execute.
Used to represent states or conditions.
Used for all levels of IoT communication.
Things to middleware.
Middleware to cloud.
Cloud to user output.
Cloud back to things.
5
Components of carrier languages
Identity of devices:
1000s of devices in a IoT system
Each one needs to have a different identification
mechanism
Sensor/Actuator data:
Sensing information
Decision information
Metadata:
Time
Geo Locations
Application Details
6
Best Practices for Carrier Languages
Keep it small.
Especially for the Things to Middleware.
Keep it with descriptions
Especially for the Middleware to Cloud and Cloud to
User related outputs
Keep a mapping between the things’ components
with the storage all throughout the IoT system
Easy to identify things
recordtemp.php?t=%d&h=%d&ts=%d
Easy to scale up
Easy for maintenance
7
Data Storage in IoT Systems
IoT End node devices have very limited storage
(often only a few KBs) and that data is lost when
the device resets.
Cloud services are used to store large volumes of
data from many devices for big data analysis.
The cloud application has to maintain the data.
i.e. establish the relationship between the data
and devices and the applications.
8
IoT Data exchange
9
IoT Data exchange
10
Storage
11
On Device Storage
On device storage could be short term (data being
stored before being packaged and deleted) or long term
(logging)
EEPROM
SD Card
Reads and writes cost power.
But so does data transmission, so this has to be
balanced.
12
Middleware Storage
Middleware devices are all about communication.
Communication settings
Routing information
Managing device identities
Storage on these devices tends to be minimal
Storage is also fairly volatile
Data is quickly deleted once it’s been forwarded.
13
Cloud Server
Data can be stored in databases
Relational (MySQL)
Non-relational (MongoDB)
Duplicate data across multiple instances.
Implement cloud and database related security issues
Encryption
Timestamp everything
SQL injections
14
Text based
communication
15
Data structure
When data is being exchanged between devices, the
data must be structured.
The sender needs to send everything the receiver
needs.
The receiver needs to understand what data is being
received so that it knows what to do with it.
Common structures include CSV, XML, JSON.
For end node devices, the data aims to be as small as
possible, sometimes packaged down to the byte.
16
Comma / Tab Separated values
Minimal data structure.
More data, less metadata.
Easy for IoT devices to parse.
Difficult for humans to parse.
Structure needs a key to explain.
Example message:
100,25.57,on,HIGH
17
XML
Highly structured data
Lots of metadata relative to data.
larger files
Easy for devices to parse, but requires more
memory/storage.
Easy for humans to parse.
Example message:
<iot id=e2f9a344>
<ldr>
<value>24 </value>
</ldr>
<distance>
<value> 10 </value>
<direction> 270 </direction>
</distance>
</iot>
18
JSON
Moderately data structure.
Less metadata than XML.
Smaller files than XML
Easy for devices to parse, but requires more
memory/storage.
Easy for humans to parse.
Example message:
{
“IoT":
[
{“type": “id", “value": “e2f9a344"},
{“type": “ldr", “value": “24"},
{“type": “distance", “value": “10", “direction": “270"}
]
}
19
Applications
and Data
20
Application Logic (for IoT devices)
The logic of what you want to achieve with the IoT
device and devices in group.
A program that tells the IoT device what to do OR
accepts inputs from device(s) to decide what to do.
e.g. authentication
Send username, password or any ID details
e.g. send records or control outputs
send temperature etc …
e.g. receive records or setting
retrieve any settings
e.g. turn on the fan and at what speed
21
IoT Data Generation
Requirements for data exchange will differ for each
IoT system will depending on what the end node
device:
wants from the internet
and can give to the internet
Request/response methods need to be agreed upon
so that they are predictable.
Communication with end node devices can not
always be guaranteed.
Often end devices sleep in between requests.
But the cloud never sleeps...
22
IoT Data Generation
Authentication requests
The device sends:
Authentication inputs ie:
Username
Password
or any ID details such as RFID card numbers,
finger prints etc…
What the device gets:
True or false response
Authentication mechanism:
PHP matches credential details and then responds
Server will log the authentication request.
23
IoT Data Generation
Log data request
The device sends:
Raw data ie:
Current temperature (float)
Time stamp
Device ID (and authentication if required)
What the device gets:
Success / Failure
How it works:
PHP matches the device ID.
The temp data and time stamp is logged to the
database.
24
IoT Data Generation
Automation request (Thermostat)
The device sends:
Raw data ie:
Current temperature (float)
Time stamp
Device ID (and authentication if required)
What the device gets:
Current fan mode and setting (on/off and percentage)
How it works:
PHP matches the device ID.
The temp data and time stamp is logged to the
database, and compared to a threshold.
The server returns the automation data for the fan.
25
MQTT
26
Traditional IoT communication
Traditional pull methods have several
downsides.
Requests are sent when the
information might not have changed.
More data is exchanged.
This uses more power.
27
P2P Data exchange for IoT
IoT devices can communicate using P2P methods
Mesh networks exchange data in this way.
IoT devices can implement TCP/IP protocols
which allow them to serve content (data) to other
devices.
This is very difficult to implement in Class 0/1
devices due to the lack of memory and storage
space.
28
What do IoT devices need for communication?
29
Message Queue Telemetry Transport
MQTT is an attempt to deliver a protocol for IoT
communications that matches its data needs.
MQTT implements a publisher/subscriber model to
address these needs.
MQTT organises data into topics.
MQTT uses a broker to manage communications and
connections.
MQTT uses a binary message format for data transfer
More compact that text-based protocols.
Messages are only sent went data is updated.
30
The Publish and Subcribe Model
31
Publish and Subcribe (Broker)
The broker is responsible for:
Authentications: checking the details of the
clients (publisher and subscribers).
Message passing: pass messages to any
subscriber.
Message Queuing: Hold on to messages for
limited period of time (but not infinitely).
32
Publish and Subcribe (Publisher)
Source of data publishes sensor and actuator
information
Example:
Temperature: 24
FanSpeed: 3
Should publish in time (as promised): They
should publish in definite time intervals.
A publisher can publish to multiple topics.
33
Publish and Subcribe (Subscriber)
A subscriber:
Can ask for any data at any time (subscribe
to a feed/channel/topic)
Expect the data in time
Can ask for multiple feeds at the same time
The broker maintains a list of subscribers and
pushes their subscribed topics to them as they
are updated.
34
Topics
Topics are pieces of data that can be published
to or subscribed to.
Topics can have paths.
example:
temperature/24
room1/temperature/27
room1/humidity/65
A subscriber can subscribe to multiple topics.
A publisher can publish to multiple topics.
35
How it works
Client 1 and 4 are publishers to the temperature
topic.
Client 2 and 3 are subscribers to the temperature
topic.
The broker accepts the most recent data to the
temperature topic.
When it’s updated, this is published to
subscribers.
36
MQTT Advantages
Can be Asynchronous as in HTTP:
Device can send the output and don’t stay
blocked up to when it gets the response
Persistent connection
Not stateless like in HTTP
Real Time
Low energy as it is more light weight (lower
overhead)
Packet headers are very small
Flexible
No need to know IP address
Only the feed name is sufficient
37
MQTT Disadvantages
Difficult to use in non-real-time environments
Devices cannot go to sleep on their own.
Need to follow a definite pattern as the data
in the channel/topic/feed is not held forever
Less Security of data
Useful for reporting “non-critical” data only.
38
Coming up...
Next week:
IoT Data Visualisatons
Thanks for attending!
39
