This report details an investigation into the functionality and performance of the MQTT networking protocol, using a locally hosted Mosquitto broker and various Python scripts. It outlines MQTT’s various QoS levels, the tests undertaken and their results, and the challenges one might face when using MQTT in a broader networking environment.
QoS Levels & Analysis
Quality of Service (QoS) refers to the level of reliability regarding message delivery in a communications system. QoS is crucial in determining how messages are transmitted, received, and processed – directly impacting communication accuracy and efficiency. The MQTT protocol offers 3 main levels of QoS. These include :
QoS 0 (At most once): This is the lowest level of service where the message is delivered once or not at all. The sender publishes a message to the broker, and the broker immediately forwards it to all subscribers. As there is no acknowledgment from the subscribers, the sender will not know if the message was received, nor will the message be stored to retry delivery. This level is typically used when message loss or duplication is acceptable. An example of this may be a basic temperature sensor, where the latest value will always be more relevant than the previous one.
Fig 1.1 – A wireshark snapshot, publishing a message to an external broker at QoS 0.
QoS 1 (At least once): In this level, the message is guaranteed to be delivered at least once to the receiver. When the sender publishes a message, the broker sends it to the receiver and waits for acknowledgment (PUBACK). If the acknowledgment is not received within a specified time frame, the broker resends the message. This ensures that the message is eventually delivered, however can lead to duplicate messages if the acknowledgment is lost or delayed. This level is typically used when ensuring message delivery is important, even if it means occasional duplicates. This has a place for an application looking to reduce the overheads of QoS 2, but offer better reliability than QoS 0. An example of this may be a monitoring system, where receiving multiple notifications about the same event is preferable to missing notifications entirely.
Fig 1.2 – A wireshark snapshot, publishing a message to an external broker at QoS 1.
QoS 2 (Exactly once): This is the highest level of service, ensuring that the message is delivered exactly once to the receiver, involving a multi-step acknowledgment process. Firstly, the client will publish a message to the broker, temporarily storing its contents locally. When received, the broker will then reply with an acknowledgement (PUBREC), confirming the packet has been received (and its contents are correct). Once this acknowledgement is received by the client, the client will delete its local copy and reply with a confirmation (PUBREL). When the broker receives this confirmation, it will send a final acknowledgement (PUBCOMP). If the subscriber requests information at QoS 2, this will also take place – with the broker acting as the publisher. This process ensures the message is delivered exactly once without duplication or loss, but it introduces more overhead due to the additional acknowledgment steps.
This QoS level is typically used when reliability is crucial to the design of an application. An example of this would be a banking system, where every transaction must be processed exactly once to maintain accuracy and consistency. Similarly, systems in industries like healthcare, aviation, or industrial automation are also examples of this – where any duplication or loss of commands may have severe consequences.
Fig 1.3 – A wireshark snapshot, publishing a message to an external broker at QoS 2.
Final Measurements and Statistics
Per the assignment specification, 180 tests were completed over 60-second increments. These were made up of : 1 to 5 instances publishing simultaneously, 0, 1 and 2 levels of publishing QoS (client and analyser) and 0ms, 1ms, 2ms and 4ms delays between published messages.
For each of these combinations, the client script kept track of the number of messages it sent. The analyser kept track of how many messages were received, the average rate of messages received a second, the median inter-message gap and the percentage of messages that returned misordered. After each 60-second test, the analyser also called the $SYS topic, logging the percentage of messages dropped by the broker, and the average number of messages being received a second. Finally, these were output to a CSV which was then imported into Google Spreadsheets for further data analysis and formatting. It is here where the rate of message loss (between the publisher and analyser) is calculated.
A copy of the final, formatted spreadsheet containing all 180 tests has been included at the bottom of this report.
Different Delays
As expected, the largest contributor to lower message rates was the four delays. As larger delays were implemented in the scripts, fewer messages could be published in the given 60 seconds. These scaled linearly with the delay times (1ms, 2ms and 4ms) with 1ms tests seeing roughly 4x the throughput compared to the 4ms tests. This is imperfect, however, with the scripts limited by the tick rate of the operating system, and Python incapable of exact millisecond precision. Still, the trend we expected to see is more than evident in the results.
Fig 2.1 – Message throughput at 1/0/x and the impact of delay (1ms, 2ms and 4ms)
Less expected, however, was the exponential increase seen in 0ms tests. By removing a limiter entirely, the publisher was able to run seemingly as fast as the host computer would let it. Being run on a high-spec, multithreaded CPU, we can see the 0ms tests producing millions of messages compared to those intentionally throttled by delays.
Fig 2.2 – The difference between no delay (0ms) and adding intentional delay.
I also tested running the scripts on a lower-spec UNIX machine, and though the results were somewhat lower – it’s still overwhelmingly apparent that the 0ms tests are capable of producing a significantly higher number of messages.
Fig 2.3 – The same test being run on a lower-spec laptop.
With hundreds of thousands of messages published every second, it’s no surprise the broker (and subsequently analyser) struggled to keep up. Querying the broker, we can see nearly 50% of publisher messages were dropped to help reduce congestion. Similarly, out of the messages that got through to the broker, only another ~25% were able to be received by the analyser. Surprisingly, the misordered rates for the packets that did arrive were very low (~0.07%) – potentially suggesting a first ‘block’ of packets was able to get through before the rest were dropped.
Fig 2.4 – Messages sent and received rates on 0ms testing.
Fig 2.5 – Messages sent and received rates on 1,2 and 4ms testing.
Outside of the 0ms irregularities (especially at QoS 0), we can see the broker and analyser were much better at keeping up. By slowing down the message throughput, both the broker and analyser were able to keep track of all messages being published with seemingly no packets being dropped or misordered.
Additional Publishers
Additional publishers resulted in a higher computational load on the broker, clients and analyser. As all components were running locally, additional instances stressed the computer, requiring more computational power to send the messages and process them. However, adding additional publishers seemed somewhat inconsistent in my testing. Though the analysers’ ‘messages per second’ statistic trended downwards with the addition of new publishers, the same cannot be said for the percentage of dropped messages or the number of messages sent. As discussed later in the report, this is likely due to the performance of the computer the tests were run on (with 5 instances only utilising ~25% of total CPU resources).
Fig 2.6 – Messages received / second when additional instances are added.
It would be interesting to run this experiment again, with the analyser and broker on separate PCs, to see how they perform when not under the stress of additional clients.
Different QoS (Publisher)
As the client publishes at higher QoS levels, the trends mostly mirror those outlined at the beginning of this report. That is, higher QoS levels result in more computational power being used (and subsequently, less throughput). Having said this, my program ran into some interesting behaviour when running the tests at QoS 0 with a 0ms delay. When these were run, significantly fewer messages were published by the client, in comparison to QoS 1 and 2 – an opposite result to what we’d expect to see.
I suspect this may be because the client is attempting to publish messages at a rate that overflows local memory buffers, essentially resulting in the script timing out before the 60 seconds are up. This also explains why the client scripts would lag behind the analyser values at higher duration increments.
Fig 2.7 – Messages sent at different QoS levels (0, 1 and 2) on 0ms tests.
Interestingly, however, is the performance of lower QoS levels when delay is added. As the number of messages decreases (due to intentional delay), we can see no difference between QoS 0 and 2. This is because, as the delay between messages increases, the network and broker have significantly more time to process each message, reducing the likelihood of errors.
| Topic | Messages Sent by Client | Messages Received by Analyser | Messages Dropped |
| counter/1/0/1 | 35932 | 35932 | 0 |
| counter/1/1/1 | 36348 | 36348 | 0 |
| counter/1/2/1 | 36416 | 36416 | 0 |
| counter/1/0/2 | 21680 | 21680 | 0 |
| counter/1/1/2 | 21792 | 21792 | 0 |
| counter/1/2/2 | 21460 | 21460 | 0 |
| counter/1/0/4 | 12112 | 12112 | 0 |
| counter/1/1/4 | 12164 | 12164 | 0 |
| counter/1/2/4 | 12216 | 12216 | 0 |
Fig 2.8 – Messages sent by client vs received by analyser at counter 1/x/x
Different QoS (Analyser)
While I hadn’t expected the different analyser QoS levels to make much of a difference, there were some small trends apparent. Notably, there was a slight decrease in the number of messages being received as the QoS level increased. This could be attributed to the additional resources required to process higher QoS levels as acknowledged earlier – but this time on the analyser end.
Fig 2.9 – Message throughput when received using different QoS levels.
The data also shows additional fluctuations in message drop rates and delays at higher analyser QoS levels. Again, these fluctuations could be explained by increased computational overhead however, it is important to note that these results were not as conclusive as other parts of the experiment.
While additional load from higher QoS levels may have caused performance degradation – it also may simply be other processes or an increased PC uptime that caused this. While on the publishing side, we see clear results – a further investigation would be required to determine the exact impact of different analyser QoS levels.
Correlations with $SYS/# Topics
As the broker is privately hosted (we are the only ones publishing to it), we can use the broker’s $SYS topics to gather some useful statistics – some of which have been referenced earlier. My program chose to subscribe to the following $SYS topics:
- $SYS/broker/messages/received: This topic shows the total number of messages received since the broker started. We can then convert this number to an average messages-per-second rate to monitor broker stress.
- $SYS/broker/publish/messages/dropped: This topic indicates the number of published messages that have been dropped since the broker started. Again, by converting this number to an average dropped percentage we can make assumptions by comparing it with data from the analyser.
Outside of the x/0/0 oddities, we can see these statistics nearly perfectly mirror those gathered by the analyser, suggesting the analyser script is capturing accurate and useable information. Furthermore, the large discrepancy between the x/0/0 results tells us the broker is struggling under load, and dropping messages rather than forwarding them to the analyser (as in all other instances).
MQTT In The Broader Networking Environment
While these results show some common limitations of the MQTT protocol, when working in a wider networking environment (with potentially millions of messages being published each second), additional performance challenges may arise. Potential chokepoints, congestion and weak links can be found in all layers of the network stack, ultimately stunting message throughput and reliability.
At the application layer, the MQTT broker acts as the sole point for message handling. Should the computer architecture lack the computational power required by the broker to process high volumes of incoming messages, congestion and queueing may occur – potentially leading to message loss or increased latency. Similarly, even if the MQTT broker were properly equipped to handle the load, it may still go down for external reasons – resulting in bottlenecks and service disruptions.
At the network layer, network latency and capacity can also present a challenge for message deliverability. Should the server be located a large distance from the client (or subscriber), delay may be more noticeable due to the increased number of router hops data packets must take. The protocol is also prone to external network congestion and conditions, which (particularly at peak hours) may lead to increased packet loss and retransmissions.
Within the physical layers, there are several spots where a message may be lost, especially when placed under load. Network switches, routers, and even physical cables may succumb to information overflow, causing messages to be dropped. Specific device components (CPU, memory, and disk I/O) may also become a bottleneck for these messages. Furthermore, inadequate provisioning or inefficient resource management can compound performance issues, leading to degraded system responsiveness and stability.
In a similar line of thinking, the various QoS levels outlined earlier can also contribute to (or alleviate) these congestion issues. QoS 0, despite its lower reliability, can ease network overhead by minimizing acknowledgment processing. As a published message does not require the broker to reply with a confirmation, the network throughput and computational processing required are effectively halved compared to QoS 1. Likewise, the more complex back-and-forth communication required by QoS 2 adds strain to the network, by requiring four different packets, again compounding the amount of traffic needing to be processed. It’s for this reason networking engineers and application developers must carefully consider the trade-off between message reliability and network capacity, especially in scenarios where hundreds of messages are being published per second.
Additional Questions
What happens if the Analyser modifies the request topics at the same time the Publisher sees the changed values – how could you ensure it doesn’t start prematurely?
Were the analyser script to modify the request at a random time (rather than known intervals), the publishing script would become considerably more complex. Assuming the analyser was able to modify the request topic at any given time – the client would need to concurrently listen for new messages, whilst publishing existing ones, substantially increasing computational costs. To implement this functionality, the client must interrupt its current operation when it hears new request topics, and begin publishing on these.
Assuming these are updated at random, the client would also need to record the length of time it has been publishing – dividing its final counter by some increment of time. This ensures that, say if the analyser published for 3 seconds, then 7 seconds, the publisher would return an average count rather than the full number of messages, for accurate statistical analysis.
My code does not choose to implement this strategy, as I am the one dictating the analyser function. Rather, it simply opts to hardcode the duration of the publishing and analysing scripts using a global duration variable, providing the analyser with a slightly longer request time to ensure no clashes occur.
How does it run with 10, or more, publishers with qos=0, delay=0ms?
As expected, as more clients are added, message throughput becomes less and less. The clients report sending fewer messages, the broker reports dropping more messages and the analyser sees fewer messages. This is due to the significant increase in computational power required. As seen in Fig 3.1 and 3.2, the CPU usage almost doubles when comparing five instances with ten. My computer was able to handle ~20 instances before its performance and usability took a significant dive (CPU running at ~100% load).
Fig 3.1 & 3.2 – My computer’s CPU usage with 5 and 10 instances running.
Try to run this on a public broker, across the internet. What happens to the performance and stability of the results?
As outlined earlier, when running over the wider network, MQTT performance is degraded significantly. I attempted running my script on several free, public brokers, however at 0ms, found myself quickly rate-limited by most free options. Instead, I opted to run a 4m test on EMQX’s free public broker. Seemingly, on counter/1/0/4 my program was only able to send 5029 messages (a significant decrease on the 12112 messages seen locally) with my analyser only seeing 2000 of these. Unsurprisingly, it’s clear that running MQTT over a wider network introduces significant performance and stability challenges, with the drop in message throughput over EMQX’s broker being a clear indication of this.
