ICTworks

I am Michael Benedict and while working with SMS-based applications I’ve noticed an air of mystery around the issue of reliability. I hear colleagues say ambiguous things like “carriers consider SMS to be lower priority than voice”, or “SMS delivery isn’t guaranteed”.

My personal experience has been that messages are almost always delivered quickly and correctly, but I’ve heard stories of hours-long delays, corrupt data, and occasionally messages that never arrive. Since I am working on two projects that depend on reliable SMS service — one involves field-based data collection and another employs SMS as a transport layer between computers — I’m curious about how factors such as network, location, and time of day impact message transmission.

I found myself in Mwanza, Tanzania last week with two GPRS modems and a local partner who was unenthusiastic about the work I was actually there to do, so I tried a little experiment. I bought SIM cards and airtime for three of the major TZ networks, put two at a time in the modems, and wrote a simple python script using Adam Mckaig’s excellent pygsm library.

The script sent messages in all permutations at an arbitrarily chosen interval of about 20 minutes. This means that four messages were sent at a time: each SIM to itself and to the other SIM, repeated about every 20 minutes for periods of up to 27 hours. Outgoing messages noted the origin and destination networks, send time, and a sequential message number to make it easier to spot lost messages. Incoming SMS were stamped with the time received, and the delay in seconds between sending and delivery. Messages were sent two seconds apart to reduce errors from the modems trying to send and receive at the same time, and results were recorded in a CSV file.

a) Airtel and Zantel

Zantel is one of the smaller networks in TZ, but I was having technical problems with my Tigo SIM so started with Zantel and Airtel. Airtel was known as Zain until recently when Zain was acquired by Baharti Airtel, based in India. The script ran from about 1AM on January 14 to just before 5AM on January 15. Here are the results:

Delivery times and dropped messages between Airtel and Zantel Tanzania. Click to see scale. Note that the Airtel modem stopped sending and receiving at about 7AM on Jan 15, and the script hung until I woke up shortly after 8 and reset the modem. I do not know what caused the problem.

331 messages were sent and 10 (3%) were dropped. Delivery times fell into three bands. Airtel-Airtel (i.e. the Airtel SIM sending to itself) was easily the fastest, with messages typically delivered in 6 or 7 seconds. Messages in either direction between Airtel and Zantel took roughly twice as long, typically between 13 and 16 seconds. One outlier, an Airtel to Zantel message sent between 2 and 3 AM on Jan 15, took 24 seconds to arrive, and the only dropped Airtel-Airtel message occurred around the same time. Delivery times for Zantel-Zantel messages were split unevenly: one cluster near 6 seconds, another around 15 seconds, and a third around 32 seconds. 9 messages sent from Zantel-Airtel were not received.

Some aspects of this plot seem reasonable. Airtel is a larger company with wider network coverage, so it’s not too far fetched to imagine that messages sent from an Airtel SIM to itself would be the fastest to arrive. By that reasoning I’d expect messages sent between the two networks to take somewhat longer, and they do. The Zantel-Zantel delivery times are less intuitive. I don’t know what would account for the three distinct time ranges for delivery. It is also unexpected to see that 9 Zantel-Airtel messages were never delivered but no Airtel-Zantel messages experienced this problem.

b) Airtel and Tigo

SMS delivery times between Airtel and Tigo SIMs on January 14, 2011. Click image to see scale. *Messages marked unreadable appear to have reached the modems but could not be read, and resulted in an (unfortunately unlogged) exception in pygsm.

192 messages were sent and 3 (1.6%) were not readable by the modem. Four distinct delivery time ranges emerged. I eliminated one outlier from the plot to make it easier to view: on January 15 at 11:47, a Tigo-Airtel message took 129 seconds to arrive. I believe that the 3 unreadable Tigo-Tigo messages reached the modem, but the script raised an exception and the messages were not read. Unfortunately I was not logging pygsm’s debugging output so I am marking the missing messages as “unreadable” instead of dropped. I will have to log output in future runs and see if I can reproduce the problem. I didn’t expect Tigo-Airtel messages to arrive slower than Airtel-Tigo. The difference is small but it would be interesting to know why it’s there.

c) Two Tigo SIMs.

The purpose of having the modems send messages to themselves in the previous two runs was to test delivery times within a network. It occurred to me that sending from a SIM to itself may not be equivalent to sending between two different SIMs on the same network, so I bought a second Tigo SIM:

SMS delivery times between two Tigo SIMs. Click to scale.

164 messages were sent and 1 (0.6%) was not received. The data shown excludes one outlier: the first Tigo A-Tigo B message took 457 seconds to arrive. So: is SIM to SIM within a network equivalent to sending from a SIM to itself? Sometimes. I don’t know why it varies, but note that the number for Tigo A is +255716379091 while Tigo B is +255717435798.

In particular, after the 255 country code the numbers have different prefixes, 716 for A and 717 for B. Typically Tanzanian SIMs on the same network have the same first 3 digits, and the difference here may indicate that Tigo’s servers treat the two numbers differently. The Airtel/Tigo data above used the SIM that is labeled as “Tigo B” here.

Thoughts and next steps

Unexpected patterns emerged, including asymmetric delivery times between networks, that I would like to understand better. The few dropped messages do not seem to follow a clear pattern, which is a concern for both human and computer-focused SMS applications where missing messages can cause confusion and frustration.

It seems possible that having the modems send all 4 messages with only a 2 second delay between sending could be impacting the results. It would be interesting to revise the script to randomly offset the message send times by a few minutes and see if that increases reliability.

This is not a scientific experiment since there are variables such as time of day, day of the week, network traffic, modem location, weather, and likely many others that are not being controlled for. Still I think the data gives a representative snapshot of network performance. Overall SMS delivery was quite good. With the exception of a few of outliers most messages were delivered in under a minute, and Airtel consistently delivered on-network SMS in less than 10 seconds.

Occasional dropped messages show that safeguards need to be in place if SMS is used for critical applications. Some preliminary timing data I took in Uganda suggests that the networks there are not nearly as reliable. I’m looking forward to looking more carefully at Ugandan networks when I’m back.

If you like this sort of thing and also have too many modems and not enough social life, feel free to try out the script yourself. I’d be interested to see what the results look like in other countries.

A special thanks to Henry Corrigan-Gibbs and Matt Berg for interesting discussions about the timing script and SMS reliability.

This was posted first as SMSpecially Reliable? Part 1 and is repulblished here with permission