RN171/Wifly Sensor Network

Background

RN171-EKI've seen too many Arduino Wifi remote sensors that proudly announce their battery life measured in days or even hours. What frustrates me most is that none of them seem to have realised that the RN-171/RN-131 Wifly module many of them are using to connect to the wireless network is a 32bit powerhouse that is perfectly capable of performing and posting the measurements by itself! They are all too happy to leave the Arduino banging away in a main loop, running off a 9V battery via the linear regulator, and ignoring the perfectly capable processor on the Wifly itself. I like Arduino, but I much prefer using the right tool for the right job!

ThingspeakPage

As such the object of this article is to outline a small, simple battery powered Wifi enabled sensor, with a operational lifespan in the range of weeks to months (depending on your setup) based around the Roving Networks (now owned by Microchip) Wifly RN-171 module: the RN171p-EK evaluation module. Our collected data is posted to Thingspeak using an interim translation service, but any cloud service or local database would do.

My test feed is located at: https://www.thingspeak.com/channels/8699

Now in theory, the Wifly would be perfectly capable of posting any data directly to a cloud service. In practice there are two limitations. The first is that for some reason I cannot fathom, the Wifly HTML Client mode terminates its requests with "\n\n" (linefeed, linefeed) rather than the much more typical "\r\n \r\n" (newline, newline) that pretty much everything else uses. As such many off the shelf webservers that I tried do not see the HTML Client mode requests at all, waiting for the "\r\n" that never comes then eventually timing out. Our other issue is that the data is posted in a fixed format. We have two ways we can approach this issue. One would be to create our own Server instance which we host and control (which would be perfectly OK using the Thingspeak source code, but beyond this simple project), or we can run a simple translator which takes the data from the sensor via the wireless network which is mounted where there is convenient power, and posts it in the correct format. For this excercise I took the translator option, written in python and running on a windows machine or a Raspberry Pi, but being python you could run it on pretty much anything. (Thingspeak guys - Feature request! Direct Wifly data submission please!)

Concept

Concept

Windows Configuration

Install Python 2.7 (get it here)
Install Twisted + Zope (get it here)

Raspberry Pi Configuration

The easiest way I have found to code and play with the Raspberry Pi has been the Adafruit WebIDE system. To use this:
Download the Adafruit Occidentalis OS (see here for configuring your SD card)
Install WebIDE (Instructions Here)
Install Twisted (sudo apt-get install python-twisted)

Import the code into webIDE and then you can run or schedule it as necessary. Do not try to Visualise or Debug as webIDE doesn't seem to cope very well with the twisted libraries!

Software notes

  • I am a hardware guy. The software is probably not perfect and probably not the Python way of doing things. Tough. It works.
  • This is for Python 2.7. Might work on 3 but not tested.
  • You must have have the Python Twisted Libraries  installed for this to work. I didn't fancy writing my own multithreaded server code, and the example I created is simply a small part of a much bigger project.
  • The host machine needs a fixed IP address for now, and should be on the same local network as the sensors.
  • Thingspeak only allows 8 data fields plus a status string to be submitted on a single channel. The Wifly module has 8 analog inputs, battery level, rssi, GPIO state, plus a lot more. As such I selected a limited subset of Battery voltage, RSSI, overall GPIO state and the 4 accessable analog sensors to log. I also populated the "Status" field with the Wifly and AP MAC addresses. You can very eaily change the translator program to submit other configurations.

The Code

 

WiFlyThingspeakBridge.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
from twisted.internet import reactor
from twisted.internet.protocol import ServerFactory
from twisted.protocols.basic import LineReceiver
import httplib, urllib
 
import logging
import optparse
import sys
 
# This is our TCP Connection manager reactor - It listens for any TCP connections on port 88 sent by the Wifly module. Data is in the format:
# GET /form?&value=0D1154202F666F726D3F2676616C75653D30&id=WiFly-EZX&mac=00:11:22:33:44:55&bss=55:44:33:22:11:00&rtc=5278f53b&bat=2832&io=d10&wake=1&seq=d&cnt=1&rssi=d7 HTTP/1.0 Host: www.example.com.
class TCPConnection(LineReceiver):
	delimiter = '\n\n'		# Wifly is a pain in that the Auto TCP connection line terminates with \n\n, rather than \r\n. This
			# is the reason we had to write a simple raw TCP manager, rather than use the "off the shelf" webservers.
	def connectionMade(self):	# This is run whenever a connection is made
		self.client_ip = self.transport.getPeer()		# Get the IP address
		logging.debug("Received TCP connection attempt from %s" % self.client_ip)
		if len(self.factory.clients) >= self.factory.clients_max:		# Have we got too many connections?
			logging.warning("Too many connections.")
			self.client_ip = None
			self.transport.loseConnection()		# Yes, dump it
		else:
			self.factory.clients.append(self.client_ip)	# Put into client list
			self.timeOut = reactor.callLater(_timeout, self.timeOut)	# call later the timeout incase the connection stalls
 
	def connectionLost(self, reason):	# This is run whenever we lose a connection
		ReasonStr = str(reason)			# Get rid of annoying midline CR LF's
		ReasonStr = ReasonStr.replace('\r','')
		ReasonStr = ReasonStr.replace('\n','')
		logging.debug("Lost connection %s.  Reason: %s" % (self.client_ip, ReasonStr))
		if self.client_ip:				# Remove from list if necessary
			self.factory.clients.remove(self.client_ip)
 
	def timeOut(self):		# This is called when the calllater timer expires
		logging.debug("Timeout on %s." % (self.client_ip))
		self.transport.loseConnection()			# Taking too long, dump the connection
 
	def lineReceived(self, line):
		heading = ['id=', 'mac=', 'bss=', 'rssi=', 'rtc=', 'bat=', 'io=', 'wake=', 'cnt=', 'seq=', 'value=']	# Fields in packet, in the order we want them in the database
		result = []
		global TCPMessageCount
 
		logging.debug("TCP data received:\r\n%s." % (line))
		self.transport.write('HTTP/1.1 200 OK\r\nConnection: close\r\n')	# The Wifly doesn't actually seem to care what is returned, but we should be good.
		self.timeOut.cancel()			# Cancel our timeout
		self.transport.loseConnection()			# Drop the connnection
		line = line.replace(" ","&")			# Quick and dirty method to improve split without resorting to RE
		line = line.replace("?","&")			# For some reason a single & gets lost sometimes, so break on ? too
		line = line.split('&')			# Split the line into each item, delimited by '&'
		for item in heading:			# Look for each item in turn
			notfound = True
			for var in line :			# Check each split line
				if var.startswith(item):
					result.append(var.lstrip(item))	# Remove the header from the result
					notfound = False
			if notfound:
				result.append(0)		# Give a default null value to keep the database fields lined up
		try:
			if (line[0] == "GET" and line[1] == "/form"):	# Basic Header check for sanity
				#Need to break out Analog sensor values, could do this in a for loop, but better readablity this way.
				#print result[10][0:4]	# GPIO's
				result.append(result[10][4:8])	#Channel 0
				result.append(result[10][8:12])
				result.append(result[10][12:16])
				result.append(result[10][16:20])
				result.append(result[10][20:24])
				result.append(result[10][24:28])
				result.append(result[10][28:32])
				result.append(result[10][32:36])	#Channel 7
				logging.info("Received TCP data from %s at %s" % (result[0], self.client_ip))
				logging.debug("TCP unpacked %s." % result)
				TCPMessageCount += 1
				print "\r> TCP Messages received: %s " % TCPMessageCount,
				params = urllib.urlencode({		
					'field1': result[5],	#battery
					'field2': (int(result[3], 16))-255,	#RSSI
					'field3': int(result[6],16),	#GPIO (encoded)
					'field4': int(result[14],16),	#Sensor 3
					'field5': int(result[15],16),	#Sensor 4
					'field6': int(result[16],16),	#Sensor 5
					'field7': int(result[17],16),	#Sensor 6
					'field8': int(result[18],16),	#Sensor 7
					'status' : 'MAC=' + result[1] + ' BSS=' + result[2],
					'key': result[0]})		# Our Device ID stores the APIKEY
				headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
				conn = httplib.HTTPConnection("api.thingspeak.com:80")
				try:
					# Try posting the data 
					conn.request("POST", "/update", params, headers)
					response = conn.getresponse()	# Get the response
					logging.debug("HTTP Post: %s, %s" % (response.status, response.reason))
					data = response.read()
					conn.close()	# Close the HTTP Connection
				except:
					logging.warning("Thingspeak Connection failed")
			else:
				logging.warning("Invalid TCP message, GET params wrong: %s %s" % (line[0], line[1]))
		except:
			logging.warning("Invalid TCP message or unhandled exception: %s" % sys.exc_info()[0])
			#print "> Invalid TCP message. ", sys.exc_info()[0]
 
# Produce our TCP Servers			
class TCPFactory(ServerFactory):
	protocol = TCPConnection
	def __init__(self, clients_max=10):
		self.clients_max = clients_max
		self.clients = []      
		logging.debug("MAX TCP Clients = %s" %clients_max)
 
# Main Program Starts here!
 
# Set up our logging level from the Command line options. Debugging is useful!
loglevel = 'WARNING'			#default level
optp = optparse.OptionParser()
optp.add_option('-l', '--log', dest='level', type='string', help='Set debugging level: DEBUG, INFO, WARNING, ERROR or CRITICAL') # Populate our options
opts, args = optp.parse_args()
if opts.level:			#If we have an option set, use it, otherwise the default will remain
	loglevel = opts.level
numeric_level = getattr(logging, loglevel.upper(), None)	
if not isinstance(numeric_level, int):		#Blat any invalid loglevels
	raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(format='%(asctime)s %(levelname)-8s:%(message)s',filename='Wifly.log',level=numeric_level)	#Setup our logging target
 
# Woo! Program start proper!
logging.info('Program Started.')
TCPMessageCount = 0
_timeout = 5			# Timeout in seconds before we flush a connection on our linereceiver
reactor.listenTCP(88, TCPFactory(5))		# Listening Reactor for TCP connections on port 88, 5 clients max
reactor.run()			# Start the Reactors!

Then run "python WiFlyThingspeakBridge.py" or via WebIDE.

 

(note: you can add --log=debug to log more information)

Hardware notes

  • The analog ADC's on the Wifly module saturate above 400mV. You must not apply a voltage above 1.2V under any conditions!

Thingspeak configuration

  • Sign up for a Thingspeak account here.
  • Sign in and go to "Channels", then click on "Create New Channel".
  • Fill in the details of your channel, being sure to add all 8 available fields. Name them as shown.
  • Then click "API Keys" and note down the "Write API Key" presented.

Module configuration

The module is configured as per the Wifly User guide. I prefer using the USB serial port, but you could perform the configuration via the Soft AP mode or Telnet.

We need to send the following commands to configure the unit:

factory RESET
Resets the unit to defaults. RESET must be in caps.

set wlan SSID <name of Wifi Network>
This is the SSID of the Wifi Access Point we want to connect to. Spaces must be entered as $.

set wlan pass <passphrase>
This is the passphrase of the Wifi AP.

set wlan hide 1
Hide the passphrase so it cannot be read out of the module.

set ip host 192.168.0.4
This is the IP address of the Host machine. Substitute your own.

set ip remote 88
This sets the port we want to use (88 for the posted source).

set ip proto 18
Sets TCP and HTTP client modes.

GET$/form?value=
This sets the beginning of our GET string (the Wifly uses $ as a space character).

set option format 0xff
This appends the data, id, GPIO and ASCII ADC values to the header.

set opt deviceid <API_KEY>
You need to enter the Thingspeak write API key here. By using the API key as the device ID, you can have many sensors each posting to a different channel with no other configuration changes needed!

turn on sensors!!
Turn on sensor power if required.

Timer options

If we are running the unit off of batteries, we really want the unit to be sleeping as much as possible, but this can make it tricky to reconfigure the unit. If there is a permanent supply, then power consumption is less of an issue and you may choose to leave the unit in run state permanently for convenience. The options are thus:

Battery mode

set sys auto 255
When the unit wakes, attempt a connection to the Host immediately, then go straight back to sleep once completed.

set sys sleep 5
Go to sleep after 5 seconds. If the unit fails to connect for some reason (Server not running, AP out of range), it would otherwise stay awake. This ensures it always goes back to sleep.

set sys wake 60
Wake up every 60 seconds. Obviously change this number if you want a different interval.

set sys trig 1
Wake up on serial recieve. Makes debugging easier!

Permanent mode

set sys auto 60
Connect to the Host every 60 seconds.

The module can also wake from a number of other sources such as wake on sensor input or wake on UART. See the user guide for more details.

When you are happy with the configuration type:

save then reboot

Start the translator program now on the host machine if you haven't yet. You can launch the translator with the "--log=debug" switch which will produce a log in the launch directory and should help to diagnose any issue.

Power consumption

As a test, I configured one of my units to wake at 20 second intervals, take all readings, then go back to sleep. The module posted approximately 28000 datapoints over a 156 hour period (>6 days), with a 35 hour gap which was caused by a power cut to the host server (this is actaully a worst case scenearo, as the module does not return to sleep as quickly on a failed connection, as can be seen by the more rapid battery depletion over this period). The module takes approximately 1.5 seconds to perform a successful post, and 5 seconds to fail.

As the sleep mode power consumption is negligable compared to the active power (4uA sleep, 40mA RX, 180mA TX), we can get a good idea of battery lifespan in terms of datapoints by extrapolation.

If we extend the time interval to once per minute, we should get around 19 days worth of endurance ((28000 / 60) / 24 = 19.4 days).
If we extend the time interval to once every 5 minutes, we should get almost 100 days worth of endurance ((28000 / (60 / 5)) / 24 = 97.2 days).

Be aware this does not take into account power consumption of any attached sensors, nor any temperature variation which may reduce the battery lifespan somewhat. I did not disable the LED's either which would save some power (but is useful to see if it is working!).

The sleep current will start to become a more dominant effect the longer we go, but at longer intervals we can expect even longer endurance. I personally feel that at a suitably long interval (20+ Minutes) a years operation from a single set of AAA's (860 – 1200mAh) should be acheiveable. Of course if we add a rechargable battery and a small energy harvester such as a solar panel, endurance should be effectively unlimited!

Next Steps

I didn't cover much on sensors themselves. I leave this as an excercise to the reader. This blog has a good writeup with a similar goal, albeit with a different approach. 

http://www.tinkerfailure.com/2013/01/wifly-sensor-pins/

Once your data is in "the cloud" it becomes possible to present that data in a variety of ways. You can always just use the built in thingspeak triggers and visualisations, or you can use its powerful data API and create your own charts or data handling systems. For example, heres a simple page that presents the data in a more specific way:

https://dl.dropboxusercontent.com/u/63481/RN171Test.html

demopage

 


Posted: 7 months 1 week ago by Jon Chandler #13707
Jon Chandler's Avatar
Thanks for a great article.
Posted: 7 months 10 hours ago by hop #13751
hop's Avatar
Yes, great article, and thank you for the heads up! I got to try one of these for outside projects for talking to my wireless network.
Posted: 6 months 1 week ago by RangerBob #13882
RangerBob's Avatar
FYI: Yet another article about datalogging using an arduino.

http://learn.adafruit.com/low-power-wif ... g/overview

Gets the right general idea using sleep modes and reduced hardware but the final solution gets predicted 33.3 Hour run time off 6 AA batteries even with optimisations, logging at 1 minute intervals. Sleep mode is almost 1mA.
Posted: 2 months 4 weeks ago by bwebtestb #14299
bwebtestb's Avatar
Hi ,

I notice from other site that we need to configure both the host ip address and hostname for http request to work . However in your example only ip address is configured. Is it the case that either the hostname or the ip address is required ?

I am thinking of using in in a small home monitoring project but e server i have doesnt have a static ip address. Thanks for clarifying.

Forum Activity

Member Access