I work at uSwitch.com and late last year we moved to some nice new offices. Unfortunately, whilst awesome, the offices are split over 6 floors, I knew immediately that this would lead to missed connections and endless stair-runs, so I set about building something that would remove that pain. The end result was this…
It’s loosely called “whereabouts”. This post is about the process of discovery that made it possible to build the two applications that would make it easier for us to find each other throughout the building. Despite being specific to our network, most of what is detailed here is based on principles that should apply to your own, and it should be possible to achieve the same results.
A quick shout out to @randomvariable, anytime you see the term “ops”, it really means him. He fixed the network the first time I nuked half of it, and provided insight throughout.
As you might expect we have Access Points (APs from here on in) on every floor, each one is providing a wireless network, but is at the same time also wired into the physical network. It’s sensible to assume that all APs would be managed centrally, and this system is usually referred to as a “controller”, so I asked the guys in IT Ops for access and logged into the interface to have a poke around.
Peering at the settings in the control panel I noticed there was an option to set a remote syslog server. Knowing that I would be running arbitrary code that would probably be processing logs, it seemed best to make use of this feature. I went with RSYSLOG and set it up on a fresh Ubuntu instance, which was as simple as doing a
sudo apt-get install rsyslog.
By tailing the log on the Ubuntu instance I started to see various bits of output related to our network, but nothing particularly useful.
The “Access Points” tab showed a list of all the APs in the building and their corresponding IP addresses, so I SSHed onto the one that my laptop was connected to, reasoning that it might have some configuration files for me to mess around with.
1 2 3 4 5 6 7
From the prompt I could tell that the device is running BusyBox, a very small, low resource Linux distribution that is often used on embedded systems. I’m aware of it but have rarely used it. So I begun looking for config filesin the obvious places.
1 2 3 4 5 6 7 8 9
Nothing in particular stood out, so rather than checking them all one at a time, I took a look at running processes.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
What’s interesting here is hostapd which I recognised from some previous projects involving Raspberry Pis. This daemon is responsible for handling WPA authentication so it seemed like there would be good chance it would know who was authenticating. Thankfully each hostapd process references a configuration file so I opened one of those next.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
The last two lines reference syslog, but hostapd documentation is not exhaustive, or if it is I simply don’t know how to find it. A brief google did turn up a sample config that contained some comments explaining the values.
logger_syslog is set to “-1” which logs for all modules so I left it be.
logger_syslog_level however is set to “2” (informational messages), wanting more verbosity, I set it to “1” (debug) in each of the config files and rebooted the device.
The tailed output from the Ubuntu instance flicked up some messages about the AP going down and coming back up, but nothing more detailed than before. Upon closer inspection I noticed that the config files had reset to their original states. Assuming this was probably by design, I started wandering around the more common places people drop their shell aliases. In
/etc/profile I found an amusing comment, and a helpful looking command.
1 2 3
I guess Ubiquiti engineers really like Ubuntu. Anyhow… the
save alias speaks of
cfgmtd, so I checked help to see what I could learn.
1 2 3 4 5 6 7 8 9
From the flags I could see it was used for reading and writing config to the flash memory of the device, which persists through reboots. It also told me that the default config is in
/tmp/system.cfg, all 480 lines of it.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Line 12 in the abbreviated sample showed the verbosity being set to 3, which would suggest that only “notifications” are being logged, but when I checked the config it distinctly showed that
logger_syslog_level is set to 2 for informational messages. At first I didn’t notice this mismatch, but eventually through repetition I caught the problem and set the correct value. With that settled I flashed the device once more.
Finally the tailed output from syslog started to show some users authenticating. The IP address is for the particular AP that received the event, the MAC Address refers to the device connecting, in this case my laptop.
I thought it would be plain sailing from here on out, but of course that was not the case. Over the next day or so “identity” messages stopped appearing in the logs. The configs for the APs were being reset. I was fairly sure it would be triggered by the controller as it was affecting all of the APs. Through trial and error I discovered that certain changes on the controller would reprovision the APs.
SSHing into the controller, I had no idea where to look, so I resorted to reviewing the running processes like before and found some interesting candidates for inspection.
1 2 3 4 5
Seeing the mongoDB process, I connected to it with
mongo --port 27117, which gave me access to the “ace” database/collections. It’s mostly stats and settings from the control panel on the controller.
The remaining Java processes referenced “unifi” paths on the filesystem, which is the name of the product itself. So I pulled down the “ace.jar” mentioned in the processes and fired up JD-GUI, a tool that takes Java bytecode and spits out source code.
Most of the time the result looks like shit, symbol names are not preserved and any meaningful structure is lost. If you’re feeling brave, I created this gist for anyone that wants to see what I mean. This was a class called “String”, which contained the code for reading and writing settings, I found it by doing a global search for “aaa” which was part of the filename for the wifi configs.
From that massive file the only interesting part is this, as the values are hardcoded it’s kind of fortunate that one would provide enough verbosity.
Òo0000 on the class
D, was potentially calling another two methods on other classes, if either case were to return the string “debug” then the desired
logger_syslog_level value will be used whenever the controller reprovisions.
1 2 3 4
I’ll spare you the pain of looking at obfuscated code and skip to the conclusion: I eventually discovered the “build.type” mentioned above would be read from a file called “product.properties”, and that the “debug” would be read from “system.properties”. Both of which appeared in the compiled JAR itself. Luckily though, the code actually attempted to read from the file
/usr/lib/unifi/data/system.properties before resorting the internal version. By adding
debug=true to it I was able to override the config. I was finally ready to start building something.
Whereabouts is comprised of two pieces: the first is the backend that tails the syslog and upserts documents in a mongo collection, and the second is a basic meteor app that tells you who is where in the building.
They are called whereabouts-syslog-tail and whereabouts-meteor respectively. They were hacked out quickly so please get in touch, create an issue or send a pull request if you need some help or want to contribute.
When the first set of “identity” messages started appearing I set up a small Node.js script to tail the syslog and write entries into a mongo collection, at first this was just comprised of the username and a MAC Address. Watching for connection and disconnection messages seemed to cover only edge cases: the collection didn’t reflect the state of the network, probably because it was missing things like association and deauthentication.
I googled for hostapd documentation but again couldn’t find anything useful, so this time I decided to checkout the code. A spate of “event X notification” messages would appear in the log, so I looked for the string in the source code.
1 2 3 4 5 6 7 8 9 10 11 12
The shortened example above shows a
wpa_event being passed to the logger function, the event turned out to be a basic enum.
1 2 3 4
Each of these seven events is implies whether a device is connected or not, for example, seeing a
WPA_DEAUTH is enough to say a device is disconnected until another message indicates otherwise. I modified the script to process these events and update the collection, adding a human-friendly name for each location instead of an IP address.
There’s a little more tidy-up done in the app, and it exposes a very basic API, but you can refer to the repository for more detail.
Meteor is particularly good for rapidly building single page apps, out of the box it gives you:
- Hot code deploys.
- Live updating HTML.
- Handy data synchronisation.
The live updates mean that data is bound to templates, when the data changes, the templates react to those changes. This saves you from having to wire up various events and write other boiler-plate code.
The data synchronisation is also pretty awesome, by default a Meteor app will use a client-side mongo implementation called minimongo. It wiring up callbacks for any changes to documents, these changes will then get pushed to all connected clients at the same time.
When you specify a real mongodb instance with the
MONGO_URL environment variable, Meteor will use that instead, keeping the synchronisation feature.
Given that the the mongo collection reflects the state of the network at any given time, the Meteor app took only a few minutes to create and simply finds all known devices. They are ordered by whether they are connected, followed by their last connection time. It’s a bit basic but it serves the purpose and acts as a reasonably straight-forward example, I will happily link to any other interfaces that are written.
An issue that arose from tracking devices is that people often have more than one. The next obvious feature is allowing people to name their devices by visiting the application with said devices. It’s not possible to obtain a client MAC address in a web-app without resorting to something horrible like ActiveX controls, so instead the app will need to compare the client IP address with those of known devices to find a match. The IT Ops guys were happy to push our DHCP logs to the remote syslog server I set up, and whereabouts-syslog-tail has the code necessary to associate an IP address with a device, but work still needs to be carried out on the front end.
Aside from that I would be happy to work on making the whole solution more generic, and thus more suitable for other networks, but in order for that to happen I’ll need volunteers to give feedback on their setup and what does/doesn’t work for them, so please get in touch if you’d like to/some help.