Information

This site presents some of the metrics my little home server collects, among the things it is currently collecting are:

  • Electricity consumption
  • Indoor environmental monitoring
  • Outdoor weather information
  • Data from my Fitbit One
  • Energy usage

Each of these is documented in more detail if you select the navigation buttons over on the left, or if you want to view the data click the navigation buttons up at the top of the screen.

Technology

The site is hosted on an HP Micro server out of my home in Ruislip, written in PHP, using the excellent Twitter Bootstrap framework to style the HTML and provide navigation elements. Most of the data is grabbed from whatever the source is and stored in RRDtool database files All of the charts are provided by the Highcharts Javascript library, with a little help from JQuery where needed. The weather station is being driven by the brilliant pywws software.

Electricity usage

Data collection

This data is collected by a Current Cost energy monitor, it is a pretty standard clamp on the wire which sends data wirelessly to a display device. But it has a serial connection which can be plugged into a USB socket on computer, this simply prints some raw XML every time a data point is received from the clamp. The format is helpfully documented by Current Cost here.

I have some perl which opens the /dev/ttyUSB0 serial device and sits waiting for data, when some turns up it throws it at a instance of Mosquitto, which is a MQTT message broker. Other things can then consume this data, in the past I had a consumer sending data to the - now defunct - Google Power Meter project. Since Google killed that I only have one consumer, which is another perl script which subscribes to the relevant topic on the buss and then puses the data into a RRDtool database.

Visualisation

Highcharts can (via a bit of JQuery) fetch some JSON from a URL and then easily parse that data into a format needed by the charting engine. Helpfully the PHP RRDtool bindings can 'export' a RRD database into a PHP data structure, which can then trivially be turned into JSON and returned to the browser. You can see the source JSON for the power chart here.

Indoor environment

I've been looking around for some relativley cheap wireless temperature sensors for a while now, late in 2013 I discovered this blog post which linked to a sensor and a usb reader which looked to fit the bill. The site you can buy them from is in German, but a bit of Google Translate later and I had a USB receiver and a few temperature sensors on the way. You can buy them from the manufacturer here.

 

Example

A quick bit of perl reads the text sent down the serial port, which is just plain ascii, it looks like this:

$1;1;;21,6;9,5;21,6;21,3;21,1;19,2;;;58;78;58;59;42;53;;;;;;;;0
$1;1;;21,6;9,5;21,6;21,3;21,1;19,3;;;58;78;58;59;42;53;;;;;;;;0
$1;1;;21,6;9,5;21,6;21,3;21,1;19,3;;;58;78;58;59;42;53;;;;;;;;0

it is documented in the manual which comes with the receiver, but essentially the 21,6 is 21.6°C from the first sensor, then there are 7 more fields (it supports a maximum of 8 sensors) then 8 fields for relative humidity, then some more fields for the optional wind and rain sensors. The rain sensors are quite expensive so I elected to monitor outdoor weather another way. A new data line appears every 3 minutes.

Once the data is in RRD I export it via some php as JSON in the same way as the power sensors.

Weather details

I have a Maplin weather station, which has a USB output. The brilliant pywws software can collect data from the USB device and store that data on the computer. It has a nice flexible templating language to output either graphs as images, or textual information. I wanted the graphs to be produced by Highcharts so they would match all the others on this site, so I needed a way of extracting data from pywws into JSON which I could then parse in the browser. Helpfully a simple pywws template which produces valid JSON is easy to create.

Examples

The template I use for the 24 hour graph is shown below:


{
    "data": [
#timezone local#
#raw#
#jump -300#
#loop 100#
    {
        "#idx "%m/%d/%Y %H:%M"#": {
            "TempOut": #temp_out "%.1f" "null"#,
            "FeelsLike": #calc "apparent_temp(data['temp_out'], data['hum_out'], data['wind_ave'])" "%.1f" "null"#,
            "HumidityOut": #hum_out "%d" "null"#,
            "DewPoint": #calc "dew_point(data['temp_out'], data['hum_out'])" "%.1f" "null"#,
            "WindDirection": " #wind_dir "%s" "-" "winddir_text(x)"# ",
            "WindAvg": #wind_ave "%.0f" "null" "wind_mph(x)"#,
            "WindGust": #wind_gust "%.0f" "null" "wind_mph(x)"#,
            "WindChill": #calc "wind_chill(data['temp_out'], data['wind_ave'])" "%.1f" "null"#,
            "Rain": #calc "data['rain']-prevdata['rain']" "%0.1f" "null"#,
            "AbsPressure": #abs_pressure "%.1f" "null"#
        }
    },
#jump 3#
#endloop#
#! This is needed because the above loop creates a traliing comma
#! on the last run through the loop
    {}
    ]
}

When pywws receives some data, it runs this template and produces a file which is valid jason, the file for the last 24 hours can be seen here.

Once I had that it was a simple matter of plugging this into the JS in the page to generate the graphs.

To Do

The one exception to this is the windrose, currently I'm using the PNG created by pywws, however highcharts can produce Wind roses (example here), I just need to produce a pywws text template which ouputs data in a format suitable for highcharts.

Fitbit

I have a FitBit One which is a bluetooth enabled pedometer, it sends data to a little receiver plugged into my iMacs usb port, or syncs with the phone on my iPhone. Whenever it syncs it pushes all the data stored on the device to the Fitbit servers.

Using their API I wrote some PHP to grab the activities for each day and save them into a directory on my computer. The API returns JSON, which I simply dump out to a text file, one file per day. I then have another php script which reads the steps and floors climbed out of all the JSON files and stores all the data as a single array into memcache. To draw the graphs you see on this page the JS in the page hits another php page which reads the array out of memcache and returns a JSON representation of it. From then on its simple Highcharts JS code to plot the chart.

Design

You might be wondering why I store the data as JSON in flat text files, when I could have stored it in a database. Well the simple answer is: I'm lazy. I don't really know what to do with the data - I just like the idea of keeping hold of it. If I were to push the sections of the data I care about into a database, I'd have to back it up, and if I ever want to make use of some of the other fields in the JSON, I'd have to adjust the schema. Additionally if FitBit ever add fields to the data returned by the API then I'd have to keep track of these and keep adjusting the schema. By saving the raw JSON I don't have to worry about any of that, all I have to deal with is backing up a few thousand files on disk, which is simple. I actually push them into a local Git repository. Keeping a few thousand small files backed up is nice and simple and I can easily parse the data when I need it.

The down side is I have to parse N files to get all the data I want to draw a graph, this is where memcache comes in. Once a day when I grab the previous days data I do the following:

  • Fetch yesterdays data
  • Save it to a text file
  • Create an array in memory
  • Loop through all text files
    • Push the date, floors and steps into the array
  • Push that array into memcache as a single name=val entry

Nice and simple, and I don't have to keep track of databases and schemas, I know (for example in the JS in the html page) that a single call to a URL returns all the data I need from memcache without having to hit the filesystem

Energy usage

Nothing very technical about this graph, but I've been keeping a note of my usage for nearly a year now so I thought I'd graph it. The data is just stored in a CSV file, which the PHP parses up into a table. Then I use a useful Highcharts module which can source data for a chart from a table.

There are two tabs, one is the source data table and the other the graph. I expect if I keep up the datalogging I'll have to do something about the columns being too small, but I've got a while to go yet!