Home Alarm Analytics With AWS Kinesis

Home security system projects are fun because everything about them screams "1980s legacy hardware design". Nowhere else in the modern tech landscape does one program by typing in a three digit memory address and then entering byte values on a numeric keypad. There's no enter-key -- you fill the memory address. There's no display -- just eight LEDs that will show you a byte at a time, and you hope it's the address you think it is. Arduinos and the like are great for hobby fun, but these are real working systems whose core configuration you enter byte by byte.

The feature set reveals 30 years of crazy product requirements. You can just picture the well-meaning sales person who sold a non-existent feature to a huge potential customer, resulting in the boolean setting that lives at address 017 bit 4 and whose description in the manual is:

ON: The double hit feature will be enabled. Two violations of the same zone within the Cross Zone Timer will be considered a valid Police Code or Cross Zone Event. The system will report the event and log it to the event buffer. OFF: Two alarms from the same zone is not a valid Police Code or Cross Zone Event

I've built out alarm systems for three different homes now, and while occasionally frustrating it's always a satisfying project. This most recent time I wanted an event log larger than the 512 events I can view a byte at a time. The central dispatch service I use will sell me back my event log in a horrid web interface, but I wanted something programmatically accessible and ideally including constant status.

The hardware side of the solution came in the form of the Alarm Decoder from Nu Tech. It translates alarm panel keypad bus events into events on an RS-232 serial bus. That I'm feeding into a Raspberry Pi. From there the alarmdecoder package on PyPI lets me get at decoded events as Python objects. But, I wanted those in a real datastore.

The alarmdecoder package provides serial-over-IP support I could have used to send not-yet-decoded events to a lambda function, but this is the cloud API era: if your bytes on the wire aren't JSON encoded, you're not wasting enough of them. I started out planning to send the events over SNS, but AWS has doubled down on their Kinesis brand with Streams, Firehose, and Analytics, so I started there.

Kinesis Streams provide real time ingestion, guaranteed ordering, and capacity auto-scaling, but I have one home not thousands. Kinesis Analytics provide event manipulation, filtering, and fan out, but at the expense of always-running ec2 nodes. Kinesis Firehose was the right fit. It takes single or batched events in HTTP POSTs and stashes them in either elasticsearch, Redshift, or directly into S3.

Relaying the data to Kinesis Firehose took a simple python event handler:

def send_message(sender, message):
    """
    Send message events from the AlarmDecoder to Kinesis.
    """
    message_json = json.dumps(message.dict(), sort_keys=True,
            default=datetime_formatter)
    response = firehose.put_record(DeliveryStreamName=STREAM_NAME,
            Record={'Data': message_json + "\n"})
    log.debug("Sent %s => %s", message_json, repr(response))

The real ad2kinesis code has actual error checking and stuff. The only trick there was getting the trailing newline on the JSON records. Without it Kinesis Streams will correctly intepret each object as a separate event record, however Kinesis Firehose (the cheap one) batches them into files in S3, and AWS's other tools will only read the first event per file. I have Firehose configured to flush to file every sixty seconds, which means I'd only get one event a minute.

And where am I consuming those events? For now in AWS Athena, which is their SaaS Presto offering. I can quickly, and paying per-query run SQL against the million events I've logged in the last month. If I want to start reacting real-time I can change Firehose to Streams and Analytics. If I want faster queries I can load the S3 data directly into Redshift or elasticsearch.

I'm doing nothing more than playing with this, but clearly one could set up a truly modern alarm monitoring company in the cloud without banks of modems and with a first rate software offering. The pieces are all there.

Published: Jan 18, 2017, 5:19:29 AM - Modified: May 1, 2019, 3:24:36 AM by Ry4an Brase
Tags: python, ideas-built, software, home, security
Permalink - Source code