SPARK is an open source in-memory cluster computing framework getting so much attention these days, just like Hadoop. Initially it was developed in UC Berkely AMPLab and now an Apache Incubator Project.

MQTT is one of the widely used protocol for ‘Internet of Things’. This protocol is appropriate for IoT as anything and everything is getting connected to internet and they all produce data. Researchers and companies predict some 25 billion devices will be connected to the internet by 2015. Plugin/Support for MQTT is available in popular MQs like RabbitMQ, ActiveMQ etc.

IoT & BigData research team in Cybersecurity have written a Streaming adapter for MQTT. It is now integrated as part of the latest release SPARK-0.9.0. This new Adaptor will now support pull requests for MQTT in SPARK which in turn will help developers to build Internet of Things (IoT) applications using SPARK Streaming for their real time data processing needs (from sensors and other embedded devices etc).

Word count example using MQTT adapter is now available in SPARK root folder https://github.com/apache/incubator-SPARK/blob/master/examples/src/main/scala/org/apache/SPARK/streaming/examples/MQTTWordCount.scala.

Key Contributor for the same was Prabeesh K from the BigData Analytics Team.

February 11, 2014
Center for Cybersecurity, Amritapuri