In this project, we have developed a rather robust means of processing and displaying large sums of IoT data using several cutting-edge, industry-standard technologies. Our data pipeline integrates physical sensors that send various environmental data like temperature, humidity, and pressure. Once created, the data is then collected at an MQTT broker, streamed through a Kafka cluster, processed within a Spark Cluster, and stored in a Cassandra database.
In order to test the rigidity of the pipeline, we also created virtual sensors. This allowed us to send an immense amount of data, which wasn’t necessarily feasible with just the physical sensors. The web interface allows users to create as many of these virtual sensors as testing requires.
Once the data goes through the pipeline, it is made viewable on the same web interface. Users can search for key sensors, look through important data, and analyze as necessary.
Our IoT pipeline enables seamless data flow and near real-time analytics. Using industry-standard technologies allows for scalability and reliability, making it suitable for all sorts of data-intensive applications
