Building an Open End-to-End Internet of Things Architecture

by | Dec 19, 2017 | Edge Computing

Designing, implementing, securely operating, managing and maintaining IoT projects is complex. In fact, there are entire organizations whose sole mission is solving a specific problem within an IoT architecture. The problems that can be found within such architectures can range from connectivity to figuring out where apps live.

Here are just some examples:

  • Solving the daunting challenge of how to connect heavy machinery over OPC-UA or modbus to the internet or an intranet network, via controllers and gateways (you can consider yourself lucky if they speak any sort of standard protocol)
  • Handling intermittent connectivity in trucks, trains, or airplanes over cellular or GPRS (let alone trying to update an application over the air or patch a remote edge system)
  • Predicting when something is going to break before it happens
  • Correlating telemetry with contextual data to generate better models
  • Storing different information in different databases (NoSQL, time series, fast access, in memory)
  • Making sure your architecture is scalable, flexible, and can be deployed anywhere
  • How to abstract your developers from where their applications will be running in this chaotic highly distributed world

But these are only pieces to the bigger puzzle of designing an end-to-end IoT system where connected devices, business processes, DevOps practices, and people collaborate together across four different complementary, but most-of-the-time disconnected, worlds:

  • Operational Technology (OT)
  • Information Technology (IT)
  • Analytics and Machine Learning (ML)
  • Traditional and modern application development

Why an IoT Architecture has to be Open

Connecting devices; ensuring communication protocols are translatable; verifying accuracy and security features are running across the entire network; capturing, managing, analyzing, and using data to create better business outcomes; integrating operational data with existing informational technology systems and applications; and gaining intelligence at the edge are all parts of the functionality of an IoT architecture as depicted below:

Based on our experience with customers deploying IoT solutions, real production scenarios need more than a single provider to successfully accomplish the functionality in the above diagram.

In our experience, successful implementations have used a combination of technology providers, network providers, security specialists, developers, and system integrators, all operating as a cooperative, open, modular, and flexible team.  

A breakdown of these functionalities in a deeper level is shown here:

Putting Together OT, IT, and Analytics the Open Source Way

At Red Hat, we firmly believe that the only way to address these complex scenarios is via the open source way. Thus, we have been working in the Eclipse IoT Working Group community for the past 3+ years with our partners, creating code at the different layers within the end-to-end architecture. By tying together Apache Camel and Apache ActiveMQ with Eclipse Kura and Eclipse Kapua, we are able to help define an open pluggable framework. This framework can run on small boards that can be embedded in vehicles or on higher-end 4-Gb intelligent gateways to process, aggregate, typify, and send over the best network different messages according to their priorities. The more resources we have available at the edge, the more advanced analytics and machine learning models we can run closer to the “things,” and take automatic decisions using rules and Complex Event Processing (CEP) techniques.

MQTT, AMQP, and Kafka are first-class citizens for the bidirectional communication between the OT world and the IoT integration hub, that provides a microservices framework with a goal of enabling more secure device management and some basic data management with ElasticSearch, while giving you the option to move the data, logs and events to a more centralized platform to store and process data with different Apache projects like Spark, SparkML, Kudu, and Impala.

All of this comes together thanks to an open, hybrid, cloud-ready container application platform: Red Hat OpenShift. OpenShift runs Linux containers at scale, open APIs are defined to connect to different systems, and IoT and data services are offered to the application layer for developer teams to do rapid prototyping and shorten development times from months to weeks. This innovation flow needs to be aligned with the operations teams that need to operate the end-to-end system and make sure continuous integration (CI), continuous delivery (CD) and security best practices within the organization are applied to this new world.

A view of all the projects together and how they fit within the architecture is shown in this image:

In the next blog post on IoT, we will delve deeper into the technology in each module of the architecture, beginning with the gateways.