KTWIN: A Serverless Kubernetes-based Digital Twin Platform

A Digital Twin is a virtual representation of a physical system or process, designed to facilitate bidirectional communication between the real-world system and its digital counterpart [6]. DTs enable the development of value-added services, such as real-time monitoring, predictive maintenance, optimization, and simulation, that can be adopted by a variety of industries such as Manufacturing, Smart Cities, Energy and Healthcare.

In the context of the Manufacturing area, a DT can be utilized to solve several field problems such as optimizing production processes, monitoring equipment health, applying optimization to the supply chain process, and performing predictive maintenance activities. Sierla et al. [7] build a Digital Twin to automate assembly planning and orchestrate the production resources in a manufacturing cell based on digital product descriptions. The concept introduced in the research is presented in general terms in UML (Unified Modeling Language) and the implementation provides a 3D simulation environment using Automation Markup Language for digital product descriptions. Tao an Zhang [8] propose a Digital Twin Shop-floor (DTS) providing an effective way to reach the physical-virtual convergence in the manufacturing industry and improve production efficiency, accuracy, and transparency. The system components are broken down into a Physical Shop-floor (PS), Virtual Shop-floor (VS), Shop-floor Service System (SSS), and Shop-floor Digital Twin Data (SDTD). The paper also presents the key technologies required to build the proposed system and the challenges during a DTS implementation process for future studies. It lacks an actual implementation and evaluation of the proposed solution.

The modernization of cities opens the adoption of Digital Twin technologies to improve problems of medium-sized cities around the globe. Digital Twins can be applied to improve public transportation systems and address mobility problems, enhance the decision-making process in urban planning, monitor and predict environmental and resource management such as carbon emission and waste reduction, or detect anomalies and optimize operations in power and water plants. Dembski et al. [9] present a novel Digital Twin case study applied in the town of Herrenberg, Germany. The research presents a prototype and the deployment of sensor devices in different city locations to collect data. The use case prototype includes a 3D city model, a mathematical street network model, urban mobility simulation, airflow simulation based on collected environment data, sensor network data analysis and people’s movement routes. The prototype allows developers to easily extend the urban digital flow by adding new code and modules in a visual interface. Ford and Wolf [10] exploit Smart Cities with Digital Twins (SCDT) for Disaster Management Digital Twin system for Smart Cities. The authors present and test a conceptual model of a SCDT for disaster management and discuss the issues to be addressed in the development and deployment of SCDT for disaster management.

Digital Twins can improve the energy industry by providing valuable insights into system performance and optimization, reducing power downtime, providing efficient energy planning to fit the supply and demand, as well as optimizing energy household consumption. In this context, Fathy and Jaber [11] propose a data-driven multi-layer Digital Twin of the energy system that aims to mirror the actual energy consumption of households in the form of a Household Digital Twin (HDT). The model intends to improve the efficiency of energy production, modeled as an Energy Production Digital Twin (EDT), by flattening the daily energy demand levels. This is done by collaboratively reorganizing the energy consumption patterns of residential homes to avoid peak demands while accommodating the resident’s needs and reducing their energy costs. In addition, Gao et al. [12] present a Digital Twin-based approach to optimize the operation of an automatic stacking crane handling containers regarding energy consumption. The article developed a virtual container yard that syncs real container yard information to improve automatic stacking crane scheduling. Then, a mathematical model followed by a Q-learning-based optimization step is applied to minimize the total energy consumption for completing all tasks. The output can be used by managers and operators to choose the appropriate strategy to accomplish energy-saving and efficiency goals. In the Healthcare ecosystem, Digital Twins can be applied for many use cases such as creating a model replica of a single patient for health monitoring and treatment planning, simulating and testing a medical device during the design phase or even modeling an entire healthcare system to monitor to analyze vaccination campaigns, for instance. Pilati et al. [13] developed a Digital Twin which integrates the physical and virtual health systems to create a sustainable and dynamic vaccination center focusing on using the minimum space and resources while guaranteeing a good service for patients. The research implements a discrete event simulation model to try to find the best configuration in terms of human resources and queues for a real vaccination clinic to maximize the number of patients vaccinated in the shortest period.

In the context of health monitoring and treatment planning, Liu and Zhang et al. [14] propose a novel cloud-based Digital Twin healthcare framework for elderly patients (CloudDTH). The solution plans, monitors, diagnoses and predicts the health of individuals by data acquired via wearable medical devices, toward the goal of personal health management. The proposed implementation can be divided into three phases: crisis early warning, real-time supervision, and scheduling and optimization. The data utilized to monitor and simulate individual health includes data acquired from physical medical wearable devices from elderly patients (e.g. heartbeat, body temperature, blood pressure), external environmental factors such as temperature change, and medical records.

In the general context of Digital Twin platforms Eclipse Ditto [15], Microsoft Azure Digital Twins [16] and AWS IoT TwinMaker [17] are the primary references. The Azure Digital Twin is a Platform as a Service (PaaS) offered by the Microsoft Azure suite that enables organizations to create digital replicas of physical environments, systems, and processes. Microsoft’s Digital Twin platform leverages the power of the Azure cloud and integrates with a wide range of other Microsoft services and tools, such as IoT Hub, Event Hubs, Event Grids, and Service Bus. Azure Digital Twins service allows to model entities and their relationship in a JSON-like language called Digital Twin Definition Language (DTDL) forming a conceptual graph that can be visualized and queried in the Digital Twin Explorer.

Amazon Web Services offers the AWS IoT TwinMaker service to build operational digital twins of physical and digital systems. AWS IoT TwinMaker allows organizations to create digital visualizations using measurements and analysis from a variety of real-world sensors, cameras, and enterprise applications to help you keep track of your physical factory, building, or industrial plant. The platform also provides tools to model your system by using an entity-component-based knowledge graph composed of entities, components, and relationships using a proprietary JSON-like language. In addition, the TwinMaker service allows configuring and loading data from time-series data sources, and it builds 3D visualizations on top of it.

Eclipse Ditto is an open-source framework for Digital Twins maintained by Eclipse Foundation. Eclipse Ditto focuses on connecting physical devices to their digital counterparts, enabling interaction and synchronization between both. It allows device virtual representation encompassing their state, properties and behaviors. Ditto supports multiple protocols for interacting with digital twins, allowing users to query, modify, and subscribe to changes in the twin’s state.

The existing PaaS solutions offer a wide variety of features, but they do not provide enough flexibility to deploy Digital Twin solutions to different landscapes, such as edge locations, and limit organizations to work with hybrid or multi-cloud environments. Moreover, they mostly are vendor-specific implementations, reducing interoperability and reusability [6], and limiting organizations to move their solutions to another vendor stack without a considerable amount of rework. Lastly, all the services mentioned above offer less control over operations and infrastructure, preventing users from personalizing settings, and monitoring application health and events propagation. In this context, an open-source Digital Twin platform built with Cloud Native solutions can offer flexibility and independence to organizations to build microservice-based distributed systems to leverage more Digital Twin use cases.

Understanding the distinct needs of the personas involved in the design, development, and operational lifecycle of Digital Twins is instrumental in shaping KTWIN’s architectural requirements and in ensuring that the platform is tailored to meet their specific challenges and objectives. Therefore, we defined KTWIN revolving around three primary personas, described as follows:

• 1.

  The DT Domain Expert is primarily responsible for the strategic direction and governance of the Digital Twin ecosystem. Their responsibilities include defining the functional requirements by modeling entities and describing their behaviors, often leveraging an ontology-based language to ensure precise representation of domain-specific entities. This requires a system capable of adapting to diverse domains through open and flexible modeling tools.

• 2.

  The DT Developer focuses on implementing and validating services that encapsulate the domain logic defined by the DT Domain Expert. The developer seeks for abstractions such as Function as a Service (FaaS), service containerization, and the usage of Software Development Kits (SDKs), to streamline service creation and deployment.

• 3.

  Lastly, the DT Operator responsibilities include the deployment and maintenance of the Digital Twin solution, combining domain requirements with system-level configurations. Their role requires an unified, vendor-neutral specification language that enables seamless deployment across cloud, on-premise, and edge environments.

This section explains the high-level architecture definition proposed for KTWIN, which can be divided into Control Plane and Application plane components as depicted in Figure 1. The Control Plane components allow human operators to define the Digital Twin entities and their relationships through a generic resource definition file, allowing the deployment of DTs of any domain knowledge. The human operator interacts with the KTWIN Operator which is responsible for orchestrating the creation of all the underlying components and services from the Application Plane required to deploy the Digital Twin scenario. The operator contains the KTWIN-related resources knowledge, meaning that it knows all the steps and underlying resources required to instantiate a new twin within the platform. The Operator executes creation, modification and deletion requests against the Orchestrator component.

The Orchestrator applies and maintains the desired resource states defined by the operator which incurs creating a new Twin Service, maintaining the expected number of Event Dispatchers, or keeping the Event Broker routing rules up-to-date. The configured resources are persisted into the Orchestrator storage. In addition, the definition of the resource model includes domain-related entities, their corresponding data model attributes, the different instances of an entity, and their relationships with other entities. The entity’s relationships compose a Twin Graph. Because Application Plane components require the Twin Graph information to process and propagate events to related entities, the graph data is replicated to a fast access cache storage. Hence, Application Plane components can consume the graph information through the Twin Graph Service, decoupling the control plane storage from the application plane components and offering higher scalability and faster response time.

The Application Plane components are responsible for processing the Digital Twin events and storing them for further data analysis and visualization. Real-world edge devices interact with KTWIN by sending events that represent a state change in some defined entities of the Digital Twin. Devices such as an actuator can also subscribe and receive events from KTWIN indicating that some action must be executed in the edge. The events are published to or subscribed from the Event Broker through the Gateway component. The Gateway acts as a load balancer from ingress requests and redirects the traffic according to the protocol or endpoint: while HTTPS requests are redirected to the Event Broker Ingress which publishes them to the Event Broker, AMQP and MQTT connections are directly forwarded to the Event Broker. Incoming broker requests are routed to the corresponding target based on in-broker routing rules that send events to different in-broker queues and are subscribed by Event Dispatchers. The Event Dispatchers are lightweight components responsible for subscribing events and dispatching them to the corresponding Twin Service service deployed by the end user.

The Twin Service are designed and deployed as a Serverless function that contains a custom implementation of some Digital Twin domain defined by the DT domain expert. Each Twin Service is associated with a real twin and is responsible for processing one more event type. Twin Services can communicate with its real-world counterpart or with another Twin Service in which it has a relationship. In addition, they can be scaled down to zero or up to thousands of instances, offering improved performance and efficiency in resource usage.

The Event Store is a highly scalable data store designed to save the large number of events generated by real-world devices and processed by Virtual Services. Events persisted to the Event Store are queued with the broker and subscribed by the Event Store Dispatcher that dispatches them to the Event Store Service. In addition, end-users can implement Twin Service services to obtain the latest status of some twins using Event Store Service APIs. The Event Store APIs also allow external systems to fetch persisted events in batches.

KTWIN provides support for clients to publish events using a variety of protocols, such as MQTT, AMQP, and HTTP. In this context, the KTWIN Dispatchers are event dispatchers built to implement protocol conversion. For instance, they convert real-world MQTT events published by devices into in-cluster events, as well as in-cluster events back to MQTT device events. They are required because KTWIN uses a set of event header metadata information to route messages to the corresponding target service, which cannot be implemented using the MQTT4 protocol. In contrast, the device’s AMQP-generated events can be directly published into Event Broker since AMQP offers routing headers support. Lastly, the Event Broker Ingress is a request ingress service used by Twin Services or real devices to publish events into the Event Broker, not requiring them to establish a stateful connection with it, for instance, HTTP. More information about these components and their implementation is provided in the next section.

KTWIN provides resources and data abstractions for Digital Twin operators to define their own Digital Twin models and relationships allowing the deployment of any DT domain. It uses the open-source Digital Twin Definition Language (DTDL) as the base entity definition framework and implements twin message routing rules on top of these definitions. In addition, KTWIN enhances the existing specification language by adding some system’s related information that can be configured to customize the deployment settings of a Digital Twin use case.

The KTWIN resource definition framework is divided into two main definition resources: Twin Interface and Twin Instance. The Twin Interface describes the attributes, relationships, and other types of contents common for any Digital Twin of that type, being reusable. A Twin Instance is an instance of some interface in the real world. In this context, a Twin Interface Car has attributes, such as color, year, and length, and relationships, such as owner and wheels. A Twin Interface can be created for each car that belongs to a Digital Twin, where each vehicle is unique and has its corresponding representation in the real world. In addition, users may also define system-specific settings such as allocated resources and auto-scaling attributes.

The Twin Interface definition allows domain experts to describe properties, relationships, commands, and inherit a parent interface. The Twin Instance definitions allow users to create Instances of interfaces, and configure relationships with other instances. Tables 1 and 2 show the fields in each resource specification.

The resources described previously are used by KTWIN Operator to build an event routing mechanism without any manual intervention from the DT operator. The messaging routing rules support the following communication flows: (1) an event generated by the real device instance and sent to the Twin Service service, (2) an event generated by the Twin Service instance and sent to the corresponding real device, (3) an event generated by some Twin Service and sent to another Twin Service (these twins must have some relationship), and (4) an event generated by some Twin Service and published to Event Store. KTWIN uses the Twin Interface and Twin Instance definitions as well their relationships to build event propagating and routing built-in rules within the Event Broker.

KTWIN divides events into four subcategories which identify the context in which an event happened: real, virtual, command, and store. The subcategories information is included in the event message, together with the Twin Interface and the Twin Instance identifiers to identify the source of the event and determine its targets. In case of an MQTT connection, real devices publish events to the Event Broker topic containing the Twin Interface and the Twin Instance information the event is associated to, such as ktwin.real.<twin-interface>.<twin-instance>. For HTTP and AMQP requests, this information is provided in the message headers. Events published by devices following the previously mentioned pattern are routed to the corresponding Twin Service. The same devices can subscribe to events generated by its Twin Service counterpart in the topic ktwin.virtual.<twin-interface>.<twin-instance>.

Twin Service with a common relationship in the twin graph can propagate events to each other using the command event category, with the following pattern in the message header: ktwin.command.<twin-interface>.<twin-instance>.<command-name>. In addition, events that are supposed to be routed to the event store for persistence use the following routing header: ktwin.store.<twin-interface>.<twin-instance>. Finally, in case some Twin Interface does not require computing any information with the generated device event, users may configure KTWIN to route real events directly to the Event Store using the routing and persisting settings in the Twin Interface specification. Figure 2 shows how the routing implementation works with hypothetical examples of A and B entities.

Although the routing keys contain information about the Twin Instance and the Twin Interface, the in-broker routing is mostly done at the interface level, thus reducing the number of routing rules required within the broker. In this context, the routing is implemented using a start-with logic, such as ktwin.<event-category>.<twin-interface>.*. Once the user-implemented service receives the event, the routing key is decoded from the message, so the application logic knows the instance owner of that message. The application later uses this information for data propagation to the corresponding related entities using command events or data persistence in the event store. In this context, in case some command event must be generated, the application must have access to the graph relations. The previously described steps introduce some complexity for end-users to implement this logic, hence KTWIN offers a Software Development Kit (SDK) to be imported by end-users when writing their functions. The SDK implements the routing key’s decoding and command-based event propagation based on twin graph data in a user-friendly way.

The Digital Twin Definition graph defined by the domain experts is stored within the Control Plane store and maintained by Orchestrator. It allows DT operators to easily manage and visualize the defined Twin Interfaces, their corresponding instances, attributes and commands. However, the Twin Graph information must also be available for the application plane services implementations during the event processing. Hence, the Digital Twin graph data is replicated and cached in the Twin Graph Cache to avoid impacts of Application Plane incoming requests to Control Plane components, reducing the response time and minimizing the control plane disruption. The graph cache implementation uses a key-value in-memory data store.

Application Plane twin events are persisted into a scalable and reliable Event Store database. Events are stored in a time series format containing the history of all events generated for a specific instance during its lifespan. Events are asynchronously published to Event Broker and routed to the Event Store Dispatcher before being persisted in the database by the Event Store Service. In addition to the event payload, the persisted data includes the timestamp in which the event occurred, and the Twin Instance and Twin Interface that the events belong to.

In addition to data persistence, the Event Store Service provides APIs to consume past events persisted in the data store. User-defined services can query the latest status of some Twin Instance stored and perform business logic using this data while processing incoming events. Additionally, external applications can consume a batch history of events of a specific Twin Instance or query by events of several instances of the same interface for further data analytics and machine learning. Data analytics and machine learning discussions are out of the scope of this initial research and are future research opportunities for KTWIN use cases.

The KTWIN implemented prototype embraces Cloud-Native technologies allowing end-user applications to be implemented, deployed, and orchestrated as containerized microservices providing resilience and scalability. Figure 3 shows the building blocks of the implementation evaluated in this investigation.

The main personas involved in KTWIN development and operational process are: DT Domain Expert, DT Developer and DT Operator. The DT Operator creates the Custom Resource Definition (CRD) files to model the Digital Twin use case within KTWIN and apply the resources using Kubernetes CLI. The CRD files are based on the ontology definition originally provided by the DT Domain Expert. The provided files are processed by KTWIN Operator, developed in Golang using Kubebuilder, an open-source SDK library that facilitates seamless interaction with Kubernetes Control Plane APIs. These CRD files are stored in Kubernetes etdc storage, ensuring reliable and consistent data management. The KTWIN Operator creates the required Application Plane resources to instantiate the defined use case. The container-based functions implemented by the DT Developer are pulled from a container registry and deployed as a Twin Service. Finally, the DT Domain Expert can analyze the processed data using some visualization tool that fetches the data from the Event Store.

The Twin Graph Service pulls the defined graph stored in etdc storage through Kubernetes Control plane APIs, split into subgraphs of Twin Interfaces that contain all Twin Instances and their relationship and caches in Redis, an in-memory key-value storage. The cache service exposes the subgraph data via Restful API so user-defined services can fetch the graph information of a particular Twin Interface while the service starts up. The subgraph content must be immediately available and provide fast data access to reduce impacts on the cold start time of the service.

KTWIN Application Plan components are implemented on top of Knative [30], an open-source Kubernetes-based platform designed to build, deploy, and manage modern serverless workloads. Knative extends Kubernetes to provide a set of middleware components that enable the deployment and management of serverless applications. It is the core foundation of Cloud Run in Google Cloud Platform (GCP) bringing together the best of both Serverless and containers, such as increasing the developer’s productivity by three times [31].

The Event Broker, one of the core components of KTWIN, was implemented by instantiating a Knative Broker resource, which deploys a RabbitMQ broker and provides an endpoint for event ingress to which producers can post their events. RabbitMQ is a scalable and reliable messaging broker [32] able to handle millions of concurrent IoT connections [33]. In addition, RabbitMQ supports open standard protocols, including AMQP and MQTT, and provides MQTT-AMQP protocol interoperability [33] and header-based routing.

The Twin Services were implemented as independent application containers and deployed using the Knative Service custom resource. The CRD creates the required Kubernetes resources to allow scaling up and down multiple replicas of the container and exposes an HTTP endpoint within the cluster. The Twin Service event subscription was implemented using a Knative Trigger custom resource. A Trigger allows users to define to each service an event that will be forwarded based on some message metadata. The routing rules are implemented within the Event Broker using the RabbitMQ Topology Operator and use Cloud Event [34] specification metadata fields in the routing rules. Knative Dispatchers subscribe to events in the Event Broker queues and post them to the corresponding Twin Services based on the provided routing rules. Twin Service generated events are published to the created event Event Broker Ingress and routed to the corresponding subscriber.

The Event Store Dispatcher is a standard subscriber created during the KTWIN installation steps that subscribes to events with Event Store as the target. These events are posted to the Event Store Service and persisted in a ScyllaDB cluster. ScyllaDB is a NoSQL distributed database, compatible with Apache Cassandra, designed for data-intensive applications requiring high performance and low latency [35]. Scylla database was deployed using the official ScyllaDB Operator [36] to automate the NoSQL cluster deployment process and operational tasks such as scaling, auto-healing, rolling configuration changes, and version upgrades.

The implemented prototype also includes a set of MQTT publisher’s applications developed in Golang and deployed as containers to produce IoT-generate data used during the evaluation scenarios. The containerized applications allow the creation of a configurable number of threads allowing the execution of thousands of MQTT publisher devices within a single container without exceeding Kubernetes pods limits per node. Although KTWIN allows devices to publish messages in different protocols, such as HTTP Cloud Event-based messages or AMQP, the implemented devices use MQTT because it is widely adopted in IoT systems due to its lightweight design [37].

The devices publish MQTT events to RabbitMQ Event Broker which provides AMQP-MQTT protocol interoperability. However, RabbitMQ does not support MQTT routing based on Cloud Event specification headers. To address this limitation, KTWIN Dispatchers convert the MQTT message routing, originally based on RabbitMQ routing-key routing, to a header key routing rule. This conversion ensures that MQTT edge device messages are properly converted to Cloud Events. The same limitation does not exist for AMQP or HTTP messages, since they are compatible with Cloud Event spec. This limitation in RabbitMQ opens up possibilities for future enhancements in the proposed implementation.

Finally, KTWIN provides observability capabilities and allows DT Operators to monitor the service’s health in Grafana dashboards. The Grafana dashboard built for this prototype displays metrics scraped by Prometheus including event throughput per event type, application CPU and memory consumption, and response time.

A Kubernetes Custom Resource Definition (CRD) is a way to extend the Kubernetes API by defining and managing custom resource types. KTWIN makes use of CRD to allow Digital Twin Domain Experts to define Digital Twin entities, their attributes and relation, in addition to system-specific definitions, such as allocated CPU or memory and auto-scaling policies. The developed KTWIN prototype enhances the open-source Digital Twin Definition language (DTDL) by adding system-specific configuration to the ontology-based definition language. The implemented prototype has two DTDL-based CRDs: Twin Interface and Twin Instance.

The Twin Interface custom resource represents some Digital Twin entity and it describes its name, attributes, relationships, commands, service auto-scaling policies, the related container image that implements the interface behavior and the amount of memory and CPU requested for the container. The Twin Instance is an instance of some interface in the real world. The instance custom resource contains the instance name, the corresponding Twin Interface identifier, and references to the related Twin Instances to which there is a relationship. Both custom resources are defined using the YAML format and can be applied to the cluster using the Kubernetes standard client kubectl.

The KTWIN Operator was built on top of Kubebuilder which provides an SDK to interact with Kubernetes Control Plane APIs. When creating a new KTWIN custom resource instance, it triggers the creation of a variety of Kubernetes resources in the Application Plane. These steps are orchestrated by the KTWIN operator and Kubernetes Control Plane APIs. The implemented solution uses RabbitMQ Topology Operator that allows the creation of in-broker resources such as Queues, Bindings and exchanges using Kubernetes Custom Resources. The KTWIN operator uses these custom resources to instantiate the required broker elements to implement the routing rules based on the provided Digital Twin entities and their relationships and commands.

KTWIN events definition and routing rely on in-broker routing mechanisms. Figure 4 shows how KTWIN enhances the routing rules implemented by Knative using RabbitMQ exchanges, queues and bindings. The Broker’s internal resources are created using the RabbitMQ Messaging Topology Kubernetes Operator which manages RabbitMQ messaging topologies using Kubernetes Custom Resources.

In Figure 4, it is possible to visualize that devices publish events to the MQTT topic ktwin.real.<twin-interface>.<twin-instance>. RabbitMQ natively supports MQTT-AMQP interoperability by emulating the MQTT broker via a unique MQTT Topic Exchange, so all publishers and subscribers are virtually connected to the same exchange with a routing key as ktwin.real.<twin-interface>.<twin-instance>. KTWIN creates routing key-based bindings in the MQTT Topic Exchange to redirect messages to the MQTT Dispatcher Queue for each Twin Interface created by the Operator. The MQTT Dispatcher subscribes to these routing-key-based messages, converts them to a Cloud Event message, and republishes them to the Broker Exchange. The Broker Exchange uses the Cloud Event type header to redirect messages to the corresponding queue and subscribers. The broker exchange bindings include routing rules for (1) events generated by the real instance and redirected by its virtual instance service, (2) events generated by a Twin Service and sent to another Twin Service (Command), (3) events propagated to the Event Store, and (4) event generated by the virtual instance and sent to the real instance. The events generated by Twin Services that must be redirected to the real devices are converted from Cloud Event to a routing key message by the Cloud Event Dispatcher, republished to the MQTT Topic exchange, and finally delivered to the device subscribed to the MQTT topic ktwin.virtual.<twin-interface>.<twin-instance>.

KTWIN provides a Software Development Kit (SDK) for handling events and propagating them to edge devices, event stores or related virtual services. The SDK implements all the complexity of event handling using the Twin Graph entity relation data, allowing end users to focus on the application logic. KTWIN provides an SDK version in both Golang and Python programming languages. An SDK implementation example is provided in the following sections.

The Twin Graph defined by end-users is stored as CRDs in the Kubernetes etcd storage, an open source, distributed, consistent key-value store for shared configuration distributed systems or clusters. The graph data are replicated and cached in the Twin Graph Redis instance, a key-value in-memory cache providing submillisecond response times. Virtual Services require access to the Twin Graph to publish data entities with relationships. This information is fetched from Twin Graph Service RESTful APIs while the service start-up, hence requiring an ultra-fast response time to reduce impacts in the cold start. The Twin Graph information does not need to be fetched by the end-user application code, since KTWIN SDK implements this logic and provides a user-friendly interface to consume these data.

The Event Store database stores the entire history of Twin Instances events during its life cycle. Event Store was implemented using ScyllaDB, an open-source NoSQL database for data-intensive apps that require high performance and low latency. ScyllaDB provides high availability, scalability, and efficient resource utilization, being used for a large number of use cases, including real-time analytics and IoT data storage. User-defined services can consume the latest state of some twin event by Event Store Service. In addition, external systems can fetch a batch of Twin Instance persisted events using the same APIs.

Observability is crucial for understanding the health, performance, and behavior of distributed systems. KTWIN Observability was implemented using Prometheus and Grafana. Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It scrapes metrics from various services and stores them in a time-series database, enabling real-time monitoring and alerting. Grafana is an open-source analytics and monitoring platform that allows users to define and configure dashboards to visualize Prometheus metrics. Together, Prometheus and Grafana provide a powerful observability stack that allows KTWIN users to gain insights into the system, detect issues early, and ensure optimal performance.

KTWIN provides a dashboard that allows the DT Operator to monitor the number of different event types processed, the CPU and memory usage of the service, as well as its response time. The dashboard was built and used during the evaluation scenarios of this investigation. Prometheus scrapes KTWIN components metrics data through the metrics endpoint at regular intervals defined in its configuration. The scraped data is then stored in its time-series database to be consumed by Grafana.

In order to evaluate the proposed solution, a Smart City use case was designed, implemented, and deployed using KTWIN. The selected use case was based on the DTDL ontology for Smart Cities developed by Open Agile Smart Cities (OASC) and Microsoft [38]. The open-source ontology model was publicly released to accelerate the development of Digital Twin-based solutions for Smart Cities. The DTDL is composed of ontology-based definitions from NGSI-LD information model specification [39] and ETSI SAREF ontologies for Smart Cities (Saref4City) [40]. The final model comprises entities in Urban Mobility, Environment, Waste, Parking, Buildings, Parks, Ports, City Objects, Administrative Areas, and more. A summary of the corresponding group of Twin Interfaces created for the evaluation scenarios is presented in more detail as follows.

Administrative Area: set of entities that represent the organizational structure of a city. At the highest level, the City represents a large human settlement comprising administrative regions such as neighborhoods or districts. A Neighborhood is a geographically localized community within a city, suburb, or rural area, functioning as a smaller administrative unit. Neighborhoods can compute and aggregate data from various entities, such as monitoring air quality in specific regions or tracking the availability of public parking spots in designated areas.

Utilities: this category encompasses a set of entities designed to provide essential resources for residents. A Smart Pole is located on a specific street within a neighborhood and is equipped with various sensors to monitor environmental quality metrics, including air quality, noise levels, crowd density, traffic flow, and weather conditions. Additionally, a Smart Pole can integrate with an Electric Vehicle (EV) Charging Station to support sustainable transportation. Streetlights are associated with Smart Poles, enabling the monitoring of lighting levels across different city regions. If a lamp becomes defective, the system can detect it and provide actionable information for timely replacement.

Environment: set of entities used to measure the environment quality observed in certain neighborhood of the city. The Air Quality Observed represents the air quality in a certain region of the city in the determined period. It registers the density of different gases in a certain area such as Carbon dioxide (CO2), Carbon monoxide (CO), and Sulfur dioxide (SO2). The implemented service classifies the Air Quality Index (AQI) [41] of the area into Good, Moderate, Unhealthy for Sensitive Groups, Unhealthy, Very Unhealthy, and Hazardous based on the gas density observed by the sensors. The service also updates the air quality index of the neighborhood interface to which it is related. The weather variables observations of a smart pole region are registered in the Weather Observed interface. The interface contains weather-related information such as temperature, precipitation, humidity, snow height, wind direction and speed, and atmospheric pressure. The implemented service aggregates the measured data and computes additional information such as the dew point, feel likes temperature, and pressure tendency. Lastly, the noise level observed entity contains information about the sound pressure level in a certain region of the city.

Mobility: set of entities used for monitoring and managing urban mobility. The Crowd Flow Observed entity tracks crowd congestion, including the number of people, flow direction, and average speed at locations like subway stations or bus terminals. Traffic Flow Observed monitors traffic conditions, providing data such as vehicle spacing and headway times to assess congestion levels. EV Charging Stations are smart facilities linked to Smart Poles, enabling electric vehicles to charge with status updates like available, in use, or out of service, and compatibility with specific vehicle types. Parking Spots represent designated parking areas, with dynamic statuses (free, occupied, or closed) and can be categorized as Off-Street (e.g., garages) or On-Street (e.g., public roads). Parking Groups optionally organize spots by criteria like floor or street.

Cross-Domain: a set of generic entities that can be shared across different domains. A Device can be any kind of sensor, actuator, or camera that generates data for the system. Devices are associated with the majority of the entities previously listed. For instance, a device was involved in order to measure the air quality of a certain area of the city or to report the parking slot availability. The device interface is designed to monitor different operational data from sensors, for example, each device contains its own battery level that can be measured to monitor the device’s health.

The open source DTDL ontology was converted to KTWIN CRDs files using KTWIN CLI implemented as part of the prototype. Each DTDL entity was mapped to a Twin Interface and a set of Twin Instances, and later applied to the cluster to be provisioned by the KTWIN Operator. The CRDs files mostly include the entity’s data model definitions, such as attributes, data-type definitions, and existing interface relationships. The Smart City design effort also included the creation of additional properties and commands, not available in the original DTDL, to fulfill the events definition scenarios described in Table 3. Finally, a set of container-based applications were developed using KTWIN SDK, a library used to abstract event handling, propagation and persistence within KTWIN. The containers were implemented as stateless functions to handle events for the deployed Twin Instances, and defined in the deployment specification files. The same artifacts included parameterized settings to configure auto-scaling policies and control the allocated CPU and memory. The generated artifacts for the prototype evaluation are available on GitHub ³³3https://github.com/Open-Digital-Twin/ktwin-article.

The evaluation scenario of this research aims to reproduce a set of the rich number of events that happen during a day-in-the-life of a Smart City. The simulation design took into account (1) events that are generated throughout the day on a defined frequency, contributing to a fixed data traffic, and (2) events that are generated at a specific moment of the day and do not follow a well-established frequency or are generated because some event criteria in the system were reached, contributing to unpredictable data traffic. In this context, the environment-measured data such as air quality index, city weather, temperature, and noise level observed are events generated at a fixed frequency throughout the full day. However, events directly related to city mobility, such as observed crowds and traffic flows, changes in parking spot availability, and the monitoring of available electric vehicle charging stations, present variable frequencies during the day depending on the city’s peak and off-peak hours. Another example of events that present unpredictable traffic is the monitoring of the battery level of sensor devices - each device has its battery level consumption and recharge particularities, so it is difficult to predict when this kind of event will be generated. In-system events, such as the Air Quality Index warning alert when some Smart Pole identifies a low-quality air measurement, are also part of the evaluation scenario.

Table 3 shows the events implemented and analyzed in the evaluation scenario with their corresponding time interval variation for peak and off-peak hours. The selected event interval considered for simulation compression was from 24 hours to 24 minutes, which means that a 10-second interval represents ten minutes in the real world, allowing the execution of several evaluation experiments in a reduced timeframe. The experiment divides the day simulation into six windows of four hours each, corresponding to 240 seconds in the simulation time. Each event type may differ in frequency within each time window, allowing the reproduction of the variation of events at different moments of the day, such as a city’s peak and off-peak hours. In addition to the generated MQTT device events, Twin Services generates virtual events back to the edge devices and publishes events for command executions and data persistence.

Table 4 shows all cloud event types implemented and deployed using KTWIN in the evaluation scenarios. Each event type contains the Twin Interface identifier the event is associated with. In addition, each event has a subtype identifier that is used to route the event to the corresponding target. The subtypes are the following: real, virtual, command, and store. While real devices generate events with real identifiers, virtual events are generated by their virtual counterpart. The event with store subtype indicates that the event must be routed to the Event Store, and the command subtype represents some message exchange between two instances with a common relationship. Table 4 also contains the throughput range of events per second published to the Event Broker for the different city sizes deployed in the evaluation scenarios.

The design of the experiment sizing was based on the New York City dataset of Mobile Telecommunications Franchise Pole Reservation Locations [42]. The public dataset contains the locations of street light poles, traffic light poles and utility poles reserved by companies authorized by the New York City Department of Information Technology and Telecommunications (NYCOTI). The report describes the number of poles installed in each of the five main boroughs: Manhattan, Brooklyn, Queens, Bronx, and Staten Island. Regarding the city mobility metrics, the number of off-street parking spots and electrical vehicle charging stations for the evaluation scenario considered public data maintained by the NYC Department of Transportation (DOT) [43].

In this context, the prototype evaluation of this study used the previously mentioned New York City public data to define the different sizing scenarios for each Twin Interface part of the Smart City Digital Twin Definition model. The evaluation was performed for 1, 5, 10 and 20 neighborhoods to analyze how the system scales when the load increases by 5, 10, and 20 times. In addition, because the experiment timeframe was compressed to a short period of 24 minutes, the number of neighborhoods was reduced to reduce the number of events generated per second at a similar rate. Regarding the amount of generated events, the reduced experiment setup represents approximately a 24-hour city of 1200 neighborhoods. The total number of Twin Instances created during the experiment is presented in Table 5.

The execution of the experiment was divided into two steps. First, the above scenarios were executed 3 times each with autoscale and scale-to-zero features enabled to verify how the prototype scales for different city sizes using Knative Serverless functions. The minimum number of pods per Twin Instances was set to zero and the maximum was set to 18. The Event Store service thresholds were set to 1 and 25, because it has a higher throughput. The selected upper thresholds correspond to the maximum amount of pods that could be created considering the memory limits of the nodes set to each they were deployed. The autoscale settings were configured using the concurrency metric, which determines the number of simultaneous requests that each replica of an application can process at any given time, as the value 5. Later, the scenarios of 1 and 5 neighborhoods were executed 3 times each with a fixed number of pods to reproduce scenarios of over-provisioning computing resources (160 pods, 14 per Twin Instance + 20 for the Event Store) and under-provisioning computing resources (13 pods, 1 per Twin Instance + 3 for the Event Store). Each simulation aimed to reproduce 24 hours of Smart City events in a 24-minute timeframe.

The experiments focused on evaluating performance and resource utilization using a set of carefully selected metrics: the number of pods, CPU, memory, and response times. The number of pods metric represents the number of deployed instances in the system and is used to understand how the system scales service instances for different workload sizes. The CPU metrics include CPU requested, reflecting the amount of CPU requested for each service instance, and CPU usage, indicating the actual percentage of resource utilization. Similarly, memory metrics comprised memory requested, representing planned memory allocation, and memory usage, showing real consumption levels. To measure performance under varying loads, response time metrics were analyzed across multiple percentiles: p50 (median) to gauge typical user experience, p90 and p95 to capture performance under higher loads, and p99 to evaluate system behavior during peak or extreme conditions. Together, these metrics provided a comprehensive understanding of the system’s scalability, efficiency, and performance across deployment configurations and neighborhood sizes.

The evaluation setup was performed in a private Kubernetes cluster comprising 20 machines. The nodes were divided into three groups based on their sizing and responsibilities: Core (Intel Xeon, AMD Opteron with 24-64+ threads, 16Gb-24Gb RAM), Services (Intel i7/i5 with 4 threads, 4-16Gb RAM) and Devices (Intel i7/i5 with 4 threads, 4-16Gb RAM). Each group of machines runs different workloads and the assignment of the workload pods to the corresponding groups was implemented using Kubernetes node selector based on predefined labels.

The Core group consists of nodes to which the main KTWIN components are deployed, such as Event Broker, Event Store, Graph Store, and the Dispatchers. The target node is selected during the KTWIN installation steps. The Service nodes run end-user services which consist of the Twin Services and their corresponding Event Dispatchers. For the services, the node selection is implemented by the Operator, so whenever the user defines a new Twin Interface resource, the corresponding service is deployed to the target nodes without any user action. Finally, the Device group executes the IoT device sensors that publish and subscribe to events of Event Broker. By following this approach, it is possible to have more granular control of where each workload will run and based on how much computing power the corresponding node has.

The results of the experiment were collected using Grafana dashboards. The dashboards contain application information, such as the number of requests per second categorized by event types, response time, CPU, and memory usage. The metrics were exposed by the open-source components adopted for the proposed implementation, such as Knative and RabbitMQ, scraped by Prometheus, and consumed by Grafana. Some custom Prometheus metrics were implemented and exposed by KTWIN components, such as the number of messages published by IoT devices and processed by the MQTT Dispatcher.

In this section, we present the evaluation results of this research. The results analysis is divided into three sub-sections. In the first sub-section, we present the high-level evaluation results and how each evaluation scenario differs from the others in their sizing, number of connected devices, and processed events per second. Next, we analyze how the system components behaved for scenarios of 1, 5, 10 and, 20 neighborhoods regarding the number of replicas created by the auto-scaler, the amount of CPU and memory requested and used, and service response time. Finally, we discuss the serverless capabilities of KTWIN by comparing the previous experiments against under-provisioned and over-provisioned scenarios.