Wireless Human-Machine Collaboration in Industry 5.0

The primary goal of human control modeling is to mathematically represent how humans perform tasks, enabling machines to understand and adapt to human control policies. This modeling is essential for the design of high-performance machine control systems that can effectively collaborate with human operators. Researchers have proposed various methods to model human control policies. For example, the human operator is often modeled as a classical machine controller, such as linear feedback controller[12, 13], proportional-integral-feedback controller[14], and impedance controller[15]. The human control behavior can also be modeled using the crossover-reference model with time-invariant dynamics [16], where human operators are characterized as an open-loop transfer function. In addition to the above deterministic models, researchers have also proposed several probabilistic models, such as hidden Markov models (HMMs) [17], partially observable Markov decision processes (POMDPs) [18] and Markov decision processes (MDPs) [19]. Despite significant progress in human control modeling, accurately formulating human control policies mathematically remains a long-lasting unsolved challenge.

Human characteristics modeling aims to represent the stochastic human traits that implicitly influence the delivery and accuracy of control commands generated by the human decision-making process. These time-varying characteristics include operator workload, proficiency, fatigue, and control lag. For example, the operator’s workload can be modeled as a uniform distribution over binary state sets of high and low workload[19]. The operator’s fatigue can be modeled as a binary state set (awake or sleepy) with a certain distribution [18]. However, the human characteristics in these works are modeled as independent and identically distributed random states. Human characteristics are commonly time-correlated. In order to capture the time-correlated feature, many works model human characteristics as a Markov process [20] and adopt HMMs to infer human characteristics based on the recorded temporal data [17, 21]. Although using the Markov process to model time-varying human characteristics has garnered significant attention, its application to modeling human control lag has been less considered.¹¹1Modeling the human characteristics impacting the accuracy of human control commands relies on the precise formulation of human control policies, which is a long-lasting unsolved challenge and beyond the scope of our current work [22]. For a specific collaboration task, the control policy of a human operator commonly remains unchanged in the short term. Thus, we focus on the human control lag, which influences the delivery of human control commands and impacts the collaborative control performance. The impact of such stochastic human control lag on system performance remains underexplored.

Stability analysis is crucial for designing a WHMC system to operate efficiently and safely. Effective stability analysis requires tractable modeling of the WHMC system. However, most works focus on the fundamental stability analysis of simplified WHMC systems [12, 14, 15, 23, 16]. In this regard, these works can perform classical analytical frameworks to enable optimal control with a stability guarantee in specific applications, such as irrigation canal [14], robotic exoskeleton [15], collaborative driving [23], and collaborative piloting [16]. These limitations make the methodology of most existing works on stability analysis limited to specified control applications, which may weaken the generalization ability of their analytical frameworks. In addition, these systems do not integrate with wireless communication links. Tractable mathematical modeling of advanced WHMC systems with the integration of wireless communication links to establish the stability condition is an unsolved problem.

Wireless networked control involves integrating autonomous control systems with wireless communication networks. It primarily focuses on establishing systematic theories related to the stability and optimization of state estimation and automatic control over wireless networks [24, 25, 26]. Existing research has largely concentrated on developing optimal control algorithms that address the challenges posed by imperfect wireless communication channels, such as errors and delays [27, 28]. Some studies investigate the impact of communication protocols and parameters on the stability of automatic control systems [29, 30]. WHMC extends wireless networked control by incorporating human intelligence into the control loop, enhancing system adaptability and performance. While traditional wireless networked control focuses on how communication systems affect control stability, it does not account for the complexities introduced by human operators. Consequently, existing methods in wireless networked control are insufficient for WHMC systems, which require new approaches to address the challenges posed by integrating human factors into the control process.

Each machine control loop takes a single time step, i.e., the period of a machine control loop is $T_{s}$, and consists of a pair of SC uplink and CA downlink transmissions. If the SC packet is not detected successfully, there is no CA transmission scheduled as the controller has no instantaneous plant state information. We consider short-packet transmissions for low-latency communications [33]. The computation time for generating a control signal is usually much shorter than the transmission delay and thus is omitted [34, 35]. A machine control loop is closed only when both the SC and CA transmissions within it are successful.

Each human control loop period is delineated by an HA downlink transmission, as illustrated in Fig. 2. A human control loop contains multiple SH uplink transmissions, a human control procedure, and an HA downlink transmission. A downlink transmission attempt marks the end of one human control loop period and the beginning of the next. Each period starts from a new packet transmission from the sensors, which contains the current plant state measurement. If the transmission fails, then a retransmission takes place using a HARQ protocol. In instances where a given maximum number of retransmissions $N$ has been reached, a new transmission is triggered.³³3Unlike machine control loops, the lag of human control, which captures the delay in human decision-making, is significantly longer than the transmission delay [36]. This results in frequent machine control actions and infrequent human interventions. In contexts where the lag of human control is substantial, the transmission delay becomes relatively insignificant. Consequently, retransmissions are used to improve transmission reliability, as a longer transmission delay caused by retransmissions does not notably affect the overall human control process. The human operator generates a control command after receiving a successful packet, and then sends the command to the actuator. There is no retransmission for the HA and CA downlinks, since retransmissions lead to unpredictable delays, making the generated time-sensitive control command useless. Let $t^{\prime}$ denote the human control loop index. Then, the transmission delay for the SH and HA transmissions are $\tau_{SH}(t^{\prime})$ and $\tau_{HA}=1$, respectively, and the lag of human control is $\tau_{H}(t^{\prime})\in\mathcal{S}\triangleq\{1,2,\dots,\tau_{\mathrm{max}}\}$. In particular, $\{\tau_{H}(t^{\prime})\}$ is modeled as a finite state Markov chain with a transition probability matrix $\mathbf{M}$, where $p_{i,j}\triangleq[\mathbf{M}]_{i,j}$. A shorter lag of human control leads to better control performance. The stationary distribution of $\tau_{H}(t^{\prime})$ is given as

We assume each transmission in a human control loop takes one time step because human-type communication generally requires a larger packet length than machine-type communication [37]. Considering the random period of each human control loop, we define $\kappa(t^{\prime})$ as the starting time slot of the $t^{\prime}$-th human control loop. The human control loop is closed once the HA transmission is successful.

Let $\gamma_{HA}(t)$, $\gamma_{SH}(t)$, $\gamma_{CA}(t)$, and $\gamma_{SC}(t)$ denote the signal-to-noise ratio (SNR) of received packets in HA, SH, CA, and SC channels, respectively. Given the packet length $l_{p}$ (i.e., the number of symbols per packet), the number of data bits $b$ in the packet, and the SNR $\gamma$ of the packet, we have the approximated decoding error probability of a packet as [38]

where $\mathcal{C}(\gamma)=\log_{2}{(1+\gamma)}$ and $\mathcal{V}(\gamma)=(1-(1+\gamma)^{-2})(\log_{2}{e})^{2}$ are the Shannon capacity and the channel dispersion, respectively, and $\mathcal{Q}(x)=(\frac{1}{\sqrt{2\pi}})\int_{x}^{\infty}{e^{-\frac{t^{2}}{2}}\text{d}t}$ is the Gaussian Q-function.

The probability of the machine control operating in an open loop at time $t$ can be obtained as

The expectation of (7) with respect to $\gamma_{SC}\left(t\right)$ and $\gamma_{CA}\left(t\right)$ is denoted as $\bar{p}_{M}$, and can be obtained by

Since each human control loop contains a successful SH packet, the probability of an open human control loop only depends on the HA transmission and is given by

The expectation of (9) with respect to $\gamma_{HA}(\kappa(t^{\prime})-1)$ is denoted as $\bar{p}_{H}$, and can be obtained by

The duration of a human control loop $\tau_{SHA}(t^{\prime})$ in (5) includes the SH channel delay $\tau_{SH}(t^{\prime})$, human control lag $\tau_{H}(t^{\prime})$, and the HA channel delay $\tau_{HA}$. The HA channel delay is constant across human control loops, while the human control lag is time-correlated across human control loops due to the Markovian property. The SH channel delay is attributed to the HARQ and i.i.d across all human control loops. We analyze the distribution of the SH channel delay before proceeding with the distribution of the duration of consecutive time steps where the human control loop is open. We consider the following three types of HARQ schemes for the SH channel, including Type I HARQ (TI-HARQ), Chase Combing HARQ (CC-HARQ), and Incremental Redundancy HARQ (IR-HARQ).⁴⁴4In TI-HARQ, the packet is re/transmitted for all re/transmissions, and all erroneously decoded packets are discarded at the receiver side. All decoding attempts during re/transmissions of the packet are independent. In CC-HARQ, all erroneously decoded packets in previous re/transmissions are saved and their signals are combined together as a single strengthened signal for decoding. In IR-HARQ, the packet in each re/transmission is a punctured version of a low-rate mother packet. If errors occur, it only retransmits the additional redundancy for the previous uncorrectable packets. The newly received redundancy is combined with the previously received packets to construct a packet with a longer length for decoding.

The number of re/transmission attempts is $r\in\{1,2,\dots,N\}$. Let $\bm{\gamma}_{r}(\kappa(t^{\prime}-1),r)$ denote the set of experienced SNRs during $r$ re/transmission attempts, that is

The decoding error probability of the packet after $r$ re/transmission attempts $\Theta(r)$ is an expectation over (11), and can be approximated as [39, 40, 41]

To facilitate our subsequent analysis, we assume that all packets have the same length $l_{p}$. For CC-HARQ, since the channel gain is exponentially distributed, $\sum_{i=0}^{r-1}\gamma_{SH}(\kappa(t^{\prime}+i))$ is gamma distributed with the probability distribution function [42]

where $\bar{\gamma}$ is the mean of the exponential distribution. Thus, for TI- and CC-HARQ, $\Theta(r)$ is obtained by leveraging (6), (12), and (13). For IR-HARQ, $\Theta(r)$ can be determined by Monte Carlo simulations.

The delay induced by the SH transmission period $\tau_{SH}(t^{\prime})$ is $k\in\mathbb{N}_{0}$. Note that the SH transmission period may contain multiple $N$-trails of the retransmission process as described in Section II-B2, and the number of experienced $N$-trails is $q\in\mathbb{N}_{0}$. The probability distribution of $\tau_{SH}(t^{\prime})$ is then given as (14). When $q=0$, the SH transmission is successful in the first $N$-trials. In this case, if $2\leq k\leq N-1$, $\tau_{SH}(t^{\prime})\!=\!k$ indicates that the first $k-1$ trials have failed and the $k$-th transmission attempt is successful. When $q>0$, the SH transmission is successful in the $\left(q+1\right)$th $N$-trials, while the former $q$th $N$-trials are decoded erroneously.

We denote the starting time of the $n$th closed human control loop as $t=k_{n}$, as shown in Fig. 3. Let $L$ and $M$ denote time steps and the numbers of (open or closed) human control loops between $k_{n}$ and $k_{n+1}$, respectively, i.e.,

where $t^{\prime}$ is the index number of the $n$th closed human control loop among all the loops. The probability distribution of $L$ in (15) can be expressed as

The probability distribution of the number of consecutive open human control loops in (16) can be expressed as

where $\bar{p}_{H}$ is defined in (10). The time interval distribution of $L$ under the condition with $m$ open human control loops in (16) consists of two independent and stochastic parts, i.e., the total delay induced by SH channel and human control lag. In the following, we analyze the conditional probabilities of the two parts. The conditional probabilities of the delay induced by the SH channel can be expressed as

where $w_{k}$ is defined in (14). The conditional probabilities of the delay induced by the human control lag can be expressed as (19),

where $\delta_{m}\in\mathcal{S}$, $v_{k}$ is defined in (2), and $p_{\delta_{m-1},\delta_{m}}=[\mathbf{M}]_{\delta_{m-1},\delta_{m}}$. Then, the time interval distribution under the condition with $m$ open human control loops is

where $w_{k,m}$ and $v_{k,m}$ are conditional probabilities defined in (18) and (19), respectively. In summary, by using (18) and (19), we can obtain (20). By substituting (17) and (20) into (16), we can obtain the time interval distribution between consecutive closed human control loops.

In the following, we propose a stochastic cycle-cost-based approach to obtain sufficient stability conditions for the WHMC system.

Sufficient conditions in stability analysis are critical because they provide guarantees that the system will be stable under the specific assumption. They are thus preferred since they give engineers and researchers a clear set of criteria to design and analyze their systems safely. The stability condition of the WHMC systems depends on the wireless communication parameters, i.e., the open loop probabilities of human and machine control $\bar{p}_{H}$ and $\bar{p}_{M}$, the control system parameters, i.e., $\alpha_{HM}$, $\alpha_{M}$, $\alpha_{H}$ and $\alpha$, and the Markov human state transition rule $\mathbf{M}$. In particular, $\bar{p}_{H}$ and $\mathbf{M}$ impact the distribution of $L$, which further affect the stability condition. The condition indicates that if the WHMC systems exhibit high dynamics (i.e., the plant state changes significantly even with very small control input), the human operator experiences fatigue with a high control lag, and the open-loop probability is high, then the WHMC system becomes difficult to stabilize through collaboration.

The stability region in WHMC systems defines the range of system parameters that ensure stable operation, as per Theorem 1. The boundary of this stability region represents the critical limits beyond which the system may become unstable. The properties of this boundary are elucidated next.

As illustrated in Fig. 4, a linear stability region (e.g., Corollary 1 (i) and (ii)) means the boundary is governed by a linear function. It implies that any combination of the control system parameters within the region will maintain system stability, offering engineers substantial flexibility in parameter selection and system tuning without compromising stability. This implication is applicable to the convex stability region, where the boundary is governed by a convex function. In contrast, a concave stability region (e.g., Corollary 1 (iii)) has a boundary governed by a concave function. This indicates that while individual parameter sets within the region ensure stability, linear combinations of these parameters may not. For any stable parameter set, all parameter sets within the rectangular area defined by this set and the origin are also stable. In addition to control system parameters, communication system parameters also impact the stability region, which is presented in Section III-C.

Given the stability condition of the general WHMC system in Theorem 1, we examine the stability conditions for three specific cases.

For an error-free channel, assuming the communication channels are perfect, we have $p_{H}(t)=p_{M}(t)=0,\forall t$. The stability condition in (1) reduces to

where $\mathbb{E}\left[\alpha_{M}^{L}\right]=\sum_{k=1}^{\tau_{\text{max}}}\alpha_{M}^{k+1}v_{k}$ and $v_{k}$ defined in (2) is determined by the human state transition matrix $\mathbf{M}$. In this case, the stability depends on $\alpha_{M}$, $\alpha_{HM}$, and $\mathbf{M}$. Since the communication channels are perfect, only human control loops may be open due to the human control lag. Thus, only the Lyapunov gains in cases one and two of Assumption 1, i.e., $\alpha_{HM}$ and $\alpha_{M}$, play a role in this scenario.

Human control only, assuming that the plant is only controlled by a human operator, i.e., the machine control loop is always open ($p_{M}(t)=1,\forall t$). The stability condition is

where $\mathbb{E}\left[\alpha^{L}\right]=\sum_{l=0}^{\infty}\alpha^{l}z_{l}$ and $z_{l}$ is defined in (16). In this case, the stability depends on $\alpha_{H}$, $\alpha$, and $\mathbf{M}$. Since the machine control loop is always open, only the Lyapunov gains in cases three and four of Assumption 1 are relevant. We note that if the human control lag is a constant, $L$ is still a random time interval due to the random SH delay.

Machine control only, assuming that the plant is only controlled by a machine controller, i.e., the human control loop is always open ($p_{H}(t)=1,\forall t$). The stability condition of this case cannot be directly obtained from Theorem 1, because the stochastic cycle-based approach in Theorem 1 is on the basis of closed human control loops. Modifications to the definition of stochastic cycles are required to analyze the stability condition. Our results are presented next:

In this case, the stability depends on $\alpha$, $\alpha_{M}$ and $\bar{p}_{M}$. Since the human control loop is always open in this case, only the Lyapunov gains in cases two and four of Assumption 1 are applicable. (27) resemble exist results [43].

In the simulated cart-pole system, the mass of the pole is assumed to be concentrated at its end mass. The states of the cart-pole system consist of the position $x(t)$ and velocity $\dot{x}(t)$ of the cart, the angle $\theta(t)$ and angular velocity $\dot{\theta}(t)$ of the pole, and the unknown weight $m_{c}(t)$ on the cart, which is denoted as $\mathbf{x}(t)\triangleq(x(t),\dot{x}(t),\theta(t),\dot{\theta}(t),m_{c}(t))^{\top}$. The dynamics of the cart-pole system are governed by the non-linear dynamic equations in (29),

where $M_{p}=2$ $kg$ and $M_{c}=10$ $kg$ are the mass of the pole and cart, respectively; $g=9.8$ $m/s^{2}$ is the gravitational acceleration; $L_{p}=6$ $m$ is the length of the pole; $I=\frac{M_{p}L_{p}^{2}}{4}$ is the moment of inertia for a point mass in terms of the center of the pole; $u_{M}(t)$ is the applied force to the cart by the machine controller in (3); $c=b=0.1$ are the damping coefficients for the pole and cart, respectively. For the dynamics of the unknown weight on the cart, $m_{c}(t)$, we assume that once the weight is successfully removed by the actuator remotely controlled by the human operator, it will reappear on the cart after a random time interval; otherwise, the unknown weight will remain on the cart continuously. Thus, $m_{c}(t)$ has the following updating rule

where $w(t)\in\{0,5\}$ is randomly generated and $u_{H}(t)$ is the human control input. The sampling period is $T_{s}=0.05$ $s$. $\ddot{x}(t)$ and $\ddot{\theta}(t)$ can be derived from (29) given $\mathbf{x}(t)$. Then, by leveraging $\mathbf{x}(t)$, $\ddot{x}(t)$, $\ddot{\theta}(t)$, $T_{s}$ and (30), we can get $\mathbf{x}(t+1)$, indicating the proposed cart-pole system follows (1). The initial state is $\mathbf{x}(0)=(0,0,\frac{\pi}{6},0,5)^{\top}$.

In this experiment, $m_{c}(t)$ is unknown to the machine controller, but all other states, parameters, and the system dynamics in (29) (excluding $m_{c}(t)$) are known. The machine control policy seeks to achieve a decaying angle as per $\theta(t+1)=\eta\theta(t)$ by applying force $u_{M}(t)$, where $\eta\in(0,1)$. Using the Euler approximation to update the angle, we obtain

The updated angular velocity $\dot{\theta}(t+1)$ is also obtained by using the Euler approximation, i.e., $\dot{\theta}(t+1)=(\dot{\theta}(t)+T_{s}\ddot{\theta}(t))T_{s}$. A smaller $\eta$ will lead to a control policy enabling a faster speed of $\theta(t)\rightarrow 0$, which is 0.7 in the experiment. By leveraging (31) and (29) with $m_{c}(t)=0$, the machine control policy can be obtained as (32) to determine $\ddot{\theta}(t)$,

where $\Gamma(t)\triangleq(M_{c}+M_{p})(2I+M_{p}L_{p}^{2})-(M_{p}L_{p}\cos(\theta(t)))^{2}$. Recall that the human control policy is to remove the unknown weight on the cart if the human operator observes it through visual feedback. Thus, the human control policy is $f_{H}(\mathbf{x}(t))=-m_{c}(t)$.

Since the control objective is to balance the pole, the Lyapunov-like function $V(\cdot)$ is defined as

where the threshold of 0.05 is set to eliminate the impacts of uncontrolled $\dot{\theta}(t)$ on the control objective $\theta(t)$ when $\theta(t)$ reaches to the desired zero point. To estimate the four control system parameters, i.e., $\alpha_{HM}$, $\alpha_{M}$, $\alpha_{H}$, and $\alpha$, we collect data by conducting the experiment in four cases, i.e., no control inputs, machine control only, human control only, and human-machine collaborative control, respectively. The parameter in each case is estimated by $\max{\frac{V(\theta(t+1))}{V(\theta(t))}}$ based on the corresponding data set, where $V(\cdot)$ is defined in (34). The estimated four control system parameters are $\alpha_{HM}=0.5271$, $\alpha_{M}=0.7949$, $\alpha_{H}=1.0196$, and $\alpha=1.0134$.

To reduce the estimation complexity, we quantize the human control lag to a two-state set of $\{3,7\}$, which corresponds to 0.15 $s$ and 0.35 $s$. The state transition matrix is estimated based on the maximum likelihood estimation approach, which is

The corresponding stationary probability distribution is $(0.3723,0.6277)$, i.e., $\mathbb{P}\!\left[\tau_{H}(t^{\prime})\!=\!0.15\right]\!=\!0.3723$ and $\mathbb{P}\!\left[\tau_{H}(t^{\prime})\!=\!0.35\right]\!=\!0.6277$.

Based on the above estimation and parameters in Table I, the left term of the stability condition in Theorem 1 is $0.3539<1$, demonstrating a stabilized WHMC system. In the machine-only control scenario, the left term of the stability condition (27) is $0.8594<1$, indicating a stochastically stable system. Conversely, in the human-control-only case, the left term of the stability condition (26) is $3.3088>1$, signifying an unstable system. This instability also explains the increasing control cost observed in Fig. 7.