Functionally Divided Manipulation Synergy
for Controlling Multi-fingered Hands

Kazuki Higashi^1∗, Keisuke Koyama¹, Ryuta Ozawa², Kazuyuki Nagata³, Weiwei Wan^1,3 and Kensuke Harada^1,3 *This work was supported in part by the NACHI-FUJIKOSHI Corporation.¹Graduate School of Engineering Science, Osaka University; 1-3 Machikaneyama- cho, Toyonaka, Osaka 560-8531, JAPAN²School of Science and Technology, Meiji University³National Institute of Advanced Industrial Science and Technology^∗ Corresponding author e-mail:
higashi[at]hlab.sys.es.osaka-u.ac.jp

Abstract

Synergy supplies a practical approach for expressing various postures of a multi-fingered hand. However, a conventional synergy defined for reproducing grasping postures cannot perform general-purpose tasks expected for a multi-fingered hand. Locking the position of particular fingers is essential for a multi-fingered hand to manipulate an object. When using conventional synergy based control to manipulate an object, which requires locking some fingers, the coordination of joints is heavily restricted, decreasing the dexterity of the hand. We propose a functionally divided manipulation synergy (FDMS) method, which provides a synergy-based control to achieves both dimensionality reduction and in-hand manipulation. In FDMS, first, we define the function of each finger of the hand as either “manipulation” or “fixed.” Then, we apply synergy control only to the fingers having the manipulation function, so that dexterous manipulations can be realized with few control inputs. The effectiveness of our proposed approach is experimentally verified.

I Introduction

Synergy provides a method to simultaneously control multiple Degrees of Freedom (DoFs) of a multi-fingered hand to produce coordinated movements from a limited set of input signals. One of the common strategies for controlling multi-fingered hands is to adopt synergy for their control inputs [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. For grasping an object, a synergy, hereafter called grasp synergy, can be constructed by applying the principal component analysis (PCA) to the human’s hand grasping postures. The grasp synergy is then used to express various postures with the multi-fingered hand using a few principal components [15, 16, 17].

The main purpose of a multi-fingered hand is to manipulate objects, including the use of tools. Using conventional synergy-based control, object grasping [12] and simple in-hand manipulation (changing the pose of an object) [1] have been realized. Nevertheless, it is very challenging to achieve the required posture expressions for using human tools with the framework of conventional synergy-based control. Therefore, we focus on the manipulation of human tools with a new synergy-based control method.

Kamakura [18] measured and analyzed human hand movements when manipulating an object. Kamakura’s research showed that in 45.5% of human hand motions observed in daily life, the position of particular fingers is fixed in place. According to this observation, we hypothesis that to improve control’s performance of a multi-fingered hand during in-hand manipulation tasks, the position of specific fingers needs to be fixed. The fingers that should hold a fix position are, for example, those that are not needed for the manipulation task or those that are needed to hold the position of a grasped object.

However, the grasp synergy-based control assumes the coordination of all joints of a multi-fingered hand in order to express human-like grasping postures. Thus, fixing the position of specific fingers heavily limits the control space of the grasp synergy. The precise posture expression, with reduced dimensionality, required for object manipulation is difficult to realize by using conventional grasp synergy-based control.

Refer to caption — Figure 1: Overview of our proposed method. Our FDMS control method consists of the differentiation of the function of each finger of the hand; the ones needed for manipulation and the ones that do not need to be moved (fixed).

Considering the above mentioned problem, this paper proposes a Functionally Divided Manipulation Synergy (FDMS) method, which provides a synergy-based control for a multi-fingered hand when manipulating an object, in which some of the fingers are used to manipulate an object while others are fixed in place. Figure 1 shows an overview of the proposed FDMS. Said figure shows an example of clicking downward on the top of a pen with one’s thumb. In this example, the position of index, middle, ring and little fingers are kept fixed to maintain the grasp of the pen. First, various grasping postures of a human hand are measured in advance. Then, we construct FDMSs based on the principal component vectors of the measured human hand’s postures including only the fingers or joints needed for the manipulation.

Similarly, we consider the situation where a robot first picks a pen and then press the top of the pen. The robot first grasps the pen by using the grasp synergy based control, then uses the FDMS method to control only the thumb to click downward the top of the pen while other fingers are used to maintain a grasp of the pen. In the said case, we can switch among multiple FDMSs and fix different fingers according to the task. Therefore, we propose a framework named the Switching Synergy Framework. The Switching Synergy Framework makes the FDMSs useful to perform various sequential tasks.

The main contribution of our work is to provide a synergy control framework for performing a task with a few dimensional control inputs. For this purpose, we introduce the FDMS method to assign a function to each finger (Manipulation or Fixed). FDMS-based control is found out to achieve a higher task success rate than conventional grasp synergy-based control. For comparison, we also introduce the task-specific synergy composed only of a set of hand postures of performing a specific task.

This paper is organized as follows. The related work on the synergy and multi-fingered hands are introduced in Section II. Our proposed method using FDMS is depicted in Section III and Section IV. Our proposed Switching Synergy Framework which make FDMS useful to apply in sequencial tasks is described in Section V. Section VI describes the experiments conducted to evaluate the proposed methods. The results and comparisons between methods are described in Section VII.

II related work

The concept of synergy has drawn attention in the robotics research community. Many researchers have researched it since it provides an effective method for controlling multi-fingered hands with a few dimensional control inputs. Santello et al. [15] measured human hand motions during daily life, assuming a 15 DoFs kinematics model and showed that 80% of the movements could be realized with just two principal components. Similarly, Todorov et al. [19] showed that four principal components represent 85% of the movements when assuming a 20 DoFs kinematics model. Xiong et al. [4] developed a synergy-based multi-fingered hand which can realize 30 kinds of grasping poses out of 33 poses defined by Feix [20]. Chen et al. [21] developed a synergy-based hand based on the 2 DoFs angular velocity synergy, implemented by using cam and underactuation mechanism.

Although the grasp synergy provides an effective method for controlling multi-fingered hands, the research on performing a task with synergy-based control is limited.

Kent et al. [6] proposed the task dependent temporally synchronized synergies (TSS) for performing a task by using a prosthetic hand according to the EMG signal of a human. TSS is an attractive method that enables us to generate dexterous motion of an object, but to create a TSS, humans need to measure the motion for each task. In general, a multi-fingered hand is required in a large number of tasks, so the TSS method has a very high measurement cost. Our method, FDMS, is constructed from a variety of grasping postures, similar to grasp synergies. Thus, the measurement cost is low, and it is only necessary to assign specific functions to each finger to perform tasks.

Santina et al. [1] developed a synergy-based multi-fingered hand and demonstrated the effectiveness of the adaptive synergy [2] through some complex tasks such as grasping and opening a screw cap. However, the conventional grasp synergy-based control is not suitable for performing a task, since if we try to fix the position of a finger of multi-fingered hand when performing a task, the control space spanned by its principal component vectors is heavily restricted and cannot express postures required for the advanced tasks. Therefore this property makes the grasp synergy poor to perform manipulation tasks. On the other hand, our proposed synergy-based control can perform dexterous tasks such as manipulating an object while grasping that without being affected by fixed fingers because we consider only fingers assigned the manipulating function.

III synergy

III-A synergy control

Synergy, in the context of control of a multi-fingered hand, is defined to be the coordination of joints and defines the basis of the joint-angle space. In this section, we first formulate synergy introduced in this research.

By using a set of hand poses, the posture sequence $\bm{X}$ used for formulating synergy can be defined as:

\bm{X}=[\bm{p}_{1},\bm{p}_{2},\cdots,\bm{p}_{j},\cdots,\bm{p}_{n}]^{T},

(1)

where $\bm{p}_{i}$ ( $i=1,\cdots,n$ ) denotes a $d$ -dimensional vector expressing a configuration of a multi-fingered hand. We assume that each pose included in $\bm{X}$ is sufficiently diverse.

In this research, we make a multi-fingered hand manipulating an object by using some of the fingers while others are used to grasp or simply fixed. We formulate such division of finger functions. By defining the indices of all joints included in the hand as $J=\{1,2,...,d\}$ , we can define the subset of $f$ joints from $J$ as $J^{\mathrm{sub}}=\{s_{1},\cdots,s_{j},\cdots,s_{f}\}\subseteq J$ where $J^{\mathrm{sub}}$ is defined to be a set of joints included in some of the fingers to realize a specific function. By using $J^{\mathrm{sub}}$ , a hand pose $\bm{p}_{j}^{\mathrm{sub}}$ can be represented as:

\bm{p}_{i}^{\mathrm{sub}}=[p_{i,s_{1}},p_{i,s_{2}},\cdots,p_{i,s_{j}},\cdots,p_{i,s_{f}}]^{T}

(2)

From (2), we can obtain the posture sequence $\bm{X}^{\mathrm{sub}}$ consisting of $\bm{p}_{i}^{\mathrm{sub}},i=1,\cdots,n$ . The principal component analysis (PCA) of the extracted hand pose is equivalent to the eigenvalue problem for its covariance matrix. The eigenvectors $\bm{a}_{1},\cdots,\bm{a}_{f}$ and the eigenvalues $\lambda_{1},\cdots,\lambda_{f}$ represent the coordination of joints and the variance of each principal component, respectively. Finally, we can define a synergy matrix $S$ for the $n_{s}$ -dimensional synergy control as follows:

S=[\bm{a}_{1},\cdots,\bm{a}_{n_{s}}]

(3)

III-B synergy definition

Now we define three different synergies, the most commonly used grasp synergy, the functionally divided manipulation synergy proposed in this paper and the task-specific synergy.

We first introduce the grasp synergy which is equivalent to the conventional definition of synergy:

Definition 1

Grasp Synergy is defined to be a synergy composed of various grasping postures in $\bm{X}$ . Grasp synergy considers the total joint space $J$ of a human hand or a multi-fingered hand.

On the other hand, we aim to make a multi-fingered hand to perform a task. We introduce the following two definitions on synergy for performing a task. Among the two definitions, the Functionally Divided Manipulation Synergy (FDMS) proposed in this paper will be define later in the next section. For the comparison study, we introduce synergy for performing a specific task:

Definition 2

Task-Specific Synergy of the $k$ -th task is defined to be a synergy composed of hand postures $\bm{X}^{T}_{k}$ performing the $k$ -th task considering the total joint space $J$ of a human hand or a multi-fingered hand. The synergy matrix of the $k$ -th task-specific synergy is denoted as $S^{T}_{k}$ .

III-C postures approximation

In synergy-based control, it is important to evaluate how precisely a hand posture can be realized for performing a requited task. To evaluate the capability to express such function-required postures, we verify whether or not a task succeeds when a hand postures are presented in the synergy space. By using the synergy matrix, an approximated posture $\hat{\bm{p}}$ is given by:

\hat{\bm{p}}=SS^{T}\bm{p},

(4)

where $SS^{T}$ denotes an orthogonal projection matrix projecting a joint angle vector to the synergy space. This accuracy of approximated posture is used to evaluate each synergy. The posture sequence $\hat{\bm{X}}$ approximated by a synergy can be expressed by:

\hat{\bm{X}}^{T}=SS^{T}\bm{X}^{T}=[\hat{\bm{p}}_{1},\hat{\bm{p}}_{2},\cdots,\hat{\bm{p}}_{j},\cdots,\hat{\bm{p}}_{N}]

(5)

IV Functionally Divided Manipulation Synergy (FDMS)

In this section, we further formulate FDMS method based on the typical movement units of human hands. We assign two kinds of functions (Manipulating and fixed) to each finger based on a study of finger motion analysis for manipulating an object [18]. We also identify necessary FDMSs based on typical finger movements and its frequency.

IV-A Typical unit movements of human hands

To analyze the human hand motion, we introduce the Kamakura’s XYZ-notation, which defines the finger movement when manipulating an object [18]. The XYZ-notation method decompose a series of human hand motions, manipulating an object, into movement units. The movement unit for a given manipulation task is defined with respect to the the movement of each finger, considering them in sequential order from the thumb to the little finger. Each finger is assigned a symbol X, Y, Z, or O according to the following conditions. Kamakura defined discretized cluster of a finger movement. If a finger movement is categorized into the same cluster, we assign the same symbol. Firstly, if there is any movement of the thumb, assign the symbol X. Next, if the movement of the index finger is the same as the movement of the thumb, assign the same symbol X. If the movement is different, assign the symbol Y. Similarly, if the movement of the middle finger is same as the thumb, assign the symbol X, if different from the thumb but the same as the index finger, assign the symbol Y. Moreover, if the movement of the middle finger is different from both the thumb and the index finger, assign it the symbol Z. The symbol O is assign to a finger that has no motion during a given manipulation task. According to [18], for most of daily tasks, each finger performs at most three different motions. This is because only XYZ are used to express finger motion. According to the XYZ-notation, movement of a human hand can be described with five characters. An example of using the XYZ-notation to characterize a task is shown in Figure 2. In said example, the task of manipulating a pen is considered as being composed of four phases; grasping the pen, preparing grasping posture for clicking the top of the pen, preparing posture of the thumb, and then clicking downward the top of the pen, to expose the ballpoint.

TABLE I: Typical movement unit and its frequency[18]

Movement unit	Frequency[%]
XXXXX	8.5
XYYYY	9.0
XOOOO	8.9
OXXXX	3.5
XXYYY	1.9
XXOOO	3.2
OOXXX	5.1
OOOXX	2.4
XYZZZ	3.5
XYOOO	4.8
OXYYY	2.6
OXOOO	8.9
XYYOO	1.9
OOXOO	1.9
OOXXO	1.3
XYYYO	1.0
Total	68.4

Table I shows the typical unit movements and their frequency obtained from the analysis of human motions manipulating an object [18]. Said table lists all the unit movements whose frequency exceeds 1%. More than 68.4% of unit movements can be expressed by using the mentioned four symbols of the XYZ-notation. In addition, unit movements in a state where at least one finger does not move is 45.5%, implying that it is necessary to fix particular fingers when manipulation an object with human hands.

IV-B Necessary FDMSs

As mentioned above, the XYZ-notation method decompose a series of human-hand motions into movement units. However, each symbol in the XYZ-notation does not explain what function each finger has. In this study, we define the following two symbols to describe the function of each finger, in relation to the XYZ-notation.

1.

Manipulation : M
is assigned to the fingers required for manipulating an object. This corresponds to X, Y, and Z in the XYZ-notation.
2.

Fixed : F
is assigned to a finger that needs to be fixed with its current posture, either because it is necessary to keep that posture (e.g. to maintain a grasping posture), or because the finger is not necessary for the task. This corresponds to O in the XYZ-notation.

To avoid assigning multiple manipulating functions to different fingers within a single movement unit, we divide the movement unit into several units to include only a single manipulating function per movement unit. For example, when we use the computer mouse with a unit movement OXYOO, the index finger is assigned the function of left-clicking and the middle finger is assigned the function of right-clicking. In said case, we divide the movement unit into two units as OXYOO $\rightarrow$ OXOOO + OOXOO. We applied the same operation to all movement units in the Table I. The resulting movement units are shown in Table II.

TABLE II: Units of FDMS (M: A finger included in FDMS construction)

Joint Subspace	FDMS Units
$J^{\mathrm{sub}}_{1}$	MMMMM
$J^{\mathrm{sub}}_{2}$	MFFFF
$J^{\mathrm{sub}}_{3}$	FMMMM
$J^{\mathrm{sub}}_{4}$	MMFFF
$J^{\mathrm{sub}}_{5}$	FFMMM
$J^{\mathrm{sub}}_{6}$	FFFMM
$J^{\mathrm{sub}}_{7}$	FMFFF
$J^{\mathrm{sub}}_{8}$	MMMFF
$J^{\mathrm{sub}}_{9}$	FFMFF
$J^{\mathrm{sub}}_{10}$	FFMMF
$J^{\mathrm{sub}}_{11}$	MMMMF
$J^{\mathrm{sub}}_{12}$	FMMMF

Finally, we can define 12 joint subspaces $J^{\mathrm{sub}}_{l}(l=1,\cdots,12)$ of FDMS consisting of only fingers assigned M in Table II. Base on these joint subspaces, we define the FDMS as follows:

Definition 3

Functionally Divided Manipulation Synergy (FDMS) is defined to be a synergy composed of various grasp postures $\bm{X}^{g}$ considering $l$ -th joint subspace $J^{\mathrm{sub}}_{l}$ of a human hand or a multi-fingered hand. The synergy matrix of the FDMS is denoted as $S^{f}_{l}$ .

FDMS are constructed in a similar fashion as grasp synergies. Therefore, the FDMS consisting of all joint subspaces is equivalent to the grasp synergy.

V Switching Synergy Framework

When performing a task, the function required for a finger changes according to the situation. Hence, it is necessary to switch among multiple FDMS according to the change of functions. In this research, we propose Synergy Switching Framework as a framework for switching synergies. Figure 3 depicts the overview of the Synergy Switching Framework. Within said framework, several synergies that can perform various tasks are prepared in advance and a synergy database is constructed. The user inputs the task to be performed into the system, and selects a synergy suitable for performing the task. Depending on the characteristics of the synergies registered in the synergy database, the granularity of the task division and the required inputs vary. In this study, we register 12 FDMSs defined in Section IV-B to the synergy database.

VI experimental environment

VI-A Human’s Hand Pose Measurement

To measure the posture of the human hand, we use CyberGlove III [22]. The CyberGlove III can measure the joint angles of the human hand at 100Hz. In this study, only the motion of fingers was considered, without considering the motion of the wrist. Therefore, the angles of 20 joints were measured.

VI-B Teleoperating System

The teleoperating system consists of a UR3e robotic arm, a 10 DoFs humanoid hand, and a CyberGlove III. The subject equipped with the CyberGlove III performs a task by transferring postures to the 10 DoFs humanoid hand. The movement of the UR3e robotic arm was controlled through keyboard inputs.

VI-C Tasks

In order to evaluate the performance of our method, FDMS, two tasks were conducted: an opening/closing task of scissors and a pushing task of a switch. These tasks utilize tools that have not been thoroughly considered in previous research. In addition, the tasks of using scissors and pressing a switch were selected as a representative example of tasks that required both to grasp and manipulate at the same time. Following the notation described in Section IV-B, the mentioned tasks are considered as follows: when using scissors, the thumb, index, and middle finger are thought to have a Manipulation function. For pushing the switch, at first, all fingers are assigned the Manipulation function. Then, only the thumb is assigned with the Manipulation function while the other fingers are assigned the Fixed function.

The overall experimental setup for the tasks described above is shown in Figure 4. In both cases, first, the target object is grasped from a fixed initial position on the table. The first task is considered completed if fingers of a humanoid hand inserted into the scissors, and then the scissors were opened/closed three times. Similarly, for the second task, it is considered successful if the switch was fully pressed.

The human operator performed each task for 20 times and measured postures sequences of the hand with the CyberGlove III. For each task, 10 postures sequences $\bm{X}^{T}_{k}$ were used to construct the task-specific synergy and the remaining 10 sequences $\bm{X}^{E}_{m}(m=1\cdots 10)$ are used for evaluation.

VI-D Synergies

We first constructed the grasp synergy from principal components of a variety of grasping postures. The major 33 grasping postures of a human were used based on the grasp taxonomy proposed in [20]. The humanoid hand postures are measured while the fingers are moved from a parallel and flatten posture to the target grasping posture. Then, the posture sequence $X^{g}$ was constructed, as mentioned in Section III-B, with postures excluding the overlapping postures. Subsequently, the grasp synergy $S^{g}$ has been constructed based on the grasp posture sequence $X^{g}$ .

For comparative evaluation, the task-specific synergy was introduced. The task-specific synergy was made of multi-fingered hand postures extracted while performing each specific task, e.g., opening/closing a scissors and pressing a switch, independently. The hand poses were extracted by teleoperating the 10 DoFs humanoid hand. In each case, the task was performed ten times. Then, the task-specific synergies were created by principal component analysis as $S^{T}_{scissors}$ and $S^{T}_{switch}$ .

TABLE III: The functions of each finger when performing the scissors-task and the switch-task.

Joint

Function

Scissors-task

Switch-task

Thumb rot.

Thumb MCP

Index MCP

Index PIP

Middle MCP

Middle PIP

Ring MCP

Ring PIP

Pinky MCP

Pinky PIP

We consider each task as having to phases, a grasping phase, and a manipulation phase. For the grasping phase, the grasp synergy-based control method is used. For the manipulation phase, the proposed FDMS-based control method is used. To construct FDMS, it is necessary to define the function of the fingers for each task, scissors-task, and switch-task. Table III shows the defined function of each finger when performing the scissors-task and the switch-task. These functions are empirically defined. The functions are assigned according to Section IV-B. The set consists of joint assigned M, and defined as $J^{\mathrm{sub}}$ . We construct FDMS by using $J^{\mathrm{sub}}$ according to Section III-B. Note that the maximum number of principal components is the number of joints assigned M. The switch-task has two phases: the grasping phase and the manipulation phase. Therefore, we switch synergies according to the Switching Synergy Framework, from grasp synergy to FDMS.

VI-E Evaluation

VI-E1 The task success rate

In this experiment, we examined how many dimensions are reduced when using each synergy-based control method to perform the tasks in the real environment. The performance of conventional synergies was evaluated mainly by the contribution ratio which is percent variance accounted for by a principal component. However, it has not been verified whether tasks can be performed by using a synergy with a high contribution ratio. To evaluate performance, we focus on the task success rate when replaying the posture sequences $\bm{X}^{E}_{m}(m=1\cdots 10)$ approximated by the synergy according to Section III-C on the teleoperating system. The task success rate is calculated based on the percentage of successful tasks played by the approximated posture sequence.

VI-E2 Practical demonstration

We considered two more challenging tasks, a Cutting-task and a Spray-task. In the Cutting-task, the scissors placed on the workspace in advance are grasped, and a string is cut by operating the scissors. Similarly, in the Spray-task, the hand grasps and manipulates a spray bottle placed on the workspace by pulling the lever. We evaluate whether the Cutting-task and Spray-task can be realized, as more practical tasks. In this experiment, the trajectory of the hand is generated by our teleoperation system, but note that the posture obtained from CyberGloveIII is approximated by our FDMS method in real-time. The pose of the object is known, and the movement of the arm is also determined in advance.

VII results

For each task, we show the plot of the success rate as a function of the number of principal components in Figure 5 (scissors-task) and Figure 6 (switch-task). Both results show that, as the number of principal components increases, the task success rate increases. Regardless of the number of principal components, the success rate of using FDMS is higher than the success rate of using conventional grasp synergy.

In the scissors-task, Figure 5, the highest success rate when using the conventional grasp synergy is 70%. In contrast, using our FDMS with four principal components achieved a task success rate of 90%. By fixing the position of some fingers, which are not necessary for the task, we reduce the overall DoF. Thus the accuracy of posture expression of FDMS improves due to the mathematical properties of PCA. On the contrary, if the position of those fingers is not fixed, then they may interfere with the manipulated object.

On the other hand, task-specific synergy achieved a task success rate of almost 100% regardless of the number of principal components used. Therefore, when the number of tasks required for a humanoid hand is small, the said result implies that a high task success rate can be achieved by switching task-specific synergies according to the Switching Synergy Framework.

In the switch-task, Figure 6, when using the FDMS with just a single principal component, the task success rate is always 0 since the fingers whose function is the manipulating do not work well. This result shows that the thumb of the multi-fingered hand needs at least two DoFs. When using the grasp synergy with two principal components and the FDMS with two principal components, a task success rate of 90% is achieved. In this case, the dimensions of the control space of the grasp synergy and the FDMS are always two, so it can be said that the effect of dimensionality reduction has been fully demonstrated.

VIII discussion

As shown in Figure 5 and Figure 6, to achieve the task success rate of 100% using the task-specific synergy, the scissors-task requires only one principal component, while the switch-task requires at least four principal components. The reason is that, in the switch-task, the fingers assigned for manipulating function changes from the index, middle, ring, and pinky fingers to the only thumb according to the transition of the task. Since the task-specific synergy assuming the coordination of all fingers, it usually cannot perform multiple functions. This result suggests that the Switching Synergy Framework, which can switch FDMS, is effective for tasks that are required to change the fingers’ function.

Figure 7 shows the hand performing the Cutting-task and Spray-task when using FDMS-based control and the Switching Synergy Framework. In the Cutting-task, the hand can grasp the scissors and cut the string by using only two principal components for the FDMS. In the spray-task, the hand has to grasp with all fingers and then push its lever with only the index finger while the other fingers maintain the grasping state. Maintaining the grasp state is essential for manipulating the lever, this fact confirms the importance of fixing some of the fingers. Thus, the grasp synergy-based control with a few principal components cannot realize the spray-task. In contrast, FDMS and the Switching Synergy Framework can realize the spray-task by using 2 DoFs grasp synergy and 1 DoF FDMS. These facts suggest that FDMS and Switching Synergy Framework effectively supply the control spaces spanned by a few principal components, which can realize various sequential tasks.

The conventional grasp synergy can generate trajectories to grasp objects [12] and perform in-hand manipulation such as changing the pose of an object [1]. Therefore, it is considered that FDMS, which is a form of grasp synergy limited by joint subspace, can generate the trajectory for grasping and manipulation.

In this study, we empirically determine the fingers’ functions and its change during each task. However, in order to use FDMS and Switching Synergy Framework in a real environment, the functions required for a task should be automatically determined and assigned to the finger. In this study, it was possible to limit the number of FDMS required for manipulating an object to 12. Interesting future research would be to construct a Markov process with these 12 FDMSs as states. It is possible to construct a system that selects an appropriate FDMS from observation information of the operation target.

We define two functions to compose FDMS, but we can introduce more detailed functions to make our FDMS more useful. For instance, we can introduce a maintaining grasp function derived from the fixed function to control the internal force of a grasped object.

TABLE IV: The number of principal components to achieve cumulative contribution ratio 80% or 90%

Joint subspace	Movement unit	Number of PCs
Joint subspace	Movement unit	>80%	>90%
$J^{\mathrm{sub}}_{1}$	MMMMM	3	4
$J^{\mathrm{sub}}_{2}$	MFFFF	1	2
$J^{\mathrm{sub}}_{3}$	FMMMM	2	3
$J^{\mathrm{sub}}_{4}$	MMFFF	2	3
$J^{\mathrm{sub}}_{5}$	FFMMM	1	2
$J^{\mathrm{sub}}_{6}$	FFFMM	1	2
$J^{\mathrm{sub}}_{7}$	FMFFF	1	2
$J^{\mathrm{sub}}_{8}$	MMMFF	3	4
$J^{\mathrm{sub}}_{9}$	FFMFF	3	4
$J^{\mathrm{sub}}_{10}$	FFMMF	1	2
$J^{\mathrm{sub}}_{11}$	MMMMF	3	4
$J^{\mathrm{sub}}_{12}$	FMMMF	2	3

The posture expression ability required to realize a manipulating function is considered to be different for each FDMS. If we develop a multi-fingered hand that realizes FDMS, it is necessary to determine in advance how many principal components should be used for each synergy. Now, we discuss a method using a cumulative contribution ratio as one of the methods. Cumulative contribution ratio is the sum of contribution ratios from the first principal component to the $f$ -th principal component in PCA and indicates how well the variance of the samples can be explained. Therefore, if an FDMS realizes a high cumulative contribution ratio, it is considered that the FDMS has a high posture expression ability. Table IV shows the number of principal components required for each FDMS shown in Table II to achieve a cumulative contribution ratio of 80% or 90%. The FDMS is the same as the one used in the experiment, defined in Section VI-D. Note that the maximum number of principal components of each FDMS is twice the number of M from the definition and the configuration of 10 DoFs multi-fingered hand. The results suggest that at most three and four principal components are sufficient to achieve the cumulative contribution ratios of 80% and 90%, respectively.

IX conclusion

In this paper, we proposed the Functionally Divided Software Synergy method, which can realize manipulation tasks using a multi-fingered hand, with a high task success rate. It was confirmed that fixing some specific fingers was important for manipulating objects. Experiments confirmed that FDMS is effective for manipulation tasks. Moreover, the required FDMS to manipulate an object can be limited to 12 based on the motion analysis of the human hand, and the cost is lower than constructing task-specific synergies, which need to configure synergies for each particular task.

Future works will aim to develop the method to design an under-actuated multi-fingered hand leveraging FDMS, and the system to assign functions to fingers according to information of an object.

References

[1] C. D. Santina, C. Piazza, G. Grioli, M. G. Catalano, and A. Bicchi, “Toward dexterous manipulation with augmented adaptive synergies: The pisa/iit softhand 2,” IEEE Trans. on Robotics, pp. 1–16, 2018.
[2] M. G. Catalano, G. Grioli, A. Serio, E. Farnioli, C. Piazza, and A. Bicchi, “Adaptive synergies for a humanoid robot hand,” in 2012 12th IEEE-RAS Int. Conf. on Humanoid Robots (Humanoids 2012), 2012, pp. 7–14.
[3] J. B. Rosmarin and H. H. Asada, “Synergistic design of a humanoid hand with hybrid dc motor - sma array actuators embedded in the palm,” in 2008 IEEE Int. Conf. on Robotics and Automation, May 2008, pp. 773–778.
[4] C. H. Xiong, W. R. Chen, B. Y. Sun, M. J. Liu, S. G. Yue, and W. B. Chen, “Design and implementation of an anthropomorphic hand for replicating human grasping functions,” IEEE Trans. on Robotics, vol. 32, no. 3, pp. 652–671, 2016.
[5] G. Palli, C. Melchiorri, G. Vassura, U. Scarcia, L. Moriello, G. Berselli, A. Cavallo, G. D. Maria, C. Natale, S. Pirozzi, C. May, F. Ficuciello, and B. Siciliano, “The dexmart hand: Mechatronic design and experimental evaluation of synergy-based control for human-like grasping,” The Int. J. of Robotics Research, vol. 33, no. 5, pp. 799–824, 2014.
[6] B. A. Kent, J. Lavery, and E. D. Engeberg, “Anthropomorphic control of a dexterous artificial hand via task dependent temporally synchronized synergies,” J. of Bionic Engineering, vol. 11, no. 2, pp. 236 – 248, 2014.
[7] M. Baril, T. Laliberté, C. Gosselin, and F. Routhier, “On the design of a mechanically programmable underactuated anthropomorphic prosthetic gripper,” J. of Mechanical Design, vol. 135, no. 12, p. 121008, 2013.
[8] N. Roy, P. Newman, and S. Srinivasa, An Object-Based Approach to Map Human Hand Synergies onto Robotic Hands with Dissimilar Kinematics. MIT Press, 2013, pp. 504–.
[9] J. Romero, T. Feix, C. H. Ek, H. Kjellström, and D. Kragic, “Extracting postural synergies for robotic grasping,” IEEE Trans. on Robotics, vol. 29, no. 6, pp. 1342–1352, Dec 2013.
[10] R. Chattaraj, A. S. Kundu, O. Mazumder, B. Bepari, and S. Bhaumik, “Tele-operation of robot hand exploiting human hand synergy,” in Proc. of the 2015 Conf. on Advances In Robotics, ser. AIR ’15, 2015, pp. 34:1–34:6.
[11] T. Wimböck, B. Jahn, and G. Hirzinger, “Synergy level impedance control for multifingered hands,” in 2011 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Sep. 2011, pp. 973–979.
[12] F. Ficuciello, “Synergy-based control of underactuated anthropomorphic hands,” IEEE Trans. on Industrial Informatics, vol. 15, no. 2, pp. 1144–1152, Feb 2019.
[13] F. Ficuciello, D. Zaccara, and B. Siciliano, “Synergy-based policy improvement with path integrals for anthropomorphic hands,” in 2016 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Oct 2016, pp. 1940–1945.
[14] F. Ficuciello, G. Palli, C. Melchiorri, and B. Siciliano, “Postural synergies of the ub hand iv for human-like grasping,” Robotics and Autonomous Systems, vol. 62, no. 4, pp. 515 – 527, 2014.
[15] M. Santello, M. Flanders, and J. F. Soechting, “Postural hand synergies for tool use,” J. of Neuroscience, vol. 18, no. 23, pp. 10 105–10 115, 1998.
[16] M. Gabiccini, G. Stillfried, H. Marino, and M. Bianchi, “A data-driven kinematic model of the human hand with soft-tissue artifact compensation mechanism for grasp synergy analysis,” in 2013 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2013, pp. 3738–3745.
[17] J. N. Ingram, K. P. Körding, I. S. Howard, and D. M. Wolpert, “The statistics of natural hand movements,” Experimental brain research, vol. 188, no. 2, pp. 223 – 36, 2008.
[18] N. Kamakura, “Form of hand and motion of hand (in japanese),” Ishiyaku Publishers, 1989.
[19] E. Todorov and Z. Ghahramani, “Analysis of the synergies underlying complex hand manipulation,” in The 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, vol. 2, 2004, pp. 4637–4640.
[20] T. Feix, J. Romero, H. B. Schmiedmayer, A. M. Dollar, and D. Kragic, “The grasp taxonomy of human grasp types,” IEEE Trans. on Human-Machine Systems, vol. 46, no. 1, pp. 66–77, 2016.
[21] W. Chen, C. Xiong, and S. Yue, “Mechanical implementation of kinematic synergy for continual grasping generation of anthropomorphic hand,” IEEE/ASME Transactions on Mechatronics, vol. 20, no. 3, pp. 1249–1263, June 2015.
[22] CyberGlove Systems LLC, http://www.cyberglovesystems.com/cyberglove-iii.

Functionally Divided Manipulation Synergy for Controlling Multi-fingered Hands