Measuring Agreement on Linguistic Expressions in Medical Treatment Scenarios

The Toronto Extremity Salvage Score (TESS) is a disease-specific measure developed for patients undergoing limb preservation surgery for tumours of the extremities [1]. It is a patient-completed questionnaire with questions framed to ask about the difficulty experienced performing daily activities over the last week aimed to monitor the effects of therapeutic interventions. TESS is commonly administered at four time points: the first session (which is commonly before surgery) and 12, 18 and 24 months from then on. The TESS consists of 30 and 29 items for lower limb and upper limb cases respectively with items such as the the one shown in Fig. 1.

As can be seen, difficulty is rated on a 5-point Likert-type scale ranging from “not at all difficult” to “impossible to do”. Commonly, after having been completed by the patient, the whole set of answers is used to generate a standardized score ranging from 0 to 100. This evaluated TESS is finally analysed by surgeons/physiotherapists in order to measure changes in physical functions over time.

Fuzzy Sets are sets in which, unlike in traditional set theory, the membership of each element is a number in the interval $[0,1]$. Given a universe of discourse $X$, a FS $A$ is represented as a set of ordered pairs of an element $x$ and its membership value within $A$, denoted by ${\mu}_{A}(x)$, i.e.

Alpha-cuts (or $\alpha$-cuts) are an important concept in FSs, given that a FS $A$ can also be represented as a collection of its $\alpha$-cuts [12]. An $\alpha$-cut of a FS $A$ is a crisp set defined as

The Interval Agreement Approach (IAA) was introduced in [4] as a method for generating FSs from surveys in which answers are given as interval-valued data representing uncertainty in people’s opinions/perceptions. It is built on top of the work presented in [13], where an agreement-based method [14] of capturing interval-valued survey data is demonstrated.

The IAA considers two types of intervals in the process of capturing responses: crisp (no uncertainty about the interval endpoints) and uncertain (each endpoint modelled itself as a crisp interval). It considers two types of uncertainty to be modelled through different dimensions of the resultant FSs, namely inter-source (variation among a group of participants) and intra-source (variation in the opinion of a particular participant). Depending on the data, the IAA can generate:

• •

  Type-1 FSs. When crisp intervals and either inter- or intra-source uncertainty are modelled in the primary degree of membership by combining multiple intervals,

• •

  Interval Type-2. When uncertain intervals and also, either inter- or intra-source uncertainty is modelled in the primary degree of membership by combining multiple intervals,

• •

  General Type-2 FS based on zSlices [15]. In this case, both inter- and intra-source uncertainty are being modelled through the primary and secondary degrees of membership.

In this paper, we are focusing on the agreement ratio among stakeholders (inter-participant uncertainty). Such uncertainty is captured through crisp intervals and consequently, it is modelled by employing the IAA to generate T1 FSs. We provide a brief review of generating T1 FSs using the IAA below:

Consider $N$ (closed) intervals ${\bar{A}}_{i}=[{l}_{\bar{{A}_{i}}},{r}_{\bar{{A}_{i}}}]$, $i\in\left\{1,...,N\right\}$ to be modelled as a T1 FS $A$ where the intervals are delimited by ${l}_{\bar{{A}_{i}}}$ and ${r}_{\bar{{A}_{i}}}$. The membership function of $A$ (denoted by ${\mu}_{A}$) given in (3).

where ${y}_{i}=\frac{i}{N}$and $/$ refers to the common notation of membership, not division. For practical applications, (3) can be calculated in a recursive and discrete manner by formulating the function as

The use of words as a means of communication between patient-medical staff is a natural way of expressing perceptions. However, the challenge of dealing with different interpretations of words (“words mean different things to different people”) leads to looking for a standardised vocabulary, as has been suggested by the European Union [16]. Therefore, there is a need for a means to analysing how similar the understanding of key words is across stakeholders in patient treatment.

The aim of the agreement ratio is to provide a number contained within the interval $[0,1]$ representing the extent of agreement among a group of surveyed stakeholders (without discarding particular responses) whose responses are modelled in the given FS. Therefore, a method for generating data-driven FS models, considering the stakeholders opinions is key. As such, we have considered FSs generated with the IAA and EIA methods to analyse the agreement obtained.
Two key assumptions for the measure are:

1. 1.

   Agreement is when 2 or more sources coincide in a given point/value/opinion.

2. 2.

   The more opinions overlap at a particular region, the stronger the agreement is conceived.

Initially, we began developing the measure considering that in the IAA given the inter-participant variation is reflected directly in the primary degree of membership $y$ (or $\mu$). To illustrate, consider two simple contrasting cases in which two intervals/participants ($N=2$) are used to create a FS:

• •

  Two identical intervals. An agreement ratio must be equal to 1 given that all stakeholders (intervals) totally agree (overlap).

• •

  Two disjoint intervals. An agreement ratio must be equal to 0 given that there are no regions in which the stakeholders agree (Fig. (2b)).

From these initial cases, it can be seen that at the highest level of membership (we will refer to it as $y_{N}$), there might be one or more intervals representing regions where the $N$ intervals agree, at the ${y}_{N-1}$ level, there might be one or more overlapping regions where at least $N-1$ intervals agree and so on. Thus, if a proportion of each of the $y$ levels contained with the next lower level is calculated, then a ratio representing their quantitative relation. For example, in the FS depicted in Fig. 2a, the length of the agreement interval at the $y_{2}$ level is 2, which is equal to the one at the $y_{1}$ level and thus, the relation can be represented as $\frac{2}{2}=1$. For the FS of figure 2b, such relation between the length at level $y_{2}$ is 0 since there is not any region where both intervals overlapped and the length at level $y_{1}$ is $2+1.5=3.5$ can be represented as $\frac{0}{3.5}=0$. Moreover, considering cases with more intervals to analyse, regions with higher agreement over others with less must contribute to the ratio with “higher relevance”.

Using these cases as basis we can proceed to generalise and propose a method for calculating an agreement ratio for a FS in Section III-B.

Let ${\bar{A}}_{n}$, $n\in\left\{1,...,N\right\}$ be a set of intervals ${\bar{A}}_{i}=\left\{{l}_{{\bar{A}}_{i}},{r}_{{\bar{A}}_{i}}\right\}$. The IAA uses the set of intervals to model the overall agreement through a FS based model where the membership value of each $x\in X$ accounts for the ratio of a given x contained in the set of intervals.

An agreement ratio $\gamma$ is obtained from a T1 FS with the following equation:

where $0\leq\gamma\leq 1$, $/$ refers to division and ${y}_{i}=\frac{i}{N}$ weights the relation between immediate agreement $y$ levels in question. It can be noticed that the lowest level $y_{1}$ is not being used because the agreement is conceived when 2 or more intervals overlap. Also, we use ${\left|\bar{A}\right|}_{i}$ to represent the total length(s) of the set(s) of intervals with all possible i-tuple intersection of intervals associated to the $y_{i}$ agreement level. For example, the length ${\left|\bar{A}\right|}_{1}$ is equal to the length of the support of $A$ since it is the union of all intervals whereas ${\left|\bar{A}\right|}_{N}$ is equal to the length of the intersection of all intervals. Finally, the overall summation is divided by the sum of “weights” so the final ratio is normalised to a number in the range [0,1].

The term ${\left|\bar{A}\right|}_{N}$ can be represented as described in (6).

Considering that such calculations can involve handling a considerable number of combinations to compute as the number of participants/intervals increases, (6) can be estimated through ”discretisations” in practical applications by using alpha cuts (instead of the so-called $y$ agreement levels) such as described in Algorithm1. A consideration for the calculation of the $\gamma$ measure using alpha-cuts is: if IAA generated FS models are used, then the number of alpha cuts can be chosen to be equal to the number of intervals/participants; if any other method is used, then it depends of the number of desired ”discretisations”.

It should be noticed that, the use of alpha cuts can allow the Agreement Ratio to be applied in any T1 FS regardless the method employed to generate it from intervals. However, as stated in the motivation, the degree of membership of the FS in question is assumed to express the extent of agreement among the surveyed stakeholders. This assumption, allows the proposed measure to take advantage of different FSs (normal or non-normal, convex or non-convex ) from an interpretative point of view, in order to show so, we will analyse different FSs shapes and provide the results in next section.

Consider the FSs depicted in Fig. 3 where no assumptions about the method used to generate them has been made other than, that the membership axis represents the level of agreement among the participants. Thus, one of the FSs depicted has been chosen to be non-normal deliberately so values from the calculated Agreement ratio can be contrasted. For comparison purposes, we also consider two common shapes for FS membership functions: triangular and trapezoidal.

For the FS A, the calculated agreement ratio using 10 alpha cuts is $\gamma(A)=0.5904$ whereas for the FS B $\gamma(B)$ is $0.9578$. Note that as expected, for the FS A the agreement is considerably smaller than for FS B since its shape is sharper and shorter due to the differences in the weighted comparisons of alpha-cut lengths.

Consider the FSs $G_{1}$,$G_{2}$ and $G_{3}$ with Gaussian membership functions depicted in Fig. 4. These FSs are both normal and convex which, from the perspective of the assumptions made in order to develop the measure, indicates that all of the intervals/participants have agreed in a region. They have been arbitrarily chosen to have the same mean ($m=5$) but different standard deviations ($\sigma_{1}=0.1$, $\sigma_{2}=1.0$, $\sigma_{3}=2.0$). Again, no assumptions about the method employed to generate them have been made.

By using 10 alpha cuts, the calculated agreement ratio for the three FSs is $\gamma(G_{1})\approx\gamma(G_{2})\approx\gamma(G_{3})\approx 0.6518$ and similar results can be found by using different numbers of alpha-cuts. Further discussion of these results are found on Section V.

Consider the FS $C$ depicted in Fig. 5 generated by two intervals ${\bar{C}}_{1}=\left[2,4\right]$ and ${\bar{C}}_{2}=\left[2.5,3.5\right]$. The agreement ratio $\gamma(C)$ is calculated by dividing the total length of the union of all combinations of intervals where there is at least an intersection of 2 intervals ($y_{2}$) by the union ($y_{1}$) of all intervals.

Note that in this example with 2 intervals, it can be simply considered as the length of the intersection of both intervals divided by the length of the union.

Consider the FS $A$ depicted in Fig. 6 generated by the intervals ${\bar{D}}_{1}=\left[2,5\right]$ and ${\bar{D}}_{2}=\left[3,5\right]$, ${\bar{D}}_{3}=\left[6,8\right]$ and ${\bar{D}}_{4}=\left[3,7\right]$. The agreement ratio $\gamma(D)$ is obtained by adding the weighted (${y}_{4}$, ${y}_{3}$ and ${y}_{2}$) similarities between the lengths of the union of combinations of intervals where there is at least 4 and 3, 3 and 2, and 2 and 1 intervals respectively:

As can be seen in the above example (marked with red crosses), the length at the $y_{4}$ level is $0$, at the $y_{3}$ level is $2$, at the $y_{2}$ level is $3$ and at the $y_{1}$ level is $6$. Note that at the $y_{2}$ level there are 2 intervals which have to be added.

Now lets consider the FS $E$ depicted in Fig. 7 created using the next intervals: ${\bar{E}}_{1}=\left[2,5\right]$ and ${\bar{E}}_{2}=\left[3,5\right]$, ${\bar{E}}_{3}=\left[4,6\right]$ and ${\bar{E}}_{4}=\left[3,7\right]$. Note that they are almost the same intervals than for FS $D$ but except one and such difference allows the generated FS to have a region with total agreement, i.e., the interval $[4,5]$ . As such, we can expect a higher agreement ratio when comparing $\gamma(E)$ to $\gamma(D)$. Calculations using (5) and (6) are shown below:

$\gamma(E)=\frac{1\left(\frac{1}{2}\right)+\cfrac{3}{4}\left(\frac{2}{3}\right)+\cfrac{2}{4}\left(\frac{3}{5}\right)}{1+\cfrac{3}{4}+\cfrac{2}{4}}=\frac{4}{7}\approx 0.5778$

In our previous work [10], we described a process of interval-valued data collection from different groups of people involved in assessment of function following sarcoma surgery, namely: Patients, Physiotherapists, Surgeons and a fourth one created from the combined responses from both Physiotherapists and Surgeons (PS) which together represent the body of “medical professionals”.

We surveyed thirty-six participants (12 sarcoma surgeons, 13 physiotherapists and 12 patients undergoing lower limb salvage surgery) on 5 linguistic terms used to describe the extent of difficulty to perform daily activities: “impossible to do”,“extremely difficult”, “moderately difficult”, “a little bit difficult”, and “not at all difficult”. Subsequently, we used the gathered intervals in order to generate T1 FSs by using the IAA and calculated basic FS attributes and their respective agreement ratio using (5) and Algorithm 1 (see Table I).

Figure 8 depicts the FSs for the inter-patient agreement for the 5 linguistic descriptions (words) and different groups. Note that at first sight, by considering the width and height of the FSs, the term A little bit difficult is the most subjective and less accepted among the different groups of stakeholders. Moreover, the agreement ratio ($\gamma$) substantially facilitates performing judgements about the acceptance of the different linguistic terms in a given scenario.

From Table I, it is worthwhile to note that the linguistic terms A little bit difficult and Moderately difficult from Surgeons are contrastingly the less and most agreed, respectively. Such information is not inferred from other FS measures (e.g., height, centroid, support size) and can help to analyse through a numerical approach the aptness of different linguistic terms in specific contexts.