External Correlates of Adult Digital Problem-Solving Behavior: Log Data Analysis of a Large-Scale Assessment

The process data analyzed in the current study came from the computer-based assessment of PSTRE in the 2012 PIAAC study. For each PSTRE item, the test environment resembled commonly seen informational and communicative technology (ICT) platforms, such as e-mail client, web browser, and spreadsheet. Test-takers were prompted to complete specific tasks on these interactive platforms. Under the PIAAC framework, PSTRE is defined as the use of digital technology, communication tools, and internet to obtain and evaluate information, communicate with others, and perform practical tasks organisation2012literacy. Thus, successful responses to the PSTRE items would require not only familiarity with ICT tools and platforms but also the ability to obtain, process, and communicate information in digital environments.

A sample item that resembles PSTRE tasks is shown in Figure 1: the instructions for the tasks are presented on the left side of the screen, and on the right is a simulated web browser, with similar functionalities as typical web browsers, such as clicking links, going back and forth, and bookmarking pages. For this particular item, test-takers are presented with five web pages returned from a search of “Job search” and are asked to bookmark all pages that do not require registration or fees. By clicking on each link, test-takers will be directed to the corresponding website. For example, by clicking the second link, “Work Links”, examinees will be directed to Figure 2. And by further clicking on the “Learn More” button, they will be directed to the webpage shown in Figure 3. When examinees have finished, they can leave the item by clicking on the right arrow icon (“Next”) below the item instructions, where they will be presented with a pop-out window with two options, namely confirming exit (recorded as “Next_OK”) or returning to the task (recorded as “Next_Cancel”).

The sequence of clicks and keystrokes (if applicable) by the test-taker on the simulated browser are recorded as an action sequence, with “Start” at the beginning indicating the start of the question, and “Next, Next_OK” at the end, indicating the examinee has confirmed to move to the next item. For example, if a test-taker clicked the second link (“Work links”) on the initial page, clicked “Learn More” on the next page (Figure 2), clicked the “Back” button on the toolbar twice to return to the home page, and clicked “Next” in the left panel and “OK” in the pop-up window, this test-taker’s action sequence will be recorded as “Start, Click_W2, Click_Learn_More, Toolbar_Back, Toolbar_Back, Next, Next_OK”.

In the current study, we analyzed the examinees’ action sequences from $14$ PSTRE items. These items shared similar environments as the one exemplified above, and both the polytomous final outcomes and the action sequences of the subjects were recorded. The data from $3645$ examinees, including $1656$ male and $1989$ female examinees, were used. The examinees’ age ranged between $16$ and $66$ years, with an average of $38.93$ years, and they came from five countries, including the United Kingdom (England and Northern Ireland), Ireland, Japan, the Netherlands, and the United States of America. These examinees completed all $14$ PSTRE items, presented as two separate blocks of $7$ items each ¹¹1In the PIAAC computer-based assessment, examinees were randomly routed to modules of PSTRE, literacy, or numeracy items. The current study used participants who received two PSTRE modules.. Examinees were additionally administered a background survey that covered their educational status (e.g., total years of education, highest level of education), vocational status (e.g., employment, income, job type), and their use of different skills (e.g., reading, writing, numeracy, and ICT) at home and at work. In the present study, we attend to the participants’ age, gender, income, use of ICT skills at home and at work, and years of education.

Table 1 presents a brief description of the $14$ PSTRE items, including task name, the types of environments involved, percent of individuals who received full credit, minimum action sequence length, maximum sequence length, average sequence length, and the number of action types. Some of the tasks required the examinees to switch between environments. For instance, in item U23: Lamp Return, examinees needed to navigate between the web browser and the email client to obtain necessary information and solve the task. For these questions that involve multiple environments, examinees could switch to a particular environment by clicking the corresponding pane label (e.g., “Web”, “Email”) located at the bottom of the test screen.

The 2012 PIAAC background questionnaire collected participants’ responses on hundreds of items, which covered their education, social background, engagement in literacy, numeracy, and ICT use at home and at work, language background, employment information, and some non-economic information, such as health status and political efficacy irwinforeword. Six background variables were considered, including the test takers’ reported age, gender, logarithm of hourly income (log(Income)), extent of ICT skill use at home (ICTHome), ICT skill use at work (ICTWork), and total years of education (YRSEdu). To adjust for country effects on income, each individual’s hourly income was first converted to US dollars based on 2012 exchange rates and then centered based on the observed median in his/her country. Similarly, the country effects in YRSEdu was adjusted by subtracting the country’s median from each test-taker’s years of education. Scores on ICTHome and ICTWork were derived based on test-takers’ responses to a series of corresponding survey items. As some of the tasks appeared to also involve numericy skills (e.g., spreadsheets), performance (i.e., plausible values²²2The PIAAC survey derived plausible values of individuals’ cognitive performance (e.g., numeracy performance) based both on the responses to cognitive assessments and background covariates. The numeracy performance variable used in the current study is based on the numeracy plausible values recorded in the official PIAAC data.) on numeracy was also included as an external cognitive variable. We note that missingness exists for some of the background variables, including income, ICTHome, ICTWork, and YRSEdu. For predicting a particular variable, individuals with missing values were case-wise deleted. The mean, standard deviation, and number of non-missing observations of each continuous variable, as well as their pairwise Pearson correlations, are reported in Table 2. Compared to male participants, female participants on average showed lower log hourly income, ICTWork, and numeracy performance, with small Cohen’s $d$ effect sizes. Additionally, the correlations between each variable and the overall PSTRE performance (derived from PSTRE item responses) are also presented.

The purpose of MDS is to find a finite dimensional representation of the observed data that can best preserve the pairwise dissimilarities between individual observations <e.g.,>borg2003modern. If we denote the observed data of individual $i\in\{1,\ldots,N\}$ by $s_{i}$, the typical aim of MDS is to find a mapping of each $s_{i}$ to a $K$-dimensional numerical vector $\mathbf{x}_{i}\in\mathbb{R}^{K}$ which minimizes

where $d_{ij}=d(s_{i},s_{j})$ is the dissimilarity between observations $s_{i}$ and $s_{j}$ according to some predefined distance measure $d(\cdot)$, and $\|\mathbf{x}_{i}-\mathbf{x}_{j}\|=\sqrt{(\mathbf{x}_{i}-\mathbf{x}_{j})^{\prime}(\mathbf{x}_{i}-\mathbf{x}_{j})}$ is the Euclidean distance between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$. In the case of process feature extraction, $s_{i}$, $s_{j}$ are the observed action sequences of individuals $i$ and $j$. Intuitively, MDS transforms each observation into a point in the $K$-dimensional Euclidean space, so that observations that are similar to each other (i.e., low on $d_{ij}$) are close as well in the Euclidean space (i.e., low on $\|\mathbf{x}_{i}-\mathbf{x}_{j}\|$), and observations that are dissimilar are farther apart in the Euclidean space after the transformation. MDS has been previously applied to psychometrics research problems to understand individual differences on various cognitive and non-cognitive tasks <see >takane200611, but MDS-based analysis of action sequences in psychometric assessments was not studied until recently tang2019mds.

Essential to MDS is the choice of an appropriate distance measure, $d$, to capture the dissimilarity between individual observations. When observations are action sequences on interactive items, tang2019mds tang2019mds proposed to use an order-based sequence similarity measure <OSS;>gomez2008similarity, which allows for the quantification of the dissimilarity between ordered, categorical, variable-length event sequences. Denote the observed action sequence of the $i$th test-taker by $\mathbf{s}_{i}=(s_{i1},\ldots,s_{iL_{i}})$, where $s_{it}$ is the $t$th action performed by the subject, $L_{i}$ is the total number of actions, and $L_{i}^{a}$ is the number of occurrences of a particular action $a$. The dissimilarity between the action sequences of any two test-takers, $d_{ij}$, is given by

Here, $C_{ij}$ is the set of common actions occurred in both sequences, $K_{ij}^{a}=\min\{L_{i}^{a},L_{j}^{a}\}$, $s_{i}^{a}(m)$ is the serial position of the $m$th occurrence of event $a$, and $U_{ij}$ is the set of unique actions taken by test-taker $i$ but not by $j$. Intuitively, $f(\mathbf{s}_{i},\mathbf{s}_{j})$ quantifies how the serial positions of common actions in the two sequences differ, and $g(\mathbf{s}_{i},\mathbf{s}_{j})$ counts the number of actions unique to each sequence. In turn, the dissimilarity between $i$ and $j$, $d_{ij}$, takes into account the differences both in the ordering of same actions and the types of actions taken.

After calculating the dissimilarities ($d_{ij}$) between each pair of individuals’ action sequences, the optimization problem that minimizes Equation (1) is solved to find $\mathbf{x}_{1},\ldots,\mathbf{x}_{N}$. As the transformed $\mathbf{x}$’s preserve as much as possible the pairwise dissimilarities in the observed action sequences, $\mathbf{x}_{i}$ could be seen as a $K-$dimensional latent feature vector which contains information of the original action sequence $\mathbf{s}_{i}$. Readers are referred to tang2019mds tang2019mds for more details on the OSS distance, an illustrative example on how the distance is calculated, and the algorithms for performing the MDS and choosing the dimension of $\mathbf{x}$, $K$.

Another approach to feature extraction from variable-length action sequences, which implements a sequence-to-sequence autoencoder (Seq2seq), was proposed in tang2019seq2seq tang2019seq2seq. Commonly used for information compression from phrases or sentences in natural languages, an autoencoder seeks to encode categorical event sequences into lower-dimensional latent vectors, which can then be used to restore the original sequences. A Seq2seq autoencoder is an artificial neural network with two main components, an encoder function $\phi(\cdot)$ that transforms the original input sequence $\mathbf{s}_{i}$ into a fixed-dimensional latent vector $\boldsymbol{\theta}_{i}$, and a decoder function $\psi(\cdot)$ that maps the latent vector $\boldsymbol{\theta}_{i}$ to a reconstructed version of the original sequence, $\hat{\mathbf{s}}_{i}$. The action sequence feature extraction procedures proposed in tang2019seq2seq tang2019seq2seq employed a recurrent neural network-based autoencoder with multiple hidden layers. The model is structured as follows.

In the encoding stage, each action in the sequence is mapped to a $K-$dimensional continuous representation (i.e., an embedding, $\mathbf{e}_{it}$) based on the surrounding context using an embedding layer mikolov2013efficient. Following the action embedding, a recurrent layer is applied to the embedded sequences. The recurrent layer creates a $K-$dimensional hidden state ($\boldsymbol{\theta}_{it}$) for each time step $t=1,\ldots,L_{i}$. The hidden state at time $t$, $\boldsymbol{\theta}_{it}$ is a function of the hidden state at the previous time step, $\boldsymbol{\theta}_{it-1}$, and the embedded action performed at time $t$, $\mathbf{e}_{it}$, that is, $\boldsymbol{\theta}_{it}=f(\boldsymbol{\theta}_{it-1},\mathbf{e}_{it}).$ Different choices for the recurrent function, $f(\cdot)$, have been proposed, including the long short-term memory <LSTM; >hochreiter1997long architecture and the gated recurrent unit <GRU;>cho2014properties architecture. For feature extraction from action sequences, tang2019seq2seq tang2019seq2seq adopted the GRU for the recurrent function, which learns from the observed data to update or reset the hidden states depending on the context and the action type. In this way, the recurrent states ($\boldsymbol{\theta}_{i1},\ldots,\boldsymbol{\theta}_{iL_{i}}$) accumulate information in the action sequence over time, and the hidden state at the final time step, $\boldsymbol{\theta}_{iL_{i}}$, summarizes the information contained in the entire action sequence. We simplify the notation for the last recurrent state (i.e., the encoder output, $\boldsymbol{\theta}_{iL_{i}}$) to $\boldsymbol{\theta}_{i}$, which will be used to reconstruct the observed sequence in the decoding stage.

The first layer of the decoding stage is also a recurrent layer with $L_{i}$ time steps, with initial hidden state $\mathbf{y}_{i1}=\mathbf{0}$ and the $t$th hidden state obtained by $\mathbf{y}_{it}=f^{\prime}(\mathbf{y}_{it-1},\boldsymbol{\theta}_{i}).$ Here, $f^{\prime}$ is another recurrent function, such as the GRU, with a different set of parameters from $f$. Note that the decoder recurrent states are obtained by feeding in the same encoder output, $\boldsymbol{\theta}_{i}$, for $L_{i}$ times and accumulating the information over time through $f^{\prime}(\cdot)$. The last layer of the decoder is a softmax layer, where, for each time point $1\leq t\leq L_{i}$, the decoder hidden state at time $t$ ($\mathbf{y}_{it}$) is used to predict the probability that individual $i$ takes each possible action. Intuitively, the decoder aims at reconstructing the distribution of the original sequence $\mathbf{s}_{i}$ using the encoder output $\boldsymbol{\theta}_{i}$.

A graphical illustration of the Seq2seq is presented in Figure 4. Because the output from the last time step of the encoder stage, $\boldsymbol{\theta}_{i}$, is used to reconstruct the original sequence in the decoder, $\boldsymbol{\theta}_{i}$ can be regarded as a summary of the entire sequence. For this reason, tang2019seq2seq tang2019seq2seq proposed the use of $\boldsymbol{\theta}_{i}$ as the extracted features from the action sequences. For more information about the Seq2seq, including the explicit formula for each layer of the neural network and the details on model estimation, readers are directed to tang2019seq2seq tang2019seq2seq.

As a concrete illustration of the extracted process features, Table 3 presents the features extracted from three participants’ processes on item U06b using either MDS or Seq2seq, with number of dimensions set as $K=5$. Omitting “Start” and “Next, Next_OK” at the beginning and the end, the three participants’ action sequences were as follows:

• •

  Examinee 1: “Click_W4, Toolbar_Web_Back, Response_Open, Response_4, Response_Close”;

• •

  Examinee 2: “Click_W2”;

• •

  Examinee 3: “Click_W1, Toolbar_Web_Back, Click_W2”.

In practice, $K$ is often chosen to be larger than $5$ to preserve high-dimensional information in the action sequences.

For each of the $14$ PSTRE items, $K=100$ features were extracted from the action sequences of $N=3645$ subjects using MDS and Seq2seq, respectively. Feature extraction with MDS was implemented in R with C++ integrations eddelbuettel2011rcpp. Feature extraction with Seq2seq was implemented in Python with the Keras module chollet2015keras. Parameters of the neural net were optimized by minimizing the categorical cross-entropy loss between the observed sequences and predicted distributions of each action at each time step in the decoder, and the RMSProp tieleman2012lecture optimizer was used. Following feature extraction, principal component analysis <PCA;>pearson1901liii was applied to the $100-$dimensional features of each item to obtain linearly independent principal features ordered by amount of explained variance.

The $100$ principal features of each PSTRE item were used to draw predictions about the examinees’ age, gender, transformed hourly income, use of ICT tools at home, use of ICT tools at work, years of education, and numeracy performance. For each item, three different types of predictors were considered, namely (1) subjects’ polytomous scores on the item, (2) the $100$-dimensional principal features of each subject extracted from action sequences using the Seq2seq, and (3) the $100$-dimensional principal features from MDS. In addition to predictions with the features from each single item, cumulative predictive powers of final scores, MDS and Seq2seq were also evaluated with features combined across multiple items. Specifically, polytomous score, MDS and Seq2seq features were added one item at a time to examine how prediction accuracy grows as the information from more items were used. Generalized linear models (GLMs) were fitted through the R glmnet package friedman2009glmnet to make the predictions, with a logit link function for the prediction of gender and an identity link function for the predicting the continuous variables. An $L_{2}$ penalty was applied in the GLMs to avoid overfitting. To reduce Monte Carlo error, ten replications were carried out for each prediction task. Specifically, in each replication, the data were randomly partitioned into $3$ subsets, a training set ($70\%$), a validation set ($10\%$), and a test set ($20\%$). The parameters of the GLM were estimated based on the training data, the optimal penalty term for $L_{2}$ regularization was chosen to minimize the loss function on the validation data, and the prediction accuracy of the GLM was evaluated on the test data. The evaluation criteria of prediction accuracy were as follows: For gender, we calculated the area under the curve (AUC) of the receiver operator characteristic (ROC) curve on the test sample. For the rest of the predicted variables, we calculated the out-of-sample Pearson correlation ($O.S.R$), which is the correlation between the predicted and observed values of a dependent variable in the test sample.

For selected items whose process features demonstrated especially high associations with a specific variable, interpretations for the process-variable associations were further sought. To do this, partial least squares <PLS;>wold2002partial decomposition was applied to extract the top $M$ orthogonal components from the action-sequence features that could best account for the covariance between the features ($\mathbf{X}$) and the dependent variable ($Y$). PLS is commonly used for obtaining lower-dimensional representations of high-dimensional features. Unlike PCA, where the top $M$ principal components capture the most variance in $\mathbf{X}$, in PLS, the top $M$ components capture the most of the covariance between $\mathbf{X}$ and $Y$. Thus, by seeking an $M$-dimensional PLS approximation to the $100$-dimensional action sequence features with respect to a dependent variable of interest, we obtain a set of $M$ linearly independent components, each explaining some of the associations between the MDS/Seq2seq features and the dependent variable. After PLS component extraction, individual’s action sequences were sorted based on each of the $M$ PLS components. By inspecting how the observed action sequences change as the PLS component score increases, specific pattern(s) that explain the covariance between process features and the predicted variable can be pinpointed.

In the current study, the dimension for the PLS approximation, $M$, was chosen as follows: We first ran PLS with $100$ components. For each $M^{\prime}\in\{1,\ldots,100\}$, we calculated the root-mean-squared-error for the prediction (RMSEP) of $Y$ using the first $1,\ldots,M^{\prime}$ PLS components as linear regression predictors, i.e., $RMSEP_{M^{\prime}}=\sqrt{\frac{\sum_{i=1}^{N}(\hat{y}_{i}-y_{i})^{2}}{N}}$. Denote the number of dimensions with the absolute lowest RMSEP by $M^{\prime}_{min}$ and the standard error of RMSEP across all $M^{\prime}$s by $SE(RMSEP)$, then the optimal number of dimensions, $M$, was chosen to be the smallest number of dimensions with RMSEP within one standard error of $RMSEP_{M^{\prime}_{\text{min}}}$, in other words, $M=\min\{M^{*}:RMSEP_{M^{*}}<RMSEP_{M^{\prime}_{\text{min}}}+SE(RMSEP)\}.$

For each of the $M$ PLS components, we searched for patterns in the action sequences associated with the component. To do this, we first ranked all test-takers based on the PLS component, then, from lowest to highest, at an interval of $50$, we inspected the observed sequences of the test-takers. By looking at how the sequences changed as the component score increased, patterns in the action sequences associated with the PLS component were identified. We further verify the relationship between the visually spotted patterns and the PLS component by plotting how the pattern changes as a function of the PLS component score. More specifically, at each PLS component score value, we looked at $100$ individuals whose observed PLS component score was closest to this value and calculated the proportion demonstrating a specific pattern or the average frequency of a specific pattern among these $100$ individuals. This proportion/averaged frequency is then plotted against the PLS score.

The single-item prediction accuracy of the continuous variables and gender is reported in Figures 5 and 7 (left), respectively. In each subplot, the $x$-axis represents the item used for prediction, the $y$-axis represent the evaluation metric, i.e., averaged test-sample $O.S.R$ or the AUC across $10$ replications, and the three bars with different shades represent the prediction results based on polytomous score, $100$ Seq2seq features, and $100$ MDS features, respectively. As was mentioned in the item descriptions, most of the PSTRE items involved one or two of the following environments: spreadsheet (SS), web browser (Web), or email box (Email), and items involving similar environments could share some common actions. To examine environment effect on the strength of associations, items are grouped by environments separated by vertical dashed lines. The combined prediction results with multiple items for the continuous variables and gender are presented in Figures 6 and 7 (right), respectively. Each line plot presents the increment of the cumulative prediction accuracy ($O.S.R$ or AUC) on a particular variable, as each item’s features are additionally added to the predictors.

From the results on the general prediction accuracy, it was observed that not only do sequence features provide additional information about the test-takers, the prediction powers of different items also differed widely. This calls for a closer look at action sequences that contribute to the additional predictive power. As the MDS features often demonstrated highest prediction power, for each continuous variable, we performed PLS decomposition on the MDS features from some of the most predictive items. The current subsection presents the findings from the PLS decomposition and interpretations of the PLS components. Results on age are presented in details, and for succinctness, results on the other variables are summarized.