SMART-TBI: Design and Evaluation of the Social Media Accessibility and Rehabilitation Toolkit for Users with Traumatic Brain Injury

In Stage 1, participants were asked to browse and use Facebook for ten minutes as usual without using aids to gather insights on their general Facebook experience. While using Facebook for ten minutes, an experimenter asked the participant questions to encourage reflection on their Facebook experience (e.g., “Can you tell me your favorite part of Facebook”).

Within stage two, participants were asked to perform the following three tasks on Facebook without using any aids.

Interpretation Task: The first task involved interpreting three posts on the Facebook account, specifically created for this study (Appendix A.2). After participants viewed each post, we asked open-ended questions to participants, such as “What emotions do you think of in this post?” They were also asked to rate the sentiment of each post on a five-point Likert scale (i.e., Very Negative, Negative, Neutral, Positive, and Very Positive). Following completion of these tasks, participants rated their confidence in their judgments and ease in performing these interpretation tasks by answering the interpretation task questions in Table 3.

Focus Task: Participants were instructed to browse and scroll through their Facebook newsfeed for two minutes. Following that, the experimenter prompted them to recall and describe the posts they just viewed, using a few sentences (e.g., “Can you recall the posts you just browsed”). Participants were then asked to provide feedback on the task by answering Focus task questions in Table 3.

Filter Task: In the third task, participants were asked to browse their Facebook newsfeed and inform the experimenter whenever they found an interesting post. Then, they were asked to refresh the page again and find a post from a friend ²²2Based on our observations, Facebook always prioritizes unseen content at the top. Thus, we asked participants to refresh the newsfeed page before and between tasks.. Following this, participants were asked to provide feedback on the task by answering Filter task questions in Table 3.

After completing the three tasks described above, the participants were asked to identify their top five most frequently used menus in the left panel of the Facebook web browser. The total duration of stage 2 ranged from 20 to 30 minutes for each participant.

During stage 3, participants first repeated the same Interpretation Task, Focus Task, and Filter Task as in stage 2 using the aids: Interpretation Mode, Focus Mode, and Filter Mode, respectively. Before starting each task, the experimenter introduced the aid that would be used for the task and demonstrated how to use it step by step. Then the user had a trial session using the aid until they felt comfortable and familiar with the tool. After the participant finished the trial session, the experimenter instructed the participant to proceed with the tasks. Additionally, participants completed a Writing Task where the experimenter asked the participant to write three hypothetical posts on provided topics (Appendix A.1). Additionally, they were asked to use the Writing Aid to improve their writing. After writing each post, the participant was asked a series of questions about their experience in the writing task (see Table 5). Lastly, participants worked on the Facebook Customization Task that asked them to customize the layout of their Facebook main page using the Facebook Customization aid.

After each task, participants evaluated each aid with the System Usability Scale (SUS) (Brooke et al., 1996) and answered open-ended questions (e.g., “How was your experience with the tool?”) to reflect on their impressions on each aid. After completing all five tasks in stage 3, the participants were instructed to rank the five aids in order of most helpful to least helpful. The total duration of stage 3 ranged from 45 to 60 minutes.

Stage 4 consisted of participants answering demographic questions (e.g., gender, age, race, education, and employment status) and questions related to their TBI (e.g., time since injury, cause of injury, and challenges since injury). This stage ended with surveying participants about their usual Facebook usage (e.g., usage amount, reasons for usage, changes to social media usage after their TBI). The study session for each participant lasted between 90 minutes to two hours.

Participants were asked to rank each aid from one to five and evaluate each aid with the System Usability Scale (SUS). To accommodate the cognitive challenges associated with participants’ TBI, we employed a simplified three-point Likert scale (Disagree, Neutral, and Agree) for each statement in the SUS.

We developed questionnaires with a three-point Likert scale (1–3; 1 = Disagree, 2 = Neutral, 3 = Agree) for each aid (Table 3). For the writing task, we utilized a seven-item questionnaire and derived two scales that measured the perceived quality of the written post (items 1, 7; Cronbach’s $\alpha=0.62$) and how well the message is received by other people (items 3, 5; Cronbach’s $\alpha=0.61$). For the Interpretation task, we used a three-item questionnaire to evaluate the confidence level, the clarity of the post, and the ease of the task. For the Focus task and Filter task, we developed a five-item questionnaire and derived two scales that measured the perceived ease of the task (items 1, 2, 5; Cronbach’s $\alpha=0.86$) and success of the task (items 3, 4; Cronbach’s $\alpha=0.94$). We hypothesized that the user would perceive the writing task to have higher quality using the Writing aid than without the aid; the user would have a higher level of confidence in the Interpretation task using the Interpretation aid than without the aid; the user would perceive the task to be easier and more successful using the Focus Mode and Filter Mode than without the aids.

We also collected and analyzed task performance data for the Focus Mode and Filter Mode aids, and we evaluated their effectiveness pre- and post-use of the aids. Specifically, we collected the number of posts recalled by the participant after viewing the newsfeed for two minutes before and after applying Focus Mode to the newsfeed. We hypothesized that the user would recall more posts using the Focus Mode than without the aid. We also counted the time the user found an interesting post and the time to locate a post from their friend before and after they applied the Filter Mode to customize the feed. We hypothesized that the user would spend less time locating the posts using the Filter Mode than without the aid.

W3C’s Cognitive Accessibility Guidelines contain eight objectives and each objective has a list of design requirements necessary to meet the objective. We customized the W3C questionnaires for each aid and removed irrelevant items. For example, “Objective 8: Support adaptation and personalization.” is not applied to the Writing Aid and the Interpretation Aid because these two aids were designed to provide communication support rather than improve the personalization of social media use. Therefore, Objective 8 was removed when participants evaluated these two aids.

In analyzing the expert questionnaires for the guidelines for Facebook in general, as well as for the five aids, we calculated the percentage of design patterns met for each objective. For example, to determine if the “Focus mode” met the objective “Help users focus,” we scored it as 50% when one participant answered “Yes” to two design patterns out of four. We also transcribed and open-coded the comments that expert participants provided while using each of our five aids.

We recorded the full study sessions with users with TBI. Audio files were first transcribed with an automatic transcription tool⁶⁶6https://otter.ai, and then one researcher from the team verified the transcriptions and corrected errors. This researcher further segmented the transcriptions according to the study procedures, differentiating between responses related to questionnaire items and answers given during the experimenter’s interview questions.

We analyzed the transcriptions using thematic analysis (Braun and Clarke, 2022, 2006). Two coders first independently open-coded three data samples (more than 10% of the data) at the sentence level and then merged their codes to develop the initial codebook. Any disagreements during this phase were resolved through discussion. The same coders continued to process the remaining data individually, updating the codebook as new codes emerged. The final codebook included categories detailing participants’ social media usage patterns, the challenges they faced using social media due to TBI, and their feedback on each aid. Given we have specific goals to understand the usefulness and usability for each aid in the toolkit, we follow the deductive approach (Braun and Clarke, 2022, 2006) to generate themes focusing on particular aspects, i.e., the perceived usefulness, usability challenges, and suggested new functions for each aid.

We had both online and in-person study sessions with TBI expert participants. The online study session was hosted through teleconferencing technology (Zoom), and we recorded the full study sessions. In one in-person study session, the experimenter experienced technical issues with audio recorders, and the experimenter took field notes for the participant’s response. The interview data were transcribed using an automatic transcription tool, and one researcher verified the accuracy of the transcripts. We followed the same approach of deductive thematic analysis as the analysis for interviews with TBI users as described above.

During the study, participants used the Writing Aid for spelling and grammar checks for the writing tasks. All participants found the aid useful, and four out of eight participants agreed that they would like to use the aid frequently (P1, 5, 7, 8). The aid was reported to support message construction and spell checks (P6, P8), facilitate sentiment analysis (P5, P7), and enhance privacy settings (P7). One participant noted it provided “a different perspective on how others might view it” (P6), thus indicating that the design objectives of the Writing Aid could be attained to some degree. Additionally, two participants emphasized the need for self-presentation in using social media (P1, P5) and thought that the aid could “help you change your story” (P5) and confirm that “Is this character (myself) polished enough to be on someone’s network” (P1).

Meanwhile, TBI user participants identified two major usability challenges: inaccurate grammar and sentiment analysis of the content (P3, P4, P7) and the inconvenience of repeating writing checks after modifying posts (P2, P7, P8). For example, P4 experienced inaccurate grammar suggestions that incorrectly flagged slang she wrote in her posts, such as “gonna” and “Ima.” P7 suggested a potential design improvement, proposing a single button to restart the sequence of checks after making edits. He stated, “I would add a button at the end to go back to the start…in case you want to redo something and maybe recheck a specific section before you post it.” (P7). Additionally, P2 desired more learning support, stating that, “it was frustrating at first” (P2), and wanted the aid to provide more “specific” instructions for improving the content.

Four out of five expert participants (EP1–3, EP5) provided positive feedback on the overall functionality of the aid, highlighting its capability to reduce communication errors and enhance writing clarity (EP1, EP2) and provide useful perspectives to tweak the written post (EP3, EP5). Further, EP5 emphasized the need to use the writing aid to support social communication; she commented: “Some of the big issues is kind of that impulsivity and not being able to kind of check and correct their own errors and stuff when posting…First of all, does it make sense, what it’s saying, but also kind of how that might come across to other people, as well.” (EP5).

Experts (EP1–5) also pointed out usability challenges of the aid based on the W3C guideline and suggested improvements, focusing on TBI users’ cognitive and sensory needs. These suggested improvements included increasing font sizes (EP1); using simple and clear sentences for instructions (EP1–3); defining keywords used in the aid (e.g., “toxicity”) (EP3); and providing more detailed suggestions or automatically implement the suggestions after the grammar checks (EP1, 3). EP2 suggested rewording the aid’s instruction in a “more direct or simple” way because of the diverse literacy levels of the patients she had worked with. EP2 shared, “A lot of my patients that I work with who use social media a lot of the time, they come from backgrounds. Some of them don’t have a lot of education.” (EP2).

Furthermore, experts (EP2, EP5) pointed out that oversharing personal information often happens for TBI users and suggested that the Writing Aid could “add in that extra level of security” (EP5) to ensure the user’s safety, such as sending out alerts when oversharing activity was detected (EP5). EP5 also suggested using verbal input to overcome the challenge of word-finding.

The Interpretation Aid was found to support comprehension of posts and images (P2, P6, P8), particularly for longer messages (P2), and two participants agreed that they would like to use it frequently (P1, P8). For example, P8 reported that “Right after the accident, I had a really hard time understanding what people meant with their words…This [aid] would have been really comforting to me.” P6 appreciated that the aid provided a different perspective of the post, stating, “it’s nice to see how other people, same as earlier, like, could interpret what you’re saying.”

The major usability challenge reported by participants for the Interpretation Aid was disagreement with the results of sentiment analysis for the content (P2, P3, P7, P8). For example, P2 disagreed with the classification of the emotion type “joy” for the post in the first interpretation task. P8 thought a post to be “positive” while the aid predicted it to be “negative.” Additionally, P2 noted confusion about the image information and thought the description of its emotion should be shortened.

Notably, two participants (P1, 5) suggested that the aid provide more detail and explain the reason behind the interpretation results. For example, P1 commented: “I want them to emphasize more, like, you know, why are they negative, why are they positive?” (P1).

Three expert participants (EP1, EP4, EP5) reacted positively to the Interpretation Aid. They appreciated that the aid could support social-emotional communication (EP1, EP4) and summarize the posts for individuals with TBI (EP5). For example, EP1 described how the aid can help users react to other people’s social media posts, stating, “The aid would help them to know what the post is sort of about and whether to ‘like’ it or not. Maybe even determine whether it’s a like or a ‘love’. Like if they were to learn that the greener it is the more chance the person is looking for a ‘love’, versus a yellow one the person is looking for a ’like”’ (EP1). Similar to the TBI users, experts also identified usability challenges for the aid, such as the need to improve the accuracy of interpretation results (EP1, EP2, EP5) and clarify the analyzed content (EP1, EP4).

Three participants (P1, P3, P8) reacted positively towards the Focus Mode aid, and two participants reported that they would like to use it frequently (P1, P8). For example, P8 described the aid “amazingly helpful” (P8) and shared, “I read more captions than I would have if I was scrolling the newsfeed. They’re bigger and easier to see.”

On the other hand, five participants (P2, P4–7) held negative views towards the aid, finding it “unnecessary” (P2) and “cumbersome” (P5). Two major usability challenges reported by multiple participants were the post navigation for viewing the full content (P3–5, P8) and the unresponsiveness of the buttons (P2, P6–8). P5 described how the buttons for post navigation were inaccessible for users with motor impairments, stating “If you have, you know, in nimble fingers, and you can’t really do that, or if you’re old and don’t know how to do it” (P5). In addition to the above challenges, two participants (P5, P8) requested to increase the size of the post creator’s profile photo and name texts.

All TBI expert participants provided positive feedback for the Focus Mode, highlighting its ability to enhance the usability of social media by streamlining the individual’s feed (EP1–5) and the simplicity of use (EP1). For example, EP4 commented: “Wow. Okay, this is the best one. Wow, I can really focus on the posts.” (EP4). Similarly, EP5 mentioned that “You can literally just focus on scrolling through your newsfeed. Yeah. This is cool.” (EP5). The major usability challenge reported by TBI experts participants (EP3, EP4) is the interactivity of the blurred area. EP4 described how she accidentally clicked in the blurred area and went to an unexpected page: “The blur, I accidentally clicked and they went to a sponsor page.” (EP4).

Four participants (P1, P4, P6, P8) held positive views towards the aid, two participants (P2) held neutral views, and three (P3, P5, P7) reacted negatively to the aid. Two participants (P1, P8) agreed that they would like to use the aid frequently. In particular, P8 found it helpful that advertisements and posts were distinguished after applying the aid, stating that “I think the advertisements being filtered away is something that makes it clear that I’m looking at people’s posts.” (P8). However, three participants experienced usability challenges that the filtering was not accurate and did not sync up as experienced by three participants (P3, P4, P7). As P7 commented: “I would say it’s not very accurate. It’s not personal to me.” In addition, the filtering caused challenges for tracking viewed posts because of the posts hiding and reordering, as what P8 experienced refreshing the page: “One that I was just on is now gone” (P8).

Participants (P2, P5, P8) suggested that the Filter Mode could have provided more options and combined multiple filters to achieve more customized results. For example, P2 compared the filter with Excel, and desired to have a wider range of filters such as “Quotes” and “Meme.”

Filter Mode received positive feedback for its sorting feature and feed personalization (EP3–5) as well as its ease of use (EP3). As EP3 noted, “It’s just helpful to be able to narrow down, like, the type of content that you want, what’s most meaningful to them” (EP3). However, some expert participants pointed out several usability challenges and provided suggestions for improvement, such as improving the accuracy of filtering results (EP2), making filter labels clearer (EP1, EP2), and increasing font size (EP3).

Seven participants (P1–3, P5–8) held positive views toward the aid, and one participant (P4) was neutral. Three participants (P3, P7, P8) appreciated the ability to control the newsfeed with the tool and mentioned that the aid could make them “more focused” (P5) and hide information that they “don’t really need” (P3), such as advertisements. On the other hand, one participant (P4) found this customization was not necessary because their goal of using social media was to “check in and look around” (P4), thus simplifying the information source would have a side effect for them to explore different content. P7 also wished that there were more options for the left-side filtering for him to “personalize it a little bit further” (P7). P7 also desired more guidance on how to use the aid.

Similar to the benefits mentioned by users with TBI, experts (EP1, EP3–5) noted that the tool simplified the newsfeed page and could help avoid distractions for users with TBI. EP1 noted, “For me as a user, I love this. I never looked at the stuff [extra fields on Newsfeed] before but I love it not being there. Look how much nicer that is to look at. I can ignore it with my own mind but I love not seeing it.” Similarly, EP3 commented, “They have control over what they’re seeing. This is very helpful” (EP3). However, experts (EP1, 2) mentioned the current design could be further improved by clarifying the wording of the options (EP2) and using icons to support the user’s understanding of the control (EP1).