Learning to Ignore: A Case Study of Organization-Wide Bulk Email Effectiveness

Ruoyan Kong [email protected] University of Minnesota - Twin CitiesUSA , Haiyi Zhu [email protected] Carnegie Mellon UniversityUSA and Joseph A. Konstan [email protected] University of Minnesota - Twin CitiesUSA

Abstract.

Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email system is a multi-stakeholder problem including recipients, communicators, and the organization itself.

We studied the effectiveness, practice, and assessments of the organizational bulk email system of a large university from multi-stakeholders’ perspectives. We conducted a qualitative study with the university’s communicators, recipients, and managers. We delved into the organizational bulk email’s distributing mechanisms of the communicators, the reading behaviors of recipients, and the perspectives on emails’ values of communicators, managers, and recipients.

We found that the organizational bulk email system as a whole was strained, and communicators are caught in the middle of this multi-stakeholder problem. First, though the communicators had an interest in preserving the effectiveness of channels in reaching employees, they had high-level clients whose interests might outweigh judgment about whether a message deserves widespread circulation. Second, though communicators thought they were sending important information, recipients viewed most of the organizational bulk emails as not relevant to them. Third, this disagreement was amplified by the success metric used by communicators. They viewed their bulk emails as successful if they had a high open rate. But recipients often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general understood the challenge, they had a limited set of targeting and feedback tools to support their task.

organizational communication, email

^†^†copyright: acmcopyright^†^†journal: PACMHCI^†^†journalyear: 2021^†^†journalvolume: 5^†^†journalnumber: CSCW1^†^†article: 80^†^†publicationmonth: 4^†^†price: 15.00^†^†doi: 10.1145/3449154^†^†conference: PACM HCI; 2021; ^†^†ccs: Human-centered computing Empirical studies in collaborative and social computing

1. Introduction

^†^†This is a pre-print version of a paper accepted to CSCW 2021 — the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing.

Bulk email is email that is sent to a large group of recipients (Dawkins, 2018). Bulk email has become an important part of organizational communication — the collective and interactive process of generating and interpreting information within organizations to achieve their purposes (Stohl, 1995). Organizations (entities comprising multiple people with a particular purpose (Handy, 2007)) often use bulk emails to deliver information to their employees (Dabbish and Kraut, 2006). Example organizational bulk emails include announcements of new staff, summaries of meetings, health and safety issues, and event invitations to relevant groups within organizations, etc. Despite its important role in organizational communication, organizational bulk emails can bring email overload to the organizations’ employees. In large organizations, the problem of overwhelming communication is widespread. Recipients are receiving emails irrelevant to them while missing important ones (Dawkins, 2018; Jackson et al., 2003).

Previous research on organizational emails mainly studied general emails from recipients’ perspectives, such as which emails employees would delay reading (Sarrafzadeh et al., 2019), the probability of an email being retained/deleted (Dabbish et al., 2003), recipients’ productivity (Mark et al., 2016). However, bulk email communications within organizations is not only a problem involving email recipients but is also an example of a multi-stakeholder problem (Andriof et al., 2002). The stakeholders in organizations include:

•

Communicators: the staff who are in charge of designing and distributing organizational bulk emails.
•

Communicators’ clients: the original producer of organizational bulk emails, who are the communicators’ internal clients.
•

Recipients: the employees who receive organizational bulk emails from the communicators.

Not only are there different goals for communicators, their clients, and recipients, but also there is a key fourth stakeholder — the organization itself — that has its priorities not always recognized by communicators and their clients, or recipients. Communicators and their clients naturally focus on their own needs — getting the word out, establishing evidence of notice or compliance, or preserving a record of communication. But recipients, faced with more communication than they can handle, have to scan, filter, or simply ignore organizational bulk emails. In turn, organizational goals around compliance, informed employees, and employee productivity may suffer.

Thus we would like to understand multi-stakeholders’ perspectives within the organizational bulk email system. We conducted a case study, a predominant approach in organizational studies (Swanborn, 2010; Buchanan, 2012) and organizational communication studies (Sarrafzadeh et al., 2019; Dabbish et al., 2003; Mark et al., 2016; Schuler, 1979; Stohl and Redding, 1987). A case study could help us 1) learn domain experts’ (communicators) information management techniques and how the bulk email system is used within the organization context; and 2) understand the links between different stakeholders within the same organization (Lazar et al., 2017). We selected a representative organization with hierarchical structures, and centralized and decentralized communication offices. We hope our case study will 1) inform the design of the organizational bulk email system (Yin, 2003); 2) provide interview protocols for future studies on the system; and 3) be generalizable to similar organizations.

Specifically, we studied communicators’ practice of designing and distributing organizational bulk emails, and communicators/recipients/managers’ experience and assessments. We combined self-reported data, logged inbox data, and inbox-review data in an artifact walkthrough (Raven and Flanders, 1996).

We found that the organizational bulk email system as a whole was strained, and showed its strain in the intermediary role of the communicator. First, communicators had high-level clients who wanted their messages sent broadly. Second, communicators did not have visibility into the effectiveness of bulk emails in eliciting behaviors or transmitting information; they only measured bulk emails by “open rates”. Communicators felt they were sending important information and got high “open rates”. Third, recipients viewed most bulk emails as not relevant to them; they often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general had an interest in preserving the effectiveness of channels in reaching employees, they had a limited set of designing, targeting, and feedback tools to support their task.

The rest of this paper includes related work (2), gaps and research questions (3), study site and its bulk email system’s structure (4), artifact walkthrough with stakeholders (5, 6), and discussion (7).

2. Related Work

2.1. Organizational Communications

Communication within organizations has been studied for more than a century (Van Vleck, 2012). Communication has been called “the life blood of organization” (Goldhaber, 1990), “the glue that binds it all together” (Katz and Kahn, 2008). Myers and Myers (Myers and Myers, 1982) defined organizational communications as “the central binding force that permits coordination among people and thus allows for organized behavior”.

Organizational communication is a multi-stakeholder problem. Organizational communication’s effectiveness is a problem not only about information recipients, but also information providers (like communicators and their clients). Randall (Schuler, 1979) did a case study in a large manufacturing firm, finding that if information producers failed to provide clear information, it was unlikely that the information receivers would continue to seek out information. Stohl and Redding (Stohl and Redding, 1987) studied a Chinese steel construction corporation, finding that workers’ cognitive failures were significantly related to the managers’ communication styles.

Organizational communication’s effectiveness is also related to the organization itself. Greenbaum (Greenbaum, 1974) defined the purpose of organizational communications as the achievement of organizational goals, accomplished through the appropriate employment of communication networks, policies, and activities. Greenbaum argued that communication effectiveness had to be measured by looking at the overall communication system besides the activities of recipients.

Ineffective organizational communication is often linked to the behaviors of multiple stakeholders, not just recipients. For example, high-level senders did not provide enough information about what was going on at the organization, gatekeepers (e.g. communicators) lost/delayed delivery of some vital information in the distribution process, or recipients’ feedback was neglected (Downs and Adrian, 2012; Odine, 2015).

However, as we discussed below, as a part of the organizational communication studies, the previous studies on organizational emails focused mostly on recipients and less on information providers and the organization itself.

2.2. Organizational Email

Organizational internal emails (we refer to them as organization emails) are emails whose senders and recipients are within the same organization. Organizational emails create what Sproull and Kiesler called a “networked organization” (Sproull and Kiesler, 1991), in which people can be available even when they are physically absent. At the same time, they bring email overload — email users’ perceptions that their own use of email has gotten out of control because they receive more emails than they can handle effectively (Dabbish and Kraut, 2006; Fisher et al., 2006; Grevet et al., 2014; Whittaker and Sidner, 1996).

Most studies of organizational emails focused on recipients and recipient-perceived effectiveness. First, previous studies around estimating organizational email effectiveness used data which only contained one kind of stakeholder — recipient’s log data/inbox data/behavior data (Sarrafzadeh et al., 2019; Paczkowski and Kuruzovich, 2016), like the Avocado dataset (Yang et al., 2017), Enron Corpora (Bekkerman, 2004; Bermejo et al., 2011; Klimt and Yang, 2004; Chakrabarty and Roy, 2014), and Outlook Dataset (Alrashed et al., 2019).

Second, previous studies around email overload focused on how email burdened its recipients, instead of the organization (Bellotti et al., 2005; Mishra et al., 2014; Malone, 1987). Dabbish and Kraut (Dabbish and Kraut, 2006) conducted a study that quantified the email burden on white-collar workers. Merten and Gloor (Merten and Gloor, 2010) found that employee job satisfaction went down as internal email volume increased in a case study in a 50 people company. Huang and Lin (Huang et al., 2009) surveyed 3 universities and found that knowledge workers were “ruled by email”.

Third, previous studies mostly assumed that information providers (communicators and their clients) should change their email actions according to recipients’ needs to reduce email overload. Jackson et al. (Jackson et al., 2003) studied 16 employees at the Danwood Group and proposed that email frequency should be controlled. Lu et al. (Lu et al., 2012) described EPIC, an email prioritization tool which combined global priority with individual priorities. Reeves et al. (Reeves et al., 2008) experimented with attaching virtual currency to emails to signal importance in a large company.

2.3. Bulk Email

Bulk email is email sent to a large group of subscribers for content delivery (Dawkins, 2018). Bulk email is a primary communication infrastructure within organizations, businesses and industries.

Most studies of bulk email are about external bulk emails — the emails sent to recipients outside of organizations, like marketing emails or advertisements. These include: 1) supporting information providers design external bulk emails: Carter et al. (Carter et al., 2011) found that bulk email was more effective when it was entertaining. Trespalacios and Perkins (Trespalacios and Perkins, 2016) examined the effects of bulk email designs and found that neither the degree of personalization nor the length of the invitation email impacted survey response; 2) supporting recipients filter external bulk emails: Gray and Haahr presented an architecture for personalized, collaborative spam filtering by content-based approaches (Gray and Haahr, 2004), and reached over $90\%$ accuracy in a 2-week case study. Al-Jarrah et al. proposed header-based approaches (Al-Jarrah et al., 2012), reaching a ROC Area of 98.5% in CEAS2008 dataset.

3. Gaps and Research Questions

We identified 2 major gaps from the previous work:

•

The studies on organizational emails focused on recipients: though studies on organizational communication pointed out that when measuring organizational communication’s effectiveness we should consider not only recipients, but also information providers (communicators and their clients) and the organization itself, previous studies on organizational emails did not learn whether recipients and information providers within an organization had different preferences about emails, and how these mismatches affected the organization.
•

The studies on organizational bulk emails focused on external bulk emails: few works on bulk emails considered the case when the bulk email sender and recipients were in the same organization, and when the ultimate goal was maximizing this organization’s interests instead of the information providers’ or the recipients’ interests. Within organizational context, whether a recipient (employee) should get an organizational bulk email may not be determined by whether the recipient likes the email. Sometimes employees may have responsibility for knowing about this email though they might not have interests in it. We see a need to understand bulk emails from an organizational context.

Therefore we identified a need to undertake a systematic investigation of bulk email communications from a whole organization’s view. Notice that we referred to organizational bulk email as bulk email and organizational bulk email system as bulk email system in the rest of this paper. We posed the following research questions:

Q1: What are communicators’ current practices for designing and distributing bulk emails?

Q2: What are the experience with and assessments of bulk emails of different stakeholders?

We conducted an in-depth case study of a representative organization. We interviewed both communicators and recipients within an organization using an artifact walkthrough approach. We delved into email cases and got their assessments of email values. We discussed communicators, recipients, communicators’ clients, and recipients’ managers in this paper.

4. Study Site and Its Bulk Email System’s Structure

4.1. Organization Environment

The organization we studied in this paper is the University of Minnesota — a large university with more than 25,000 employees and several campuses. It is structured much like many large universities, with “a mixture of collegiate, managerial, senatorial, centralized and decentralized governing styles” (Dill and Sporn, 1995):

(1)

There were central units led by university leaders (Office of President, University Relations, Information Technology, etc). They were responsible for the core business functions.
(2)

There were decentralized units, like colleges and campuses led by academic leaders (deans and chancellors) and organized into departments. Most of these offices performed a piece of centralized functionality locally, with own human resource, information technology, communicators. These offices also worked with the centralized offices for other functions.
(3)

Each of the units had a communication office, with communicators (communication director/staff), organized according to the location of the units. There were central communication offices in central units and decentralized communication offices in decentralized units.

The decentralized and centralized structures might cause the segmentation of values and responsibilities of units, as Kuh and Whitt found (Kuh and Whitt, 2000): “Universities are not monolithic entities. Subgroups have their own artifacts and values, which may differ from the host”. For example, faculty felt a responsibility not only to their university but also to their discipline/program (Masland, 1985). These conflicting responsibilities led to the decreased potential for coordination (Shriberg et al., 2002).

4.2. Bulk Email System’s Structure

We met with a group of 9 communicators within this university. In the meeting, we found agreement that the current bulk email system might not be what the university wants it to be, since the communicators felt that bulk emails were ignored too often. They encouraged us to move forward. Fig 1 is the structure of the bulk email system of the university:

(1)

They used a Gmail system based on G Suite for Education (Celik and Ölçer, 2018) with a Salesforce platform, and a domain-based email authentication system to identify legitimate bulk emails.
(2)

The central communication offices conducted university-wide email communications. The decentralized communication offices worked with the central communication offices and conducted local bulk email communications.
(3)

Only the trained communicators in each unit can send and manage the targeting of bulk emails. Senior communicators were also responsible for drafting emails for university leaders.
(4)

Communication offices submitted bulk email content to the newsletter editors in offices like University Relations. Employees would be subscribed to these newsletters by default.
(5)

The access to sending bulk emails to large campus-wide or system-wide groups through existing mailing lists, or pulling out lists from PeopleSoft’s database (a Human Resource System) was controlled by the central information technology services.

Refer to caption — Figure 1. Structure of the Bulk Email System of the University; “comm” stands for communicators.

Each bulk email contained one or more messages. We defined a message as a single cohesive amount of information; for the bulk email with several messages, a message is usually a subsection of it. Where participants used “message” differently from our definitions, we clarified in the quotes with bracketed notes “I scan for urgent messages [emails].” (R5)

Among the organizational emails of this university (the emails sent from the university’s email address @umn.edu), according to the channels and the number of messages of the emails, we identified three categories:

•

Non-Bulk Emails: The organizational emails not sent by bulk email channels, like individual emails to other employees or to small groups without a mailing list.
•

Mass Emails: The bulk emails with a single message. For example, an email sent to all employees to announce a free financial workshop from the human resources office.
•

Newsletter Emails: A special kind of bulk email that had a collection of messages and was sent to the recipients periodically. For example, a weekly news digest sent to all employees, featuring administrative, legislative, budgetary, event, and other information.

5. Artifact Walkthroughs with Stakeholders — Methods

What are the practices, experiences, and assessments of different stakeholders of the bulk email system within this university? We conducted artifact walkthroughs with 17 stakeholders — 6 communicators, 9 recipients, and 2 managers within the university. Artifact walkthrough is used to walk the interviewer through the process of completing a task. It helps the interviewer infer possible design intent by asking probing questions about why one thing was done rather than another (Beyer and Holtzblatt, 1999). In our study, for example, recipients used certain queries in their inbox to retrieve specific types of emails; then answered questions about their practice (trashed/opened/scanned/read in detail) and assessments (sometimes they were asked to reread the email) on each email case retrieved.

5.1. Recruitment and Participants

We interviewed 17 participants in the same university by the following process:

(1)

We worked with the same group of communicators in section 4.2 to generate a list of potential participants by a stratified sampling approach, which included 8 communicators and 20 recipients, based on length of work experience, nature of the office, and job responsibility.
(2)

We started inviting them for a 30 to 60 minutes interview in our lab or their own offices.
(3)

We stopped inviting after we interviewed 6 communicators and 9 recipients because we had stopped finding new observations from adding more communicators/recipients.
(4)

To avoid recipients’ answers being influenced by their awareness of manager involvement, we waited till after the interview to ask the recipient if they would allow us to invite their direct manager to discuss the non-personal emails we collected from the recipient. Two recipients agreed and we invited their managers for an extra 30-minute interview.

This study was reviewed and approved by the IRB of the University of Minnesota. We had 17 participants ( Table 2): 1) there were 6 communicators — 2 communication directors who had over 10 years of experience and 4 communication staff, from 5 different central offices and 1 college office; their responsibilities included editing newsletter, writing drafts for university leaders, leading technical support, etc; 2) there were 9 recipients — 5 staff (1 from a central office, 2 from college offices, 2 from departmental offices) and 4 faculty; 5 recipients had been at the university for over 10 years; 3) R5’s manager (a college office’s director) and R6’s manager (a program director) were also invited.

5.2. Interview Protocol

A. Communicators. The artifact-walkthroughs with communicators included 3 parts:

•

General practice questions on their duties, goals, and mechanisms of distributing bulk emails.
•

Email case questions on how an email was designed, sent and measured. We asked communicators to select important/unimportant cases from their points of view.
•

General assessment questions, see Table 7 in the appendix.

B. Recipients. The artifact-walkthroughs with recipients included 3 steps:

•

Collect inbox logged data: They were asked to copy and paste 10 email queries to retrieve subsets of organizational emails, non-mass emails, mass emails, and newsletters they received in the past week. We recorded the number of emails left unread/opened of each category (see the actions’ definitions in Table 1 — logged left unread/opened).

Notice that we retrieved emails in all categories, including those archived and in trash/spam, by command “in:anywhere” (see Table 8), except those permanently deleted from the trash. By default, only emails that have been in the trash for more than 30 days would be automatically deleted by Gmail. We did not observe the case that the trash was manually cleaned by the recipients. We assumed that we could retrieve all emails received in the past 7 days.

These queries were pretested in one of the author’s inbox to ensure they retrieve the right subset of emails. We recorded the size of each subset, and how many were opened.
•

Collect self-report data: For the mass/newsletters we retrieved above, we investigated up to 12 emails for each category. For each email, we recorded the recipient’s actions on: 1) whether the email was left unread/opened/scanned/read in detail/trashed (see the actions’ definitions in Table 1 — self-report left unread/opened/ scanned/read in detail/trashed); 2) whether asked for actions/took actions; 3) importance/urgency/relevance on scale 1 to 5; 4) reasons for the answers/actions, and 5) whether they would change their mind about answers/actions if they reread it now (see appendix B). We investigated 163 cases in total.
•

General questions: We asked them some general questions like how frequently they checked their email accounts, how many emails accounts they had, did they feel that it was too many to read all of them, and how often did they not read an email.

C. Managers. The answers from the employee-manager pair were confidential from each other. We showed the managers the emails we collected from their employees, and asked them to give their preferred actions that their employees ought to have done with those emails — trashed/left unread/opened/scanned/read in detail, and estimations on importance/urgency from 1 to 5. We investigated 31 emails in total with the two managers. Notice that the “manager” here means the direct supervisor of the recipients, not a reflection of a particular position in the organization. And the recipients listed as ”director” in Table 2 are heads of their offices or programs but are not listed as ”manager” because we interviewed them about their own received messages, not about the messages of one of their employees.

To summarize, for participants invited as communicators/recipients/managers, we discussed the bulk emails they sent/received/their employees received correspondingly.

The Actions of Recipients

Definition

Logged Left Unread

The email was labeled unread by Gmail.

Self-Report Left Unread

The recipient claimed that they did not open the email.

Logged Opened

The email was labeled read by Gmail.

Self-Report Opened

The recipient claimed that they opened the email.

Self-Report Scanned

The recipient claimed that they read the email quickly to get its general idea only.

Self-Report Read in Detail

The recipient claimed that they read the whole email thoroughly and carefully.

Self-Report Trashed

The recipient claimed that they archived the email or moved it to the trash folder;

untrashed email was left in the inbox or moved to a non-trash folder.

Self-Report Opened and Trashed

The recipient claimed that they opened the email first then trashed it.

Table 1. The definitions of recipients’ actions with emails. Logged data was collected by using queries to retrieve subsets. Self-report data was collected by asking recipients their actions directly.

Stakeholder

Type

Years at

University

Position

Level of

Office

Stakeholder

Type

Years at

University

Position

Level of

Office

Communicator

6 - 10

Staff

Central

Recipient

1 - 5

Staff

Central

Communicator

11- 20

Director

Central

Recipient

1 - 5

Staff

Departmental

Communicator

1 - 5

Staff

College

Recipient

11 - 20

Director

Departmental

Communicator

11 - 20

Director

Central

Recipient

Staff

College

Communicator

1 - 5

Staff

Central

Recipient

1 - 5

Staff

College

Communicator

6 - 10

Staff

Central

Recipient

Professor

R5’s Manager

6 - 10

Director

College

Recipient

11 - 20

Professor, Director

R6’s Manager

Professor

Recipient

11 - 20

Professor

Recipient

1 - 5

Professor, Director

Table 2. Demographics of participants. C1 — C6, R1 — R5, M1 are staff and the rest are faculty. Central-level offices are in charge of university-wide affairs, like university services, information technology; college-level or departmental offices are located within a college or department, like the dean’s office.

5.3. Data Analysis

To compare the experience and assessments of the effectiveness of the bulk email system from different stakeholders, we took a grounded theory approach (Charmaz, 2014), specifically with the following iterative procedure:

(1)

Identifying the themes from the communicators’ transcripts.
(2)

Searching for relevant text and themes in the recipients’ transcripts.
(3)

Comparing the actions taken by the recipient and preferred by the manager (if applicable).
(4)

Inviting new participants and repeating the steps above until we reached data saturation. We stopped when we had interviewed 17 stakeholders because we found strong repetition in the themes identified.

6. Artifact Walkthroughs with Stakeholders — Results

6.1. Number of Emails Received and Actions Taken

Table 3 reported the statistics of the 9 recipients on emails received, opened, read, and trashed by category. The recipients, on average, received 376 emails in the past week: 153 of them were organizational emails and 30 of them were bulk emails (25 mass emails and 5 newsletters). The number of messages in one of the 55 newsletters investigated in the interview could be as many as 35, with an average of 8.5. Participants (recipients) reported that they, in general, received too many bulk emails. R6 said “Sometimes I felt overwhelmed.”. R4 and R8 said that a large number of bulk emails became a burden to them.

Faculty received 175 organizational emails a week on average (148 non-mass emails, 23 mass emails, 4 newsletters), and for staff, this average number was 136 (103 non-mass emails, 27 mass emails, 6 newsletters). The average logged open rate of mass/newsletters of faculty recipients (¿90%) was higher than their self-report open rate (¡70%). Sometimes they clicked an email’s title and removed its unread tag, but they did not think that they “opened” the email.

Though many bulk emails got opened, few of them were read in detail. In fact, $58\%$ of mass emails were reported being opened by staff and only $28\%$ of them were read in detail; $67\%$ mass emails were reported being opened by faculty and only $13\%$ of them were read in detail.

Many bulk emails were trashed, $52\%$ mass emails and $22\%$ newsletters were reported being trashed by faculty, for staff the 2 numbers were $27\%$ and $15\%$ .

Faculty

Email Type

Overall

Organizational

Non-Bulk

Organizational

Mass

Organizational

Newsletter

no.= 4

# Received Per Week

560

175

148

% Logged Open Rate

55%

96%

97%

92%

94%

% Self-report Open Rate

67%

59%

% Self-report Read in Detail Rate

13%

% Self-report Trash/Archive Rate

52%

22%

Staff

Email Type

Overall

Organizational

Non-Bulk

Organizational

Mass

Organizational

Newsletter

no. = 5

# Received Per Week

229

136

103

% Logged Open Rate

54%

73%

78%

52%

50%

% Self-report Open Rate

58%

% Self-report Read in Detail Rate

28%

30%

% Self-report Trash/Archive Rate

27%

15%

Table 3. Average per-capita email volume by category; % logged open rate is calculated by the inbox logged data; % self-report open/read-in-detail/trash/archive rate is calculated by the self-report data, based on all the bulk emails investigated in the corresponding category, for example, self-report read in detail rate of mass email = # mass email read in detail/# mass email investigated.

6.2. Nature of Bulk Email System as an Organizational System

Bulk email system was part of the university’s administrative system, with communicators trying to achieve the communication goals of this university. They had clients, performance metrics, and tools.

6.2.1. Communicators as a gateway for organizational leaders to reach employees.

Communicators had clients from their own offices or other offices who were not professionals in communication and wanted to promote events, notify of changes of policies, etc. Many clients were the communicators’ leaders:

We work with our leadership whoever sends the message, so for example, we send a few messages for the associate vice president for research, who’s also the head of responsive administration. (C1)

Naturally, clients wanted to send more content and reach more people. A communicator talked about their clients:

We’re working with other departments coming up with certain final messages that may be close to the amount of information that people can digest. But some of them think that people have a more of an appetite for reading. (C1)

Thus communicators usually needed to narrow mailing lists and shorten the length of emails. One mechanism they used was letting clients fill in templates:

The U Relations developed a partnerships with emergency management. When there is a safety alert, there are certain blanks in the template they fill in. That’s why you can see the safety messages [emails] are often similar. Because we give them the authority to send them, so we want to limit the scope of the content they can choose to put in there. (C4)

Sometimes communicators could not modify the emails. Some bulk emails were required by law:

We have things called safety advisories. Maybe somebody is grabbed from the Ford Hall last night and we just found out about it. That person is gone but we are required by federal law to tell you about that. (C4)

6.2.2. Communicators are anchored to the “open rate” metric

Optimizing a system depends on using appropriate metrics. The metrics used by communicators to measure bulk email’s effectiveness did not match recipients’ assessments. We asked communicators about the metrics they used in the bulk email system, and measured recipients’ assessments for each bulk email from various metrics like whether trashed/opened/scanned/read in detail, and ratings on urgency/importance/relevance. For communicators, the priority order of the metrics used to report a bulk email’s performance was: open rate ¿ click rate ¿ replies and others. They only needed to report a good open rate to their clients. $50\%$ was a good open rate to communicators:

For all campus messages [emails], open rates are usually at least 50%, which is good for us, so pretty high. We look at click rate but not closely. Most times we look at open rates. (C5)

However, “open rate” might be a flawed metric for measuring recipients’ feedback, because:

(1)

“Being opened” did not mean “being read”. As we discussed above, the logged open rate, which was used by communicators, might be higher than the self-report open rate — sometimes recipients clicked an email’s title and removed its unread tag, but never read any sentences in the email. Some faculty recipients, like R7, opened 17 of the 22 bulk emails we investigated, but only read 2 of them in detail, see Table 4.

Recipients were not opening bulk emails because they considered them relevant but because they were unsure of their relevance and wanted to verify:

I read the first line then deleted it because I’ve seen similar information in other news. (R7)
(2)

Most of “opened” bulk emails were only scanned and got low ratings in importance/urgency/ relevance. As shown in Figure 2 (3), 43/54/36 bulk emails got 1 or 2 on importance/urgency/ relevance ratings over 66 scanned emails.
(3)

Some bulk emails were opened and then trashed. As shown in Table 4, R1, R3, R5, R6, R7, R9 all trashed some bulk emails after they opened them.

Trashing was more like a personal preference for managing inboxes. For example, R1 trashed almost every bulk email after they opened and read it (see Table 4), “If it doesn’t require actions, or not related to my work, I just trash it.”. In contrast, R2 did not trash any email.

Trashing a bulk email did not necessarily mean that it was useless. For example, R5 trashed a relevant bulk email after reading it: “I skimmed it because I was going to see whether we have all clear on the water issue … I delete it because the water is good, no action needed.”, and trashed a relevant bulk email without opening it: “I didn’t need to open it because all the useful information are in the title — come and get ¡person name¿ … I trashed it.”.

Though “open rate” was a flawed metric, communicators had few tools in the bulk email system to get performance besides open/click rate: “I don’t know, how long people spend, whether they share with other people.” (C1)

Implication 1: Provide End-to-End Metrics. Focusing on “open rate” distracted communicators from real measurements of both channel and bulk email’s effectiveness. Communicators usually were not involved in the transactional process of the bulk emails. They did not know whether the communication really worked out — the recipients read the emails and took corresponding actions to complete the organization’s goals. Thus showing metrics on the transactional process of bulk emails to communicators could be helpful, for example, showing how many people missed the due dates of events that were sent to them in emails. When connecting heterogeneous systems is difficult, we could build systems similar to some human resources analytic systems (Rasmussen and Ulrich, 2015), like collecting assessments from clients/leaders for each bulk email.

#	type	# investigated	# opened	# read in detail	# trashed	# opened and trashed
R1	staff	20	5	3	14	5
R2	staff	23	19	12	0	0
R3	staff	12	4	1	1	1
R4	staff	8	5	2	1	0
R5	staff	20	15	6	3	2
R6	faculty	17	12	1	7	4
R7	faculty	22	17	2	8	5
R8	faculty	24	10	2	3	0
R9	faculty	17	12	4	14	9

Table 4. The number of bulk emails investigated/opened/read in detail/trashed/opened and trashed. The data was self-reported by participants on the specific bulk emails we investigated during the interviews.

6.2.3. Diverse types of communication with different content and delivery channels, and different matchings between content and channels.

A. Bulk email’s contents. Communicators mentioned three types of bulk email’s content:

•

Transactional content: which aimed to facilitate the work of the university, and asked recipients to take action. Communicator C4 talked about the goals of transactional bulk emails: “Generally, we want you to change your behavior or be aware of something happening, or think differently about behavior or things like that.”

Some transactional content was urgent and required immediate actions:

Around something we want people to immediately know about, we usually send those transactional versus commercial, we have that option within the salesforce. (C1)
•

Highlighted News: These were the most important news that offices wanted employees to be aware of (e.g., announcements by the university president of new officials), though for most employees these would not be actionable. A communicator introduced this kind of email:

The broadest thing we do is that we have a blog, that we produce content and stories every month, and every month we sent highlights of the blog, that we want people to know about, to be aware of. (C1)
•

Good-to-know News: These were other updates that offices felt employees would benefit from knowing, or that the institution would benefit from having more people aware of. Communicators recognized that individual employees could miss them without consequence.

A bulk email’s content might be good-to-know for some recipients, while being transactional for some others. We asked a communicator whether all the recipients (all students, faculty, and staff) of a bulk email about campus-safety tips should read it: “ Students for sure. For faculty and staff, it’s a reminder, but it’s not something you have to know.” (C5)

B. Channels and Matchings between Message’ Content and Channels. There were 3 channels of bulk emails: single bulk emails (mass emails), newsletters (newsletter emails), and redistribution mechanisms (sent to the communicator’s contacts in decentralized units, see 6.5.2). The matching between types of messages’ content to types of channels were different across communicators. Some communicators sent good-to-know news through newsletters and some sent them through single bulk emails. Some communicators sent highlighted news and transactional content through single bulk emails and some communicators sent them through newsletters:

We send single messages [individual bulk emails] when something is really timely we need to get it out, and when it’s critical that people receive and see it. If something is a story, less critical then that might find its way into our newsletter instead. (C1)

If there is an actionable thing we want people to do, that typically will go into a newsletter bulletin. The single message [individual bulk email] would be like a topic that needs to have a lot more detail but not necessarily an action needed to come out of it. So it’s an awareness thing. (C5)

The different matchings between bulk email’s content and channels between communicators resulted in good-to-know news, highlighted news, or transactional content being put into a newsletter at the same time. We asked C5 whether the recipients of an email should read all of the 10 messages in it: “ Not really. So we really want the message “have the ID card with you” get out, that was important, the most actionable thing, but the rest of it is to communicate that we care.” (C5)

Recipients needed to go through all the messages in each email to screen messages relevant to them. A recipient talked about a newsletter with over 30 messages:

What I do is I go through it, ok, ¡program name¿ … I’m not interested in it, upcoming programs … if I see something interesting then I’ll read and click on that. (R6)

This mismatch between bulk email’s content and channels across different communicators caused confusion about what the newsletters were used for and whether the recipients should view them as important.

6.2.4. Communicators’ various roles — Communicator Directors VS Communicator Staff.

Communicators’ worries with bulk emails varied based on their roles. When we asked what needed to be improved in the bulk email system, the communicator directors cared about the engagement of the community, “We struggle on the engagement, make people know the service we have.” (C2), “Our hardest things are how to let people feel that we all have a part of responsibility in safety, each individual have to make effort, it’s much easier for us to keep the campus safe.” (C4)

Communicator staff would like more tools in the bulk email system which could support their operational work, “It’s great to have people to say ”I don’t want this type of thing” or ”Please send more content like this.”” (C5), “From a user perspective, I think that maybe adapting some common templates could be helpful.” (C6)

6.3. Different Perspectives on Bulk Email’s Values

6.3.1. Communicators’ assessments — high-level bulk email is important.

High-level bulk emails were the emails communicating what was going on generally in the university, like changes in the benefits, leadership, public safety, administration, or working tools; they were usually from the university-wide offices (see Table 6 for the definitions of emails levels). They were recognized as the bulk emails with general importance by communicators. Communicators usually sent them to all faculty and staff: “Public safety is everybody who works at the U should know about, and any big change in benefits or leadership will affect everybody.” (C1)

Communicators viewed the bulk emails from the leaders important, as we asked C5 what kind of emails were considered more important: “Message from the president, from the vice president of university services about campus safety, those types of things.” (C5)

6.3.2. Managers’ assessments — have senses about high-level bulk emails.

Managers sometimes agreed with their employees (recipients) that some bulk emails were not important. Within the 31 emails we let managers assessed, there were 5 emails that the recipient and the manager both agreed that they should scan; and there were 13 emails that the recipient and the manager both agreed that they should be left unread.

Managers sometimes disagreed with their employees. There were 8 cases where employees opened these emails while their managers thought they should skip it. These 8 cases were interesting to employees, but not particularly job-related, like alumni marketing, a company’s on-campus recruitment feedback questionnaire, etc.

Managers thought that employees should have some sense about what was going on at the high level of this university, about leadership and administration changes, as the 5 cases shown in Table 5. They were the bulk emails which managers thought their employees should read while the employees did not, or thought the employees should read in detail while they just scanned. For example, M2 thought R6 should read the bulk email with the title “Your Help Matters Now” and content “Legislative session ends Monday. Please take 60 seconds to make sure that when the final negotiations are done.” in detail, but R6 just left it unread.

6.3.3. Employees’ assessments — high-level bulk email is irrelevant.

Employees often skipped bulk emails about legislation of the university and leadership changes (see Table 5) because 1) they had time pressure; 2) they felt these emails were too high level to be related to themselves as a non-leader staff: “I skimmed and deleted it. It’s a program that’s important to my boss (the dean), but not directly related to my office.” (R7)

Actually, the higher the bulk email’s level, the lower the probability that the bulk email would be opened/read in detail, as we showed in Figure 6, where we compared the levels of the investigated bulk emails (see Table 6) with their open/read in detail rates. A bulk email from a university-wide office had 48% chance of being opened and 14% chance of being read in detail, while for a bulk email from a department’s office these numbers would be 95% and 57%.

The bias on high-level bulk emails made employees miss some useful emails. Employees actually found some of them helpful when being asked to read the unread bulk emails which the managers thought they should. R6 found that the #1 bulk email in Table 5 should be read: “When I read it now, I may click on here and take 30 seconds to make sure the final negotiation. So I didn’t realize that it was so easy to click this button. I was thinking it might involve more like writing a short letter. ”

#	Manager & Recipient	Manager’s Expectation	Employee’s Reaction	Title	Sender	Employee’s Reason
1	M2 - R6	Read in Detail	Unread	Your Help Matters Now	University Government Relations	I just deleted it because I think it’s important to advocate for the university but I don’t do that for myself.
2	M2 - R6	Scan	Unread	¡College Name¿’s Update • Summer 2019	Newsletter Center of ¡College Name¿	I was too busy, this email is not like so important.
3	M1 - R5	Scan	Unread	Reminder - Incorporating GoldPASS into your Career Courses session tomorrow	University Employer Relations	I will not open it, because it’s for undergrad department.
4	M1 - R5	Scan	Unread	PeopleSoft Navigation Update: MyU PeopleSoft Outage This Weekend	PeopleSoft Update Team	I already learn it from other sources.
5	M2 - R6	Read in Detail	Scan	[¡College Name¿ - Staff] ¡College Name¿ Highlights, Leadership Announcements and Resources to Share, 05/20/19	¡College Name¿ Leader’s Office	I am busy and I will spend less time on this.

Table 5. Title/sender of the bulk emails which managers thought their employees should read while they did not, or thought the employees should read in detail while they just scanned, and the employees’ reasons.

#	Level	Meaning
1	University	Sent from university wide offices.
2	College	Sent from college wide offices.
3	Department	Sent from department wide offices.

6.4. Costs of Communication

6.4.1. Recipients’ feeling of burden and their reading strategies.

Controlling the burden on recipients was not one of the goals of the current bulk email system, as stated by a communicator when asked whether a recipient’s time cost was considered: “No, the email we sent are most not nice-to-know emails. They are required information for people.” (C2)

Receiving lots of emails did not necessarily make recipients feel a burden. R9 received over 774 emails per week but still did not get a sense of email overload. R9 talked about their strategy: “I can handle it (bulk email) … I scan it right away … I delete the things I know I don’t have to read … I look at the subject … Mentally you do it in hierarchy — you determine what’s the next, what’s the first.”

The reading strategy ”checking the content whenever a new email came in” might bring on a sense of burden, like R6 said: “ I checked my email pretty quickly. Even when I am working on something, I may go back to my email to see is there anything important that’s coming, and I respond pretty quickly … To be honest, sometimes I felt overwhelmed.”. This result matched with Mark’s findings on work emails and productivity (Mark et al., 2016).

Retrieving the right bulk email from inboxes could be a burden: “I might miss something. Sometimes my supervisor ask me whether you received it — I trashed it then I try to find it. That’s frustrating because in Gmail sometimes I can’t find it. You try to think what’s in the subject line, date …” (R4).

Implication 2: Make Recipients’ Burden Visible to Communicators. The time cost caused by bulk email and the collective burden put on recipients was inapproachable for the communicators. They did not know whether they sent a bulk email to too many recipients and let them spend unnecessary time, or collectively let some recipients feel overloaded. For example, we could monitor employees’ time spent on bulk emails, or the financial cost of each bulk email (Jackson et al., 2006).

6.4.2. Communicators’ Costs — Loss of channel credibility (organizational communication capital).

Communicators, requested by their clients, had to send some bulk emails at the cost of the sender’s credibility. We asked a communicator whether the credibility of bulk email channel was considered:

Yes, we definitely think about it (the credibility of the sender’s name). But we work with different clients, they request the audience they want. (C1)

Recipients learned from experience and stopped opening bulk emails from low-credibility senders. For example, R9 trashed an email from the university fitness center without opening it: “I recognized the sender and trashed it because I am not interested in it.”

Implication 3: Make the Credibility of Bulk Email Channels Visible to Communicators. There was no tool to tell communicators how credible a channel is to its recipients — instead they discarded ineffective channels and replaced with new channels, which was a time consuming strategy. Thus quantifying channels’ credibility and setting standards of usage of channels could be helpful. For example, calculating the credibility and success probability of communication (Dewatripont and Tirole, 2005), and avoiding sending optional bulk emails via high-credibility channels.

6.5. Understanding Current Practice and Its Failings

6.5.1. Lack of personalization tools.

Communicators tried to help recipients find important messages in a newsletter by putting important messages in the top position or putting their keywords in the subjects: “Sometimes we put important messages in the subject line. It’s not the same importance (to all recipients), but they can recognize that it’s important or not. We do try to keep it relevant and make it clear at the top.” (C1)

However, as C1 said, importance of the messages in a bulk email may vary for different recipients. Recipients tended to close a newsletter whose first message was not relevant; and sometimes they did miss the later messages which were useful to them. R6 told us why a bulk email was unread and R6 found it useful when we asked they read the email:

They started with sports, then I thought it’s not relevant to me. But when I go back with you, I found out that it has research that I am interested in. (R6)

Given this situation, communicators thought that personalization of email designs would be helpful, but they did not have the technology to do personalization. Currently, all recipients receive the same subject lines/subtitles/content for each bulk email. We asked a communicator whether they had felt the need for personalization of emails):

We haven’t, although it’s something we would like to explore doing. I don’t think our team has technical expertise. I don’t know where and how to get that type of thing. (C6)

Implication 4: Explore Personalization. Develop personalization tools for communicators to design bulk emails. For example, for a user, estimate the relevance of each message in a bulk email to this user and rank the messages from high to low relevance (Trespalacios and Perkins, 2016).

6.5.2. Flawed Email Targeting Tools

A. Target through mailing lists — recipients cannot opt-out. There were some mailing lists built for certain groups, e.g. all faculty of the university, all staff of a department. Communicators assumed that recipients would unsubscribe from mailing lists they were not interested in. Communicators also designated certain lists without opt-out option to avoid core recipients opt-out. For example, a communicator talked about a monthly newsletter:

For the people who work at administrators, they cannot opt-out of that. I check this. Everyone else is optional, they can unsubscribe. (C3)

However, there were cases that a mailing list was used to send important and unimportant bulk emails at different times. Recipients could not opt-out from this mailing list even though unimportant bulk emails were sent through it because they would not then receive the important emails sent through that mailing list in the future. For example, a communicator talked about an optional bulk email sent to communication directors:

It’s optional for them to take action to do what I ask them to do. This is meant to be kind of helpful service I’m giving … They could (opt-out of this mailing list) but this is a mailing list for the communication directors so they probably won’t. (C5)

B. Target through querying position-database — limited to the human resource system. If there were no mailing lists existing for the target population, communicators would query the database of employees. However, the current bulk email system only connected with the human resource system, which only supported querying based on positions, but not on recipients’ networks or interests. A communicator introduced this process:

We will go to HR, and ask them to pull a list from PeopleSoft. Sometimes, for example, we were trying to reach everyone with family, so we go to HR, and they were not able to pull out. They can only do if the data is in PeopleSoft and we can get it by job code. (C5)

A communicator talked about a bulk email whose target population was all employees who had not downloaded documents from Webex. But the communicator did not have such information in the bulk email system, it was sent to all Webex users instead:

This is the final reminder to Webex users: action required to download reports. Receivers are people who have been using Webex in the last year. That went to 9000 receivers.

We cannot tell who has downloaded their file or not. The audience is who is using Webex and have files. Anybody who has already moved to Zoom, downloaded their files don’t need to take action, but we don’t know who are them. (C2)

C. Redistribution mechanisms — no tracking of bulk emails. If communicators did not have enough information to target recipients, expecting that lower-level units would have more information about their employees, communicators sometimes sent bulk emails to their contacts in units and asked them to forward these emails to the related recipients:

We have in each unit one person in charge of the facility things. And they may have people in their department, in their buildings. Sometimes we will send information to the unit facility leads, which is a small set of people (author’s note: this is a set of 400 to 500 people). That’s when we want other people to distribute something more locally versus from the University Relations. We want people in ¡college¿ to get some messages [emails] from their unit. We may ask them to distribute that message [email]. (C4)

However, communicators could not track whether and how the contacts sent these bulk emails, nor the performance of these bulk emails, when we asked a communicator if they could track the performance of the bulk emails sent to the contacts: “Unfortunately we can not track that.” (C6)

D. When it is hard to target — overwhelm everyone. As both mailing lists and querying were limited to certain scenarios, precise targeting was difficult for the communicators. C6 introduced a group difficult to narrow: “Open enrollment is tough because we don’t know how many people need to take action. That’s kind of a dilemma. Because we try to make that system easier for people so you don’t have to do anything. But that makes it harder for us to do communications.”

Motivated by “getting the proof of delivered”, communicators thought that it was better to overwhelm everyone than to miss a single person. The bulk email above finally went to all employees: “We moved everyone to do it to ensure that no one is left behind.” (C6)

“Sent to all” was one of the reasons that recipients received irrelevant bulk emails. R4 talked about a bulk email they found not useful: “This is what I call blanket message [email] — it goes to everyone, it’s common knowledge, it’s not that related to me.”

Implication 5: Precise targeting is difficult. As previous research showed that emails’ labels would change recipients’ behaviors (Reeves et al., 2008), a mechanism could be designed to hint an email’s value to its recipients before they opened it, for example, labeling whether a bulk email is optional or mandatory, action-needed or not (Wainer et al., 2011).

6.5.3. Communicator transferred the responsibility of being aware

The responsibility of knowing a bulk email was transferred to the recipients after the communicators sent it out. A communicator talked about a newsletter from the college’s dean that they sent to all employees weekly, with 17 messages and 5 tweets from the dean:

They are all important emails, sent once a week. So our expectation is — “This is the important email you get from your college, regarding your job. This email is the business of the college, you should read it.” (C3)

However, recipients found the bulk emails were too many or too long for them to filter out unimportant ones, find important ones, and be aware of all of the content. They felt that it was not the recipient’s responsibility to be aware of all the content. A recipient talked about a weekly newsletter sent to all employees about government activities:

It took so much time to read, 10 paragraphs, nobody gonna read that, it’s gambling. People use emails like a safety cover, but there is no understanding from the recipients’ side. The responsibility shifts from the sender to the recipient, that’s unfair. (R3)

6.5.4. Vicious cycle — recipients did not read then communicators sent more.

Similar to Randall’s findings in organizational communication (Schuler, 1979), the ineffectiveness of bulk email system also brought a vicious cycle — 1) recipients did not read bulk emails on time; 2) to get recipients to read bulk emails/take actions on time, communicators sent bulk emails multiple times/more widely; 3) recipients received more irrelevant bulk emails, lost trust in bulk email channels, and read less bulk emails in the future. A communicator talked about the collection process of the employee engagement survey: “Those messages [emails] start from September, October, in terms of emails, go through January … we do have to do emails more than once because our email open rate is about 61 or 62 percent.” (C6)

Communicators did not realize the existence of the cycle and thought repeated bulk emails would be appreciated: “My assumption is if somebody is emailing me about a service that is going away, and I have taken actions, I will never need to open it, but I will be grateful if I get reminders.” (C2)

However, recipients found they usually opened a bulk email and found that they already read it/still not interested/already took action. A recipient trashed a bulk email about reimbursement:

It is important and relevant but this is not the first time she sent. I already picked up what I need … I only read the first email a couple of months ago … She even attached the same document again … so I just deleted it. (R9)

Implication 6: Provide Feedback to Communicators. To interrupt the vicious cycle, recipients’ feedback could be collected, such as whether the recipient wanted more like this/less like this (like what was applied in social medias (Khan, 2017)), how much time the recipient had spent on it, whether they had already seen/replied to this bulk email. Communicators then could decide whether to keep sending these bulk emails to the recipient, or only distribute reminders to people that have not read or confirmed them.

7. Discussion

7.1. Align Different Stakeholders’ Priorities

We found a large gap between different stakeholders’ priorities. The communicators thought they were sending important high-level information about the organization; and communicators’ clients wanted their messages to be sent broadly. However, our recipients naturally did not view high-level bulk emails as important. This gap was caused by different stakeholder priorities.

On one side, the major task for communicators was to get the “proof of delivering” of bulk emails to their clients, whose first priority was to let the target population be aware of their information and take appropriate actions. On the other side, recipients, who were collectively burdened by the communicators, wanted to receive bulk emails that were interesting and relevant to them.

Outside the organizational environment, these priorities were helpful for companies, like sending advertising emails to customers broadly (Balakrishnan and Parekh, 2014). However, within the organization where all employees had responsibilities of completing the organization’s tasks and minimizing the organization’s costs, this misalignment of priorities brought costs to the organization. The communication channels’ credibility was harmed. The organization’s tasks were not done.

Thus it is important to align different stakeholders’ priorities, and also to include the priority of the organization itself in the bulk email system. Recipients should be reminded that besides the interesting stuff, they have the responsibility of being aware of what is going on in the organization. Communicators and their clients should be reminded that besides distributing their information, they have the responsibility of maintaining the effectiveness of organizational communication channels and avoid employees having to spend time on unnecessary emails. At an extreme, a Key Performance Indicator System (Kueng, 2000), which takes the organization’s priority into consideration, could be integrated into the bulk email system. For communicators, there could be indicators on the time cost, the credibility cost, and their clients’ feedback for each bulk email. For recipients, there could be indicators on the speed of taking actions required by bulk emails, the time spent on reading the bulk emails important to the organization.

7.2. Limitations — Organizational Culture/Structure’s Influence on Bulk Email System

This is a qualitative case study of one study site. The observations might not be generalizable to all kinds of organizations. It is possible that the stakeholders of other organizations would have different practices/perspectives on bulk email systems with respect to their organizations’ cultures/structures.

The organization we studied had a hierarchical management structure — from university leadership (Board of Regents, Office of the President, Office of the Provost), to the college office (deans, academic affairs), to the department office (heads, program directors). The higher the level, the more information they had. Thus the information flow of the bulk email system was mostly one directional — low-level employees got information from the high-level leaders by bulk emails, while high-level leaders rarely needed to get information from low-level employees by bulk emails.

As for stakeholders within an organization, bidirectional interaction proved to be an important factor in their engagement levels of organizational affairs (Norris et al., 2017). Thus this bulk email system’s audience might have less willingness to participate in the bulk email communication and that might cause its ineffectiveness. It is possible that the recipients in the organizations who had a flat structure would have a higher willingness to participate in bulk email communication.

Though hierarchical, the organization’s communication system was to some extent decentralized — each communication offices worked for their own units, had their own goals, served their own clients from the same units or other units. There were no central alignments of these communication offices and some information would be sent multiple times from different offices — these repeated bulk emails then became a burden that multiple communication offices collectively placed on recipients. As the previous research found, “decentralized organization is not a very efficient organization model for work that requires a lot of intensive interaction between different employees.” (Mäki et al., 2004), it is possible that a bulk email system with a central communication office to arrange different communication offices’ tasks and collaboration would work more efficiently.

8. Conclusion and Future Work

Conclusion: Through this study we found that the failure of the organizational bulk email system was systematic — the organizational bulk email system had many stakeholders, but none of them necessarily had a global view of the system or the impacts of their own actions. First, the communicators’ high-level clients wanted their messages sent broadly. Second, the communicators felt they were sending important high-level information through bulk emails; they got high “open rates” and only reported that to their clients. Third, the recipients viewed most of the bulk emails not relevant; they often opened and then rapidly discarded bulk emails without reading the details. Last, though the communicators across the organization worked together in a network that attempted to improve the quality of communication practice, different communicators used bulk email channels differently and there was a limited set of targeting and feedback tools to support their task.

In theory, the answer is simple; there is a long history of information filtering technology to help recipients avoid spam based on their preferences, and email marketing technology to help communicators advertise products to external recipients. But in an organizational setting, recipients or communicators’ preferences are not enough; the organization’s priorities matter. Recipients may not want to know certain things, but their employer expects them to know them nonetheless; communicators and their clients may want to send bulk emails broadly, but they need to consider the cost to the organization.

Impact: We hope that by this case study, we could 1) provide detailed information in how an organizational bulk email system works and fails, with respect to this organization’s structure and nature; 2) to the best of our knowledge, this is the first work focusing on multi-stakeholders’ perspectives on bulk emails within an organization; we hope our protocols on inbox artifact-walkthroughs are useful for future research; 3) given that many conventional in-person communication channels are no longer available for organizations in the current pandemic, organizational bulk email system now takes on even greater importance, especially in internal crisis communication; we hope our study provides facts and possible directions for studying organizational bulk email system in the remote-work environment.

Future Work: The next steps include: 1) collaborate with more organizations to study whether our findings in this case study are common; 2) informed by the design recommendations we make above, design an organizational bulk email system to support multi-stakeholder prioritization. It would include personalization, but tied to both the interests of the recipient and the needs of the organization. It would help communicators predict how their bulk emails would be received based on aggregated prior feedback and would provide mechanisms for managing the cost and value of bulk emails. At an extreme, an organization might even allocate communication budgets to units to reduce wasteful or poorly-targeted organizational bulk emails, but even without this step, we believe the visibility could provide sufficient incentive to already well-motivated communicators.

Acknowledgements.

This work was supported by the National Science Foundation (NSF) under Award No. IIS-2001851, IIS-2000782, and CNS-1952085. The authors would like to thank Anjali Srivastava, University of Minnesota GroupLens Lab members, and our reviewers for providing us with very valuable feedback.

References

(1)
Al-Jarrah et al. (2012) Omar Al-Jarrah, Ismail Khater, and Basheer Al-Duwairi. 2012. Identifying potentially useful email header features for email spam filtering. In The sixth international conference on digital society (ICDS), Vol. 30. 140.
Alrashed et al. (2019) Tarfah Alrashed, Chia-Jung Lee, Peter Bailey, Christopher Lin, Milad Shokouhi, and Susan Dumais. 2019. Evaluating User Actions as a Proxy for Email Significance. In The World Wide Web Conference. 26–36.
Andriof et al. (2002) Jorg Andriof, Sandra Sutherland Rahman, Sandra Waddock, and Bryan Husted. 2002. Introduction: JCC theme issue: Stakeholder responsibility. The Journal of Corporate Citizenship (2002), 16–19.
Balakrishnan and Parekh (2014) R. Balakrishnan and R. Parekh. 2014. Learning to predict subject-line opens for large-scale email marketing. In 2014 IEEE International Conference on Big Data (Big Data). 579–584.
Bekkerman (2004) Ron Bekkerman. 2004. Automatic categorization of email into folders: Benchmark experiments on Enron and SRI corpora. (2004).
Bellotti et al. (2005) Victoria Bellotti, Nicolas Ducheneaut, Mark Howard, Ian Smith, and Rebecca E Grinter. 2005. Quality versus quantity: E-mail-centric task management and its relation with overload. Human–Computer Interaction 20, 1-2 (2005), 89–138.
Bermejo et al. (2011) Pablo Bermejo, Jose A Gámez, and Jose M Puerta. 2011. Improving the performance of Naive Bayes multinomial in e-mail foldering by introducing distribution-based balance of datasets. Expert Systems with Applications 38, 3 (2011), 2072–2080.
Beyer and Holtzblatt (1999) Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42.
Buchanan (2012) David A Buchanan. 2012. Case studies in organizational research. Qualitative organizational research (2012), 351–370.
Carter et al. (2011) Owen BJ Carter, Robert Donovan, and Geoffrey Jalleh. 2011. Using viral e-mails to distribute tobacco control advertisements: an experimental investigation. Journal of health communication 16, 7 (2011), 698–707.
Celik and Ölçer (2018) Bünyamin Celik and Zeynep Ölçer. 2018. The Importance of Corporate Email Service and Google Education Service for Education Institutions (A Case Study of Ronaki Duhok Education Company in Iraq). International Journal of Social Sciences & Educational Studies 5, 1 (2018), 168.
Chakrabarty and Roy (2014) A. Chakrabarty and S. Roy. 2014. An optimized k-NN classifier based on minimum spanning tree for email filtering. In 2014 2nd International Conference on Business and Information Management (ICBIM). 47–52.
Charmaz (2014) Kathy Charmaz. 2014. Constructing grounded theory. Sage.
Dabbish et al. (2003) Laura Dabbish, Gina Venolia, and JJ Cadiz. 2003. Marked for Deletion: An Analysis of Email Data. In CHI ’03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI EA ’03). Association for Computing Machinery, New York, NY, USA, 924–925. https://doi.org/10.1145/765891.766073
Dabbish and Kraut (2006) Laura A. Dabbish and Robert E. Kraut. 2006. Email Overload at Work: An Analysis of Factors Associated with Email Strain. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (Banff, Alberta, Canada) (CSCW ’06). Association for Computing Machinery, New York, NY, USA, 431–440. https://doi.org/10.1145/1180875.1180941
Dawkins (2018) Roger Dawkins. 2018. Mass email at university: current literature and tactics for future use. Open Learning: The Journal of Open, Distance and e-Learning 0, 0 (2018), 1–17. https://doi.org/10.1080/02680513.2018.1556090 arXiv:https://doi.org/10.1080/02680513.2018.1556090
Dewatripont and Tirole (2005) Mathias Dewatripont and Jean Tirole. 2005. Modes of communication. Journal of political economy 113, 6 (2005), 1217–1238.
Dill and Sporn (1995) David D Dill and Barbara Sporn. 1995. Emerging Patterns of Social Demand and University Reform: Through a Glass Darkly. Issues in Higher Education. ERIC.
Downs and Adrian (2012) Cal W Downs and Allyson D Adrian. 2012. Assessing organizational communication: Strategic communication audits. Guilford Press.
Fisher et al. (2006) Danyel Fisher, AJ Brush, Eric Gleave, and Marc A Smith. 2006. Revisiting Whittaker & Sidner’s” email overload” ten years later. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work. 309–312.
Goldhaber (1990) Gerald M Goldhaber. 1990. Organizational communication. Technical Report. WCB,.
Gray and Haahr (2004) Alan Gray and Mads Haahr. 2004. Personalised, Collaborative Spam Filtering.. In CEAS.
Greenbaum (1974) Howard H Greenbaum. 1974. The audit of organizational communication. Academy of Management Journal 17, 4 (1974), 739–754.
Grevet et al. (2014) Catherine Grevet, David Choi, Debra Kumar, and Eric Gilbert. 2014. Overload is overloaded: email in the age of Gmail. In Proceedings of the sigchi conference on human factors in computing systems. 793–802.
Handy (2007) Charles Handy. 2007. Understanding organizations. Penguin Uk.
Huang et al. (2009) Chien-Chih Huang, Tung-Ching Lin, and Kuei-Ju Lin. 2009. Factors affecting pass-along email intentions (PAEIs): Integrating the social capital and social cognition theories. Electronic Commerce Research and Applications 8, 3 (2009), 160–169.
Jackson et al. (2006) Thomas W Jackson, Anthony Burgess, and Janet Edwards. 2006. A simple approach to improving email communication. Commun. ACM 49, 6 (2006), 107–109.
Jackson et al. (2003) Thomas W. Jackson, Ray Dawson, and Darren Wilson. 2003. Understanding Email Interaction Increases Organizational Productivity. Commun. ACM 46, 8 (Aug. 2003), 80–84. https://doi.org/10.1145/859670.859673
Katz and Kahn (2008) Daniel Katz and Robert L Kahn. 2008. Communication: the flow of information. Communication theory (2008), 382–389.
Khan (2017) M Laeeq Khan. 2017. Social media engagement: What motivates user participation and consumption on YouTube? Computers in human behavior 66 (2017), 236–247.
Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning. Springer, 217–226.
Kueng (2000) Peter Kueng. 2000. Process performance measurement system: a tool to support process-based organizations. Total quality management 11, 1 (2000), 67–85.
Kuh and Whitt (2000) GD Kuh and EJ Whitt. 2000. Culture in American colleges and universities. Organizational & Governance in Higher Education. MC Brown, Ed.
Lazar et al. (2017) Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.
Lu et al. (2012) Jie Lu, Zhen Wen, Shimei Pan, and Jennifer Lai. 2012. EPIC: a multi-tiered approach to enterprise email prioritization. In Proceedings of the 2012 ACM international conference on Intelligent User Interfaces. 199–202.
Mäki et al. (2004) Eerikki Mäki, Eila Järvenpää, Kirsi Ziegler, and Kirsi Ziegler. 2004. Communication and knowledge sharing in a decentralized organization. (2004).
Malone (1987) T Malone. 1987. The information lens: An intelligent system for sharing information in organizations. Proceedings, Computer Human Interaction, Human Factors in Computing Systems (1987).
Mark et al. (2016) Gloria Mark, Shamsi T Iqbal, Mary Czerwinski, Paul Johns, Akane Sano, and Yuliya Lutchyn. 2016. Email duration, batching and self-interruption: Patterns of email use on productivity and stress. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1717–1728.
Masland (1985) Andrew T Masland. 1985. Organizational culture in the study of higher education. The Review of Higher Education 8, 2 (1985), 157–168.
Merten and Gloor (2010) Frank Merten and Peter Gloor. 2010. Too much e-mail decreases job satisfaction. Procedia-Social and Behavioral Sciences 2, 4 (2010), 6457–6465.
Mishra et al. (2014) Karen Mishra, Lois Boynton, and Aneil Mishra. 2014. Driving employee engagement: The expanded role of internal communications. International Journal of Business Communication 51, 2 (2014), 183–202.
Myers and Myers (1982) Michele Tolela Myers and Gail E Myers. 1982. Managing by communication: An organizational approach. McGraw-Hill College.
Norris et al. (2017) Jill M Norris, Deborah E White, Lorelli Nowell, Kelly Mrklas, and Henry T Stelfox. 2017. How do stakeholders from multiple hierarchical levels of a large provincial health system define engagement? A qualitative study. Implementation Science 12, 1 (2017), 98.
Odine (2015) Maurice Odine. 2015. Communication problems in management. Journal of Emerging Issues in Economics, Finance, and Banking (JEIEFB) 4, 2 (2015).
Paczkowski and Kuruzovich (2016) William F Paczkowski and Jason Kuruzovich. 2016. Checking email in the bathroom: monitoring email responsiveness behavior in the workplace. American Journal of Management 16, 2 (2016).
Rasmussen and Ulrich (2015) Thomas Rasmussen and Dave Ulrich. 2015. Learning from practice: how HR analytics avoids being a management fad. Organizational Dynamics 44, 3 (2015), 236–242.
Raven and Flanders (1996) Mary Elizabeth Raven and Alicia Flanders. 1996. Using Contextual Inquiry to Learn about Your Audiences. SIGDOC Asterisk J. Comput. Doc. 20, 1 (Feb. 1996), 1–13. https://doi.org/10.1145/227614.227615
Reeves et al. (2008) Byron Reeves, Simon Roy, Brian Gorman, and Teresa Morley. 2008. A marketplace for attention: Responses to a synthetic currency used to signal information importance in e-mail. First Monday 13, 5 (2008).
Sarrafzadeh et al. (2019) Bahareh Sarrafzadeh, Ahmed Hassan Awadallah, Christopher H. Lin, Chia-Jung Lee, Milad Shokouhi, and Susan T. Dumais. 2019. Characterizing and Predicting Email Deferral Behavior. In WSDM ’19.
Schuler (1979) Randall S. Schuler. 1979. A role perception transactional process model for organizational communication-outcome relationships. Organizational Behavior and Human Performance 23, 2 (1979), 268 – 291. https://doi.org/10.1016/0030-5073(79)90058-8
Shriberg et al. (2002) Michael P Shriberg et al. 2002. Sustainability in US higher education: organizational factors influencing campus environmental performance and leadership. Ph.D. Dissertation. University of Michigan Ann Arbor, MI.
Sproull and Kiesler (1991) Lee Sproull and Sara Kiesler. 1991. Computers, networks and work. Scientific American 265, 3 (1991), 116–127.
Stohl (1995) Cynthia Stohl. 1995. Organizational communication. Number 5. Sage.
Stohl and Redding (1987) Cynthia Stohl and W Charles Redding. 1987. Messages and message exchange processes. (1987).
Swanborn (2010) Peter Swanborn. 2010. Case study research: What, why and how? Sage.
Trespalacios and Perkins (2016) Jesús H Trespalacios and Ross A Perkins. 2016. Effects of personalization and invitation email length on web-based survey response rates. TechTrends 60, 4 (2016), 330–335.
Van Vleck (2012) Tom Van Vleck. 2012. Electronic mail and text messaging in CTSS, 1965-1973. IEEE Annals of the History of Computing 34, 1 (2012), 4–6.
Wainer et al. (2011) Jaclyn Wainer, Laura Dabbish, and Robert Kraut. 2011. Should I Open This Email? Inbox-Level Cues, Curiosity and Attention to Email. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 3439–3448. https://doi.org/10.1145/1978942.1979456
Whittaker and Sidner (1996) Steve Whittaker and Candace Sidner. 1996. Email overload: exploring personal information management of email. In Proceedings of the SIGCHI conference on Human factors in computing systems. 276–283.
Yang et al. (2017) Liu Yang, Susan T Dumais, Paul N Bennett, and Ahmed Hassan Awadallah. 2017. Characterizing and predicting enterprise email reply behavior. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 235–244.
Yin (2003) Robert K Yin. 2003. Case study research: Design and methods (Vol. 5). (2003).

9. Appendices

Appendix A Communicator Interview Protocol

See Table 7.

Part		Questions
1.Practice	1	How do you send bulk emails for your unit and why do you send it in that way ?
		Who are your client? What are their requests? Do they send emails themselves?
		Who offer the mailing list and content?
	2	When do you send bulk emails to all employees versus subgroups?
	3	When do you decide to put a message in a bulletin of newsletter versus an individual bulk email?
	4	Besides newsletters and individual bulk emails, what are the other mechanism you use to communicate?
	5	What happened with the bulk emails you sent? Are you aware of how often they read or do they understand it carefully?
2. Email Cases	1	The title, goal, and recipient of the email
	2	Which channel did you use to send it?
	3	Who do you imagine should read this? Who do you imagine did read this? Are they the same?
	4	Is who should read this the same as who the email was sent to?
	5	Who should read in detail and who can just scan it?
	6	Who do you imagine should take actions (click links, take surveys, reply)? Did they take the actions?
	7	Do you have experience that some recipients asked about/forgot messages that you’ve already sent in an email? When will you send it for multiple times?
3. Assessment	1	What’s your sense of how email communication between university and employees work (well or poorly)?

Table 7. Interview Protocol of Communicators.

Appendix B Recipients Interview Protocol

See Table 8 for search commands for different types of emails. The following are the questions we asked the recipient for each email:

•

Did you label/trash this email? Did you open this email or leave it unread? Why?
•

Did you recognize the sender?
•

If you opened it, did you scan or read it in detail? Why?
•

If you didn’t open it, open it now, do you find anything important to you?
•

Did this email require actions? Did you take actions and how soon did you take? Why?
•

Rate the importance/urgency/relevance of this email from 1 (lowest) - 5 (highest).

Questions	Command
How many emails did the participant receive within 1 week?	newer_than:7d,in:anywhere AND NOT from:me
How many emails did the participant receive and not read within 1 week?	newer_than:7d,in:anywhere AND label:unread AND NOT from:me
How many emails did the participant receive from their organizations within 1 week?	newer_than:7d from:umn.edu AND NOT from:me ,in:anywhere
How many emails did the participant receive from their organizations and unread within 1 week?	newer_than:7d,in:anywhere AND NOT from:me AND label:unread AND from:umn.edu
How many mass emails did the participant receive from the organization within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND (list:(local) OR list:(list) OR to:(lists) OR (category:(Forums\|Promotions) )) ,in:anywhere
How many mass emails did the participant receive from the organization and unread within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND label:unread AND (list:(local) OR list:(list) OR to:(lists) OR (category:(Forums\|Promotions))) ,in:anywhere
How many newsletters did the participant receive from the organization within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND (list:(local) OR list:(list) OR to:(lists) OR (category:(Forums\|Promotions) )) AND (subject:(news\|update\|brief)) ,in:anywhere
How many newsletters did the participant receive from the organization and unread within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND label:unread AND (list:(local) OR list:(list) OR to:(lists) OR (category:(Forums\|Promotions))) AND (subject:(news\|update\|brief)) ,in:anywhere
How many personal emails did the participant receive from the organization within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND to:umn.edu AND NOT (list:(local) OR list:(list) OR to:lists OR (category:(Forums\|Promotions))) ,in:anywhere
How many personal emails did the participant receive from the organization and unread within 1 week?	newer_than:7d,from:umn.edu AND NOT from:me AND to:umn.edu AND NOT (list:(local) OR list:(list) OR to:lists OR (category:(Forums\|Promotions))) ,in:anywhere ,label:unread

Table 8. Email searching commands for Gmail. The University of Minnesota used a Gmail system based on G Suite for Education.