By
Subject Institutions and Programs
CHAPTER THREE
METHODOLOGY
This research required the investigation of a broad and complex subject area. A detailed
study was needed of variables which affect the content, functionality, and value of
academic World Wide Web sites. It was necessary to assess the perception of faculty
members in regard to social order variables-as those variables affect Web site creation,
functionality, and value. It was also important to determine the opinions of faculty
members, as those opinions relate to the Web sites of their academic programs.
The content analysis method was chosen for examination of Web sites. Content analysis allows for measurement of communication content in "a systematic, objective, and quantitative manner" (Wimmer & Dominick, 1994, p. 164). Content analysis is widely favored among researchers investigating electronic or published media content. This is because content analysis allows for a "systematic examination of materials that are more typically evaluated on an impressionistic basis"-such as Web sites (See Babbie, 1990, p. 30).
The survey research method was chosen for assessing the' perceptions of social order variables of faculty, and determining the opinions of faculty in regard to the success of Web sites hosted by their programs. The survey research method was chosen because it allows for the investigation of problems in realistic situations. It allows for the collection of large amounts of data, from a variety of different people, in a relatively short period of time (Wimmer & Dominick, 1994). The survey research method "can be used profitably in the examination of many social topics and can be especially effective when combined with other methods" (Babbie, 1990, p. 40).
Combining content analysis with the survey research method results in a data-gathering strategy which allows for quantitative understandings of the different dimensions of Web site content-in the context of the social order relationships which bring them about and allow them to be sustained. A research methodology of this type is uncommon. As already stated in the Review of Literature, nearly all existing research into World Wide Web sites is content analytic in nature; much of the existing research into Web site use is survey research. Combining these strategies in one investigation will allow for a much more thorough analysis than has heretofore been offered, particularly in relation to the key social order variables which both affect and are affected by site content.
This chapter begins with a description of the subject institutions, programs, and World Wide Web sites involved in the content analysis research. It follows with information about faculty who were asked to participate in the survey. Web site content analysis procedures are explained. Then, the survey instrument, procedures, and response rate are discussed. The chapter ends with a description of statistical analysis procedures used in the study.
Subject Institutions and Programs
Higher education institutions and academic programs selected for study in this research were identified through listings in the 1998-99 Directory of the Association for Education in Journalism and Mass Communication (See Appendix A). The AEJMC directory was used because the organization is the primary professional group for journalism/ mass communication educators and their academic programs. It can be argued that any journalism/ mass communication program wanting to be considered 'serious' within the discipline should maintain AEJMC membership. Membership would include a directory listing. The AEJMC directory includes an alphabetical listing of approximately 400 journalism/ mass communication programs in the U.S. and Puerto Rico. From this list, 200 programs, or about 50 percent of the total, were chosen randomly for inclusion in the population for survey (See Appendix B).
A World Wide Web search was conducted in November, 1998, through Yahoo, Web Crawler, Lycos and InfoSeek search engines determined that each college or university in the population for survey had at least one institutional or program WWW site listed with search engines. Thus, all 200 journalism/ mass communication programs were initially included in the research effort.
An effort was made to access each of the listed WWW addresses. A listed WWW address for one institutional site returned the message "file not found" from its host server. Five academic program WWW site addresses also returned the "file not found" message from their host servers. Since WWW site content associated with these institutions and programs could not be evaluated, they were disqualified from the research effort. Another program was disqualified because the content of its journalism/ mass communication site was exclusively devoted to the program's television station. Site content could not be evaluated using the content analysis instrument.
Of the 193 qualified colleges and universities, 126 (65 %) were public institutions. The remaining 67 (35 %) were private institutions.
Each institution's Carnegie Foundation classification (Carnegie Foundation Classification. . ., 1994) was noted. The Carnegie classification scheme offers an already-established objective criteria for evaluating structure, organization, and curricula of higher education institutions (See Bowen, 1996; Pascarella & Terenzini, 1991; Boyer, 1987). Using these classifications within the research framework allows for distinguishing institutions and the academic programs they house in an ordered way. This strategy is consistent with past research in higher education functions and outcomes (Clark, 1991; Dill, 1991). The study population of 193 institutions included colleges and universities in all eight Carnegie classes: Research I, 32 (17 %); Research II, 16 (8 %); Doctoral I, 13 (7 %); Doctoral II, 18 (9 %); Masters I, 86 (45 %); Masters II, 10 (5 %); Baccalaureate I, 3 (2 %); and Baccalaureate II, 15 (8 %).
Each institution's journalism/ mass communication program structure and subject area description also were noted, to allow a more thorough means of distinguishing among academic units in an ordered and recognizable way. The structure of qualifying academic programs, as indicated by AEJMC directory information and Web site content, was as follows: College, 6 (3 %); School, 41 (21 %); Department, 131 (68 %); Division, 7 (4 %); and Program, 8 (4 %). The subject area description of qualifying academic programs, as indicated by AEJMC directory information and Web site content, was as follows: "Journalism," 41 (21 %); "Mass Communication," 23 (12 %); "Communication" or "Communications," 67 (35 %); or, a combination of different descriptors, 62 (or, 32 %).
Each institution was also classified by presence or absence of graduate-level offerings within the journalism/ mass communication curriculum, to identify program characteristics which would be relevant to a discussion of social order issues within the academic unit. According to AEJMC directory information and Web site content, 90 of the qualifying academic programs (or, 47 %) offered graduate degrees, and 103 (or, 53 %) did not.
A survey population of program faculty also was identified for study. Faculty members were identified through e-mail links on qualified academic program Web sites. An effort was made to identify and contact faculty whose roles were as follows:
1) The program head, chairperson, or other similarly-titled administrator who is responsible for program administration, budget and resources, office management, and other such duties and has been identified as to be "the chief architect of the department's future" (Tucker, 1984, p. 35). Because the chair is a primary agent of social order within the educational environment (See Peca, 1991) his or her perceptions of social order are critical.
2) The Web site administrator, manager, or other similarly-titled employee. This person is responsible for design and development of content of the program's Web site, management of technical issues and monitoring of user feedback (See Harrison, 1996). Regardless of whether the site administrator is a journalism/ mass communication faculty or staff member or is housed elsewhere in the institutional hierarchy, the site administrator is immediately responsible for coordinating a variety of different people and tasks to create Web site content for the program. Thus, the administrator plays a key role in establishing social order for the program as mediated by and through the WWW site.
3) At least one other journalism/ mass communication faculty member who has neither department head nor WWW maintenance responsibilities. The perspectives of these faculty members are important because these faculty work in an academic community which is "created and sustained in the enactment of roles" (Duncan, 1968, p. 63). Their perspectives of these roles affect and are affected by the Web site.
Existing content analysis designs located by the author were insufficient to fully extrapolate the visual, operational, and informational elements of Web sites hosted by academic programs. Therefore, the author created a content analysis instrument (See Appendix C) and procedure similar to that used by Mitchell, 1996. The data gathering procedure was broadened to address the three content dimensions to be studied. The content analysis procedure identified presence or absence of 15 types of visual enhancements, 18 types of operational enhancements and 24 categories of information which are commonly found on Web sites (Vora, 1998; December & Randall, 1995). Sites were scored to receive one point for each enhancement type present, regardless of the number of instances of the enhancement which were evident. No points were given for enhancement types not present. Total site scores were calculated for visual, operational, and informational enhancements.
Content analysis of qualified World Wide Web sites was carried out by the author between January 1-10, 1999. Site access was accomplished on an IBM Aptiva 233 MHz PC running a 56K Flex Modem and operating with an IBM high color, 16-bit, 800x600 resolution monitor. The Internet was accessed via Southwestern Bell Internet service running Netscape browser version 4.05. A content analysis form for each accessed site was completed manually as each site was accessed and analyzed. Some sites which were not electronically accessible on first attempt. In each case, two additional attempts were made. Upon completion of content analysis, data from content analysis forms were verified and entered into Minitab Version 12, and later into SPSS Version 8.0 for statistical analysis.
Existing research on social order can be categorized as observational study (Brown, 1997; Taylor, 1997; Becher, 1989), historical case study (Cowan, 1997), commentary (Postrel, 1998), meta-theoretical discussion (Malone, 1995), content analysis (Gronbeck, 1997), or dramatic or rhetorical analysis (Bitenc, 1998; von Busack, 1997). Because the author could locate no existing research which addressed Web site issues in the context of social order, the author was unable to replicate an existing survey instrument to quantify faculty perceptions of order or disorder as they relate to Web sites used within academic programs.
Therefore, an original survey instrument was created (See Appendix D). The questionnaire addressed social order through its previously-defined key components--predictability or coordination (Elster, 1989) as they relate to division of labor, trust among colleagues, regulation of power, and legitimization of activity (Eisenstadt, 1992).
Sixteen statements were used, as follows: Division of labor x predictable action (two statements); division of labor x coordination (two statements); trust x predictable action (two statements); trust x coordination (two statements); regulation of power x predictable action (two statements); regulation of power x coordination (two statements); legitimization of activity x predictable action (two statements); and, legitimization of activity x coordination (two statements).
An additional seven statements were posed in an effort to gauge the opinion of each respondent of the Web site hosted by his/her academic program, and whether the site is complete, professional, attractive, relevant, clearly-defined, professionally maintained, and involving of faculty, staff and students.
Using the computer described above, 750 individuals qualified for survey were identified via e-mail links on journalism/ mass communication academic program Web sites in the population for study. Those qualified included 92 faculty identified on Web sites as program chairs, 76 faculty identified as Web site administrators, and 582 faculty members who were not identified as either the program chair or Web site administrator. One request to participate in the survey (See Appendix E) was sent to the linked e-mail address for each of the qualified individuals between January 1-10, 1999. The request included an embedded hyperlink to allow respondents to access the World Wide Web site containing the survey instrument.
It became evident early in the survey that response rates would be low. As a result, between January 23-25, 1999, a "reminder" e-mail notice (See Appendix F) was sent in an effort to increase response percentages. Those participants who had already submitted a completed survey form were asked to disregard the reminder. Between February 5-7, 1999, a second "reminder" e-mail notice (See Appendix G) was sent in a final effort to increase response percentages. Those participants who had already submitted a completed survey form were asked to disregard the reminder. Both reminders included an e-mail version of the questionnaire, so that any participant unable to access the WWW survey instrument could participate in the survey. A final survey response rate of 20 percent was obtained.
The survey asked each respondent to indicate agreement to statements posed on a 1-to-5 scale, with "1" representing "strongly disagree" and "5" representing "strongly agree." Each statement in the Web form defaulted to a "0" to allow identification of statements for which responses were not offered. An open-ended comment section was included, for respondents to elaborate on responses if desired. No length limit was imposed for comments.
In drafting the survey questionnaire, the author reviewed instruments previously administered electronically to measure attitudes about Web-based enhancement (Client survey. . ., 1998; Web development survey, 1998) in order that the instrument used might be as simple and understandable as possible. Several guides which address the key concerns involved when surveying online users were consulted (Turner & Turner, 1998; Narins, 1998; Smith, 1997) and sought feedback from several academic researchers and statisticians outside the population for survey. Pilot testing showed it possible for respondents to complete the Web-based survey form in five minutes or less.
Survey responses submitted by participants through the WWW-based survey form were automatically e-mailed to the author's e-mail address of tsgroup@swbell.net. The respondent-encoded ZIP code allowed each response to be verified to assure it came from an institution and academic program within the population for survey. Survey responses returned as a reply to the "reminder" e-mail version of the survey were downloaded and verified using similar procedures.
The survey resulted in 127 usable responses from participants. This reflects a response rate of 20 percent, which is seen as lower than acceptable for most forms of survey research (See Babbie, 1990). A total of 90 e-mail messages were returned to the author as "undeliverable" due to invalid e-mail links on host program Web sites. A total of 29 intended participants sent e-mail replies to the author, declining to participate in the survey for one or more of a variety of reasons. Two survey responses were received lacking ZIP codes and were disqualified. The remaining 502 requests to participate in the survey sent by the author were not acted upon by recipients.
Upon completion of the content analysis, the collected data was assembled in the Minitab Version 12 software program, and later in SPSS Version 8.0, to provide a basic statistical picture of the sites analyzed. Basic percentage, mean, and standard deviation scores were calculated to address most of the issues brought forth through the research questions.
A series of t tests was conducted at the .01 and .05 levels to determine whether mean enhancement scores of journalism/ mass communication program Web sites were significantly different based on public or private institutional affiliation, or presence or absence of graduate program offerings. The t tests were used because they offer an effective method for determining whether a group of scores is normally distributed around a given mean.
A preliminary review of subject institutions and obtained data revealed that no relationships were likely to be identified between journalism/ mass communication academic program structure or subject area description and levels of Web site enhancement. These areas for examination were dropped from the study.
One-way analysis of variance tests were conducted at the .01 and .05 levels to compare the total journalism/ mass communication Web site enhancement mean scores among programs within each of the Carnegie classifications. ANOVA tests were used because the data involved were non-proportional mean scores; each institutional affiliation classification and responses obtained from faculty within it were independent of all others.
One-way analysis of variance tests were conducted at the .01 and .05 levels to compare mean scores given by faculty on social order statements. Mean responses were compared by faculty member job responsibility classification, by public or private institutional affiliation, by Carnegie classification, and by presence or absence of a graduate program within the host institution.
A series of t tests was conducted at the .01 and .05 levels to determine whether mean rankings given by faculty to their own academic program Web sites and related socially-ordered procedures were significantly different, based on faculty member job responsibility classification, public or private institutional affiliation, Carnegie classification, or presence or absence of a graduate program within the host institution.
