Papers

Porter, S.R., & Whitcomb, M.E. (2002). The Impact of Contact Type on Web Survey Response Rates.  Paper presented at the meeting of the Northeast Association of Institutional Research, Annapolis, MD.

Although research on web surveys has increased over the past few years, research on aspects of the email contact is very limited.  We use a web survey of over 11,000 high school students to test whether contact techniques used to increase response rates in paper surveys successfully translate to the electronic realm.  Two experiments tested various aspects of the email contact.  The first examines the effect of personalization and authority of the sender by altering the email address of the sender, the title and office of the sender, and the salutation to the recipient.  The second experiment examines the effect of scarcity by altering the inclusion of statements of selectivity and a survey deadline.  We found no differences between the control and experimental groups in the first experiment, but large differences between groups in the second experiment.

Porter, S. R., & Umbach, P.D. (2002). College major choice: An analysis of person-environment fit. Paper presented at the meeting of the Association for Institutional Research, Toronto, Canada.

Although recent research suggests that congruence between students and academic environment is critical for successful student outcomes, little research has been done on student college major choice. Using Holland’s theory of careers, we analyze college major choice using a multinomial logit model with robust error estimates.  This approach allows us to analyze double and triple majors.  We use the CIRP Freshman Survey and institutional data for three cohorts of first-year students at a selective liberal arts college to study the factors that affect college major choice, both at entry and at graduation.

Porter, S.R., & Whitcomb, M.E. (2001). The impact of lottery incentives on student survey response rates. Paper presented at the meeting of the Northeast Association of Institutional Research, Boston, MA.

Lottery incentives are widely used by institutional researchers despite a lack of research documenting their effectiveness.  A controlled experiment tested the effects of lottery incentives using a prospective college applicant web survey, with emails sent to over 9,000 high school students.  The impact of the level of lottery incentive on response rates, time to response, and response bias is discussed.

Umbach, P.D. & Porter, S.R. (2001). How do academic departments impact student satisfaction? Understanding the contextual effects of departments. Paper presented at the meeting of the Association of Institutional Research, Long Beach, CA.

Using multilevel modeling to analyze survey data from over 1300 alumni from a large research university, this study proposes to examine the impact that departments have on student satisfaction.  Controlling for individual characteristics, we found that characteristics of departments such as size, faculty contact with students, research emphasis, and proportion of female undergraduates had a significant impact on satisfaction with education in the major and the perceived impact that college had on skill development.

Porter, S.R., & Umbach, P.D. (2001). What works best? Collecting alumni data with multiple technologies. Paper presented at the meeting of the Association of Institutional Research, Long Beach, CA.

We present results from an experiment in which alumni surveys were sent to one-year alumni of a large, public research university divided into four groups that differed by 1) whether they received a check-box or machine-scannable survey form and 2) whether they were told of a website where the survey could be filled out instead of using their paper form. We analyze the data to determine which of the four approaches was most effective in terms of response rates and response bias.

Porter, S.R. (2000). Understanding retention outcomes: using multiple data sources to distinguish between dropouts, stopouts and transfer-outs. Paper presented at the National Forum on First-Year Assessment, Houston, TX.

Both assessment practitioners and administrators often view retention as a stay versus go outcome, but first-year outcomes are much more complex, as some students decide to stay, others to transfer, some simply take some time off and others decide to discontinue their education altogether. This paper discusses theoretical and empirical problems in confining retention studies to stay-versus go outcomes, and details data resources such as exit surveys, transcript requests, withdrawn student surveys, state transfer student databases, and the National Student Loan Clearinghouse’s Enrollment Search program that can all be used to gain a richer understanding of first-year outcomes.

Porter, S.R., & Umbach, P.D. (2000). We can't get there in time: Assessing the time between classes and classrooms disruptions. Paper presented at the meeting of the Northeast Association of Institutional Research, Pittsburgh, PA.

In response to student and faculty complaints about the amount of time available to travel between classes, an analysis of the time between classes problem was conducted at a large, public research university. Using facilities, course scheduling and student survey data, we discovered that many students had distances to travel between classes that would normally take longer than the allotted ten minutes.  This forced them to leave class early, arrive to class late or skip class altogether and often left them with an inadequate amount of time to complete exams.  These analyses supported a decision to implement a policy regarding student scheduling.

Porter, S.R. (2001). Predicted graduation and retention rates as performance indicators: The neglected role of uncertainty. Paper presented at the meeting of the Southern Association of Institutional Research, Myrtle Beach, SC.

Using predicted retention and graduation rates from statistical models to evaluate universities and colleges has grown increasingly popular. More recently, several states have begun to link budgets to performance on these indicators. Proponents of predicted rates fail to recognize that these predictions contain substantial error, and that this error, or uncertainty about the estimates, must be taken into account when evaluating performance. Using data from a small state higher education system to estimate a system-wide one-year retention model, the analysis reveals that several so-called under-performing institutions are performing exactly as expected.

Porter, S.R., & Umbach, P.D. (2000). Will increasing courseloads save money? A multilevel analysis and simulation of faculty research productivity. Paper presented at the meeting of the Southern Association of Institutional Research, Myrtle Beach, SC.

Although there is a growing movement to mandate minimum faculty teaching requirements, there have been no rigorous analyses of the impact of such requirements on research productivity. Using the 1993 NSOPF data, we estimate multilevel models of research dollars generated and refereed publications produced. We then use the results in simulations to determine the reductions in teaching costs and losses in grant dollars and publications at both the university and academic discipline level under two scenarios of mandated teaching requirements. Implications of our findings for the accountability movement are discussed.

Porter, S.R., & Umbach, P.D. (2000). Analyzing faculty workload data using multilevel modeling. Paper presented at the Association of Institutional Research meeting, Cincinnati, OH. 

Research on faculty productivity fails to account for the hierarchical nature of the data. Faculty within an academic discipline more closely resemble one another than faculty in other disciplines, resulting in dependent observations and thus inaccurate statistical results. Unlike ordinary least squares, multilevel modeling takes into account this grouping effect. The paper analyzes the research productivity of 1,104 tenured/tenure-track faculty from the 1993 NSOPF survey to compare traditional regression models with a random coefficients model. The results indicate a large grouping effect on research productivity, and the statistical as well as the substantive results of the random coefficients model differ significantly from the regression approach.

Porter, S.R. (2000). Can statistical modeling increase annual fund performance? An experiment at the University of Maryland, College Park. Paper presented at the CASE/AIR conference, St. Louis, MO.

Annual funds face pressures to contact all alumni to maximize participation, but these efforts are costly. This paper uses a logistic regression model to predict likely donors amongst alumni from the College of Arts & Humanities at the University of Maryland, College Park. Alumni were grouped according to their predicted probability of donating and then solicited for contributions during the current year’s Annual Fund drive. Donation rates between likely and unlikely donors were not statistically significant from one another. Possible reasons for this null result are discussed.

Porter, S.R. (1999, 2000). Including transfer-out behavior in retention models: Using the NSLC Enrollment Search data. Paper presented at the meeting of the Association of Institutional Research, Cincinnati, OH, and the meeting of the Northeast Association of Institutional Research meeting, Newport, RI.

Almost all studies of retention inappropriately combine stopouts with transfer-outs due to a lack of data. The National Student Loan Clearinghouse has created a new database that tracks students across institutions. These data in combination with institutional databases now allow researchers to take into account both stopout and transfer-out behavior. Using NSLC data for the University of Maryland, College Park, the paper analyzes one-year retention with dichotomous and multinomial logit under two specifications: the traditional binary retained/not retained dependent variable and a three-outcome dependent variable where students are coded as retained, transferred to another institution, or stopped out. Taking into account transfer-out behavior affects not only the statistical significance of the explanatory variables but also their substantive interpretation. 

Porter, S.R. (1999). Viewing One-Year Retention as a Continuum: the Use of Dichotomous Logistic Regression, Ordered Logit and Multinomial Logit. Paper presented at the Association of Institutional Research meeting, Seattle, WA.

Studies of one-year retention commonly use a binary outcome (retained after one year, yes/no) as the dependent variable. This paper examines the use of alternate specifications of one-year retention that include information about spring semester stopout behavior. Different types of discrete choice models are first described. Next, the multinomial logit model using four outcomes (enrolled first fall semester only; enrolled first fall semester and spring semester; enrolled first fall semester, stopped out and returned the following fall; and enrolled all three semesters) is compared with the traditional dichotomous logit approach. The pseudo R-square indicates the multinomial model fits the data better, but for practical reasons it has very poor predictive power. The multinomial model does yield interesting results for the impact of independent variables on retention: for example, increases in high school grade point average impacts the probability of a student being in the fall-spring outcome more than the probability of falling in the first fall semester only outcome.

Porter, S.R. (1999).  Assessing Transfer and Native Student Performance at Four-Year Institutions Paper presented at the Association of Institutional Research meeting, Seattle, WA.

Do transfer students perform poorly in comparison with native students? This paper answers the question through an analysis of transfer and native student performance in four areas: retention, graduation, grade-point average and academic dismissals. The emphasis is on making appropriate comparisons between the two groups, because differences in performance may be due to social factors such as integration difficulties in addition to academic factors such as poor preparation. Using returning students rather than new students and controlling for the number of credits earned results in groups of transfers and natives that are much more homogenous than the traditional cohorts of new transfers and natives, allowing for more appropriate performance comparisons. With this approach natives score better on all four measures, although the difference lessens when controlling for access to resources such as financial aid and demographics.

Porter, S.R. (1999).  The Robustness of the 'Graduation Rate Performance' Indicator Used in the U.S. News and World Report College Rankings Paper presented at the AIR-CASE Conference, Washington, DC.

This paper analyzes the robustness of the U.S. News graduation rate performance indicator, calculated as the difference between an institution’s actual graduation rate and their predicted graduation rate from a linear regression equation controlling for student aptitude and institutional expenditures. The sample is 198 of the 218 national universities used in their 1999 rankings. Robustness is examined in four areas: the effect of small changes in the sample due to missing data or changes in how the sample of national universities is defined; the effect of seemingly irrelevant changes in variable definition; the effect of different model specifications that take into account additional measures of student quality and institutional constraints; and how the use of confidence intervals for the predicted values changes conclusions about performance. Changes in the sample and variable definitions can cause the predicted graduation rate for an institution to fluctuate by plus or minus two percentage points. More refined model specifications reduce the number of institutions with extreme performance differences and can actually change an institution from under-performance to over-performance, or vice-versa. Finally, the use of confidence intervals for the predicted graduation rates reveals that only about 5% of the institutions in this study have a predicted graduation rate that significantly differs from their actual graduation rate. The implications of these findings for these types of models and recommendations for future research are discussed.

last updated: November 21, 2002