Are Online Degrees Substitutes for TraditionalDegree Programs? Evidence from a

Correspondence Study

Conor Lennon∗

October 9, 2015

Preliminary and incomplete, please do not cite.

Abstract

This paper reports the findings of a correspondence study which examines

the impact of online degrees on labor market outcomes. The experimental

variation is not between selective and non-selective schools but in the medium

of instruction within major U.S. universities who have both online and tradi-

tional degree programs. In the preferred specification, callback rates for in-

dividuals who have a traditional degree were on average 16.05% after con-

trolling for covariates. In contrast, callback rates were only 4.46% for online

degree holders. These findings suggest labor market outcomes for online de-

gree holders will be significantly worse than traditional degree holders, all else

equal.

This research is important for understanding the future of college educa-

tion. It suggests the move to online only degrees is less predictable than sug-

gested by the research focused on learning outcomes in online settings.

JEL Codes: I26, I23, I28, J23, O33, M51

∗University of Pittsburgh; Email:cjl55@pitt.edu

1

1 Introduction

Allen and Seaman (2014) find that, in the Fall of 2012, 33.5% of all post-secondarystudents in the U.S. were taking at least one class online. In 2002, this figure wasjust 9.6%. Such striking growth begs the question: what is higher education go-ing to look like in 20 years time? Will all students be taking at least some classesonline? Will students still attend any scheduled, face-to-face classes as they donow?

The answer to these questions critically depends on whether or not onlineclasses are preferable to students. Curiously, the literature seems to be stuck onmeasuring how the technological changes in higher education are affecting stu-dent learning outcomes. Bowen et al. (2014), Figlio et al. (2013), Ary and Brune(2011), Bennett et al. (2007) and Hernandez-Julian and Peters (2012) are just someof the recent examples. Each considers how student learning is affected by themove to various types of technology-enhanced instruction and/or homework as-signments. The work finds that learning (measured in a variety of ways) is muchthe same regardless of the medium of instruction. These studies suggest onlineeducation can provide comparable learning at arguably a lower cost (see Bowen etal.). It is tempting to therefore conclude that the move towards a landscape withexclusively online college degrees is both inevitable and unavoidable.

What is missing from these studies is a consideration of the individual choicesthat will determine how third-level education is delivered. Learning outcomesare of crucial importance to university administrators and scholars of educationand pedagogy. Conscientious students may also be pleased to hear that learningoutcomes are not affected by taking classes online. However, many may be at leastmildly concerned about how taking classes online affects their outcomes in thelabor market. This paper is for them.

To try to answer the question of how students’ outcomes are affected by pur-suing a online degree program, this paper presents the findings of a resume auditor what is more commonly referred to as a “correspondence” study. A correspon-dence study, where fictional resumes are submitted for real jobs, is ideal for test-ing how employers view applicants online education. In a typical correspondencestudy, resumes which randomly vary a single piece of information are sent to em-ployers and callbacks for interview are tracked. Despite the effort these studies

2

require, the correspondence study method is popular as it tends to provide con-vincing evidence on how labor markets work and what determines differencesin individuals’ outcomes. Typical examples include Bertrand and Mullainathan(2004) who vary applicants’ names to indicate race/ethnicity, Neumark (1996) whoexamines the effect of gender, Kroft et al. (2013) tackle the effect of longer spells ofunemployment, Pager (2003) examines how having a criminal record effects call-backs, and Lahey (2008) considers the role of age. There are many more.

The correspondence study set up allows researchers to determine causal rela-tionships between individual characteristics and a single labor market outcome:how many interviews the individual receives. While callbacks are not a perfectmeasure of labor market success, the assumption underlying these studies is thatapplicants with more interviews will face shorter spells of unemployment andhigher wages.

The enduring popularity of these types of papers stems from the fact that theresearcher can truly randomize the characteristic of interest, rendering it orthogo-nal to other observed worker characteristics. In contrast, labor market survey datais tainted by selection, measured with error, struggles to account for the effects ofjoint labor supply decisions, and may not report important characteristics of inter-est.

Using a correspondence study to examine the impact of online education oncallback rates avoids those problems. Of particular importance is the fact thatlabor market surveys do not ask about the medium of instruction for surveyedindividuals. Even if they did, the decision to pursue online education is highlyendogenous and would render causal identification impossible. Creating fictionalresumes and then randomly assigning some to report online education providesclean identification of the effect on callbacks for interview.

The experimental set up, described in greater detail in section 3, gathers abank of resumes from online job search websites, anonymizes them by chang-ing dates, names, contact information, resume format and style, the applicant’saddress, cover letter, and previous work names and locations. Each resume isassigned a unique cover letter, a realistic but ultimately fictional address, alongwith their own email address and a contact phone number. The resumes are thenrandomly assigned to either convey that the degree was obtained via a specificUniversity’s “online” wing or from a traditional University.

3

Once the bank of resumes is created, they are used to apply to many real jobsacross a number of major U.S. cities. After the resumes are sent requests (calls oremails) for interview are tracked.1 The experimental variation is not designed tocompare selective traditional universities to for-profit schools like the Universityof Phoenix or DeVry. Instead the focus is on the effect of having an online degreefrom a well-known school that is completed online. Many large schools have asignificant online presence offering dozens of completely online degrees. Theseinclude Arizona State, Ohio State, Penn State, and Northeastern University.2

While reporting that having an online degree may seem contrived and a po-tential source of bias, a 2010 survey by the Society of Human Resource Managersfound that only 17% of Human Resource Professionals had never seen an applicantclearly indicate an online degree.3 In the five years since the survey, the number ofpeople completing online classes and degrees has risen considerably: the NationalCenter for Education Statistics reports that the per cent taking exclusively onlineprograms rose from 3.7% in 2007-2008 to 12.5% in 2012-2013.4

To date, the author has completed a pilot phase of almost 400 job applications.The data collected represents less than 20% of the intended total. The applicationswere completed using 15 unique resumes which were randomly assigned to reflectonline or traditional face-to-face mediums of education. On average, just over 25applications were completed for each resume.

There is variation in the total number of applications sent for each individual.The number of applications is essentially a function of the time since college grad-uation and is a reflection of the design of the study. As all of the resumes had topotentially represent someone who has a degree earned wholly online, the resumesrepresent individuals who have limited experience in their field. This lack of expe-rience means there are sometimes not enough advertised openings that meet thestudy’s selection criteria. The study’s method is explained in detail in section 3.

The variation in the number of jobs per resume was unavoidable in the shorttime-frame of the study’s pilot phase. To ensure enough jobs met the criteria, the

1Note that if a request for interview is received by the researcher the employer is contactedimmediately to thank them and politely decline the request.

2For example, Penn State Univeristy offers 24 completely online bachelor’s degree programsthat students can complete from anywhere in the world.

3See http://www.shrm.org/research/surveyfindings/articles/pages/hiringpracticesandattitudes.aspx.4See http://nces.ed.gov/pubs2014/2014023.pdf and https://nces.ed.gov/fastfacts/display.asp?id=80

for more details.

4

key characteristic that would need to be changed is work experience. However, itwould make no sense to create a resume which lists a degree earned online in 1995,or even 2005.5 In addition, all jobs had to require at least but no more than a bach-elor’s degree. These restrictions make finding suitable openings extremely time-consuming. In time, as more jobs become available underrepresented resumes willnot be an issue.

This study’s restrictions are in contrast to the breadth of options for applica-tions and resumes Bertrand and Mullainathan and others have. Interested in theeffect of black sounding names on callbacks, Bertrand and Mullainathan apply to1,300 unique jobs. They send four different resumes to each of these for a totalof almost 5,000 observations. The jobs they apply to are typically retail, clerical,sales management, or administrative positions. These jobs rarely require a degreeand Bertrand and Mullainathan are not limited to sending resumes which repre-sent younger applicants with a bachelor’s degree but with little experience in theirfield. In fact, Bertrand and Mullainathan ensure that many of their applicants areover-qualified, thus generating enough callbacks to obtain statistical power.

Despite the limited data, a statistically significant negative effect on callbacksis observed. While traditional degree holders are called back on average 16.05%of the time (after controlling for individual characteristics), online degree holdersare called back on average only 4.46% of the time. In other words, applicantswho listed traditional degrees were more than three times as likely to be calledfor interview. For context, the magnitude of this difference is approximately twicethe size of the largest effects seen in callback rates by race, gender, or age in othercorrespondence studies.

Section 2 reviews the literature on the relative merits of online education alongwith the benefits and drawbacks of using correspondence studies to understandthe labor market. It also relates this work to the literature on school reputationeffects. This paper is in some ways an examination of the within-school reputa-tion differences created by having brand-name universities offering in-person andonline degrees. Section 3 describes the correspondence study procedure. Section4 provides summary data and checks on the experimental randomization. Sec-tion 5 provides the main results, considers their robustness, and explains how the

5It would also not be realistic to compare the labor market outcomes of recent graduates withonline degrees to recent and experience graduates from 10 or 20 years ago.

5

remainder of this paper will be completed. Section 6 provides concluding remarks.

2 Literature

Lack (2013) provides an exhaustive literature review of the available researchon learning outcomes in online education through the end of 2012. The review de-tails large and small scale studies examining how students in accounting (Rich andDereshiwsky,2011), management (Daymount and Blau, 2008), economics (Bennettet al., 2007), engineering (Enriquez, 2010), sociology (Driscoll et al., 2012) and morefare in online settings.

The conclusions that can be drawn from these studies are complicated by dif-ferences in research method. In addition, many of the studies lack rigor or randomexperimental variation. They are also affected by subject attrition, treatment andcontrol group cross-contamination, small sample sizes, different populations of in-terest, along with each study having a unique institutional setting and time-frame.What the review highlights is that there is no concrete evidence that students, con-trolling for observable characteristics, learn less when the medium of instructionis altered.

Since Lack’s review a number of newer studies with larger samples sizes andtrue experimental variation have been published. One of these is Figlio et al.’s ex-amination of the effects of watching online rather than attending introductory eco-nomics lectures at a large selective research institution in the United States. Unlikethe majority of studies in Lack’s review, the authors find some mild support forthe hypothesis that learning outcomes are worse in online settings. In the study,students were randomly assigned to taking introductory microeconomics onlineversus in-person. The authors observe that regardless of sex or race, average testscores were higher for those who were assigned to the face-to-face instruction.However, the effects were modest and not always statistically different from zero.Figlio et al.’s work is not without limitations. In particular, their results rely onthe comparison between just 215 students assigned to online lectures and 97 as-signed to traditional classes. In addition, the online instruction they compare to isextremely primitive. The live lecture is recorded and the “treated” students watchit online. Many would argue this set up is not the relevant comparison. Studentswatching a recording of the live lecture with no other changes is perhaps not what

6

people mean when they refer to online classes.Bowen et al., in a larger-scale study, improve upon Figlio et al.’s work by allow-

ing for the online instruction to be augmented by a new interactive learning plat-form. They perform their experiment at six large universities, randomly assigningundergraduates into traditional and hybrid statistics classes. The hybrid classesmet once per week. Students accessed sophisticated machine-guided instructioninstead of a second weekly class meeting. Bowen et al. find that “students in thehybrid format are not harmed by this mode of instruction in terms of pass rates,final exam scores, and performance on a standardized assessment of statistical lit-eracy.”

While the research on learning outcomes in single classes suggests that it maybe a valid alternative to traditional in-person instruction, no study considers howa purely online degree program compares to a traditional degree. Indeed, if learn-ing outcomes are comparable in a single semester introductory class, that doesn’tnecessarily mean that an entire degree program can be effectively delivered online.To validly examine the outcomes from entirely online programs versus traditionaldegrees, experimental variation at the program level would be required. Of course,randomly assigning students into an entire degree program would be problematic.First, volunteer subjects would likely be quite a unique and small group. Secondly,students who were randomly assigned into online degrees may take other steps tomitigate its effects. This is aside from the fact that such a design, given the signif-icance of the intervention it entails, would be unlikely to obtain approval from anInstitutional Review Board.

What is unusual is that none of these authors discuss how labor market out-comes may be affected by the move towards online coursework. To be fair, inthe context of the early studies described by Lack and the more rigorous work ofFiglio et al. and Bowen et al., labor market outcomes would be impossible to con-vincingly relate to a change in the mode of instruction of a single 3-credit collegeclass. At the same time, if the conclusion drawn from these studies is that collegedegrees could be delivered mostly via technology-enhanced instruction in the fu-ture, it seems reasonable to wonder how students who pursue online degrees willbe treated in the labor market.

Rechlin and Kraiger (2012) appear to be the only authors who have consid-ered this issue. Their study “experimentally” examines the attitudes of employed

7

Industrial-Organization (I-O) Psychologists towards applicants who have com-pleted their degree online who do and do not have internship experience. Theirgoal was to inform the educational investment decisions of fellow I-O Psycholo-gists. The survey finds that “applicants with online degrees are viewed less favor-ably than are applicants with traditional degrees.” Rechlin and Kraiger’s method,however, is open to criticism. Firstly, they present estimates from just 23 surveyrespondents in a specialized profession. These respondents viewed just a one-sentence description of each hypothetical individual and their education. Respon-dents viewed a total of eight sentences, where each sentence varied the level of ed-ucation (M.A. vs Ph.D.), its delivery (online versus traditional), or whether or notthe individual had internship experience. The one sentence descriptor for an ap-plicant with an online education was typically “this applicant received his/her de-gree in I-O psychology from an online program and did not complete an internshipduring graduate school.” This was compared to “this applicant received his/herdegree in I-O psychology from a traditional terminal degree program and did notcomplete an internship during graduate school.”6 Given these descriptions, theindividual respondent is left to make an inference not just on the medium of edu-cation but also the institution itself. That is, if the respondent considers applicantswith online degrees likely to have graduated from less-selective and lower-rankedschools, then it is unclear if the survey respondent is reacting to the medium ofinstruction or the inferred institutional prestige.

Moreover, as all respondents viewed all eight potential descriptors and wereasked how likely they were to offer the hypothetical applicant a position, we can beconcerned that respondents figured out the purpose of the survey and that demandeffects could be driving the results.7

The lack of discussion and rigorous research on the impact of online educa-tion on labor market outcomes motivates this paper. The fact that information onthe medium of instruction is not recorded by labor market surveys means that acorrespondence study is likely the only way to causally relate the attitudes of realemployers towards potential employees to degrees that are earned online. Corre-spondence studies are a reliable solution when crucial information available to em-

6Note that these sentences are paraphrased.7That is, if respondents weren’t being implicitly asked to compare online and traditional pro-

grams for research purposes, they may have reacted differently.

8

ployers is not available to or cannot be controlled for by researchers. Bertrand andMullainathan provide the ideal example of the value and purpose of such stud-ies. The authors were interested in the perennial question of how race affects labormarket outcomes. One particular channel they are interested in is how employ-ers respond to names which indicate the applicant’s race. The assumption beingthat employers screen resumes using indicators for race which then causes wageand employment gaps to persist. Their paper’s title “Are Emily and Greg MoreEmployable than Lakisha and Jamal?” provides a clear illustration of their ap-proach. Crucially, labor market survey data cannot shed any light on this questionas respondents’ names are not reported. Bertrand and Mullainathan send almost5,000 fictional resumes for real jobs and find that white sounding names received50 percent more callbacks for interview, holding all else equal.

An even larger-scale correspondence study involving 12,000 applications byKroft et al. (2013) examines the role of unemployment duration in call backs forinterview. They find that fictional applicants who report being unemployed foreight months have 45% fewer callbacks compared to those who were unemployedfor just a single month. While labor market survey data could, in theory, be usedto examine this question, researchers would have to find extraordinarily creativeways to deal with issues of selection. A naive approach would regress the proba-bility of returning to employment in some time period on the duration of unem-ployment and observable characteristics. Such an approach would not providecausal estimates mainly because while applicants with longer and shorter spells ofunemployment may appear similar to researchers, they likely look quite differentto potential employers. Kroft et al.’s approach provides tight control and allowscausal relationships to be accurately determined. This paper uses a correspon-dence study to obtain the benefits of both Bertrand and Mullainathan’s and Kroftet al.’s studies. Labor market survey data does not provide the required detail onthe medium of instruction and even if it did, the choice to pursue an online degreeis likely endogenous and difficult to control for.

Lastly, while not directly discussing the benefits of college reputation, the pa-per could be seen as a contribution to that literature. Dale and Krueger (2014)provide a detailed analysis of the literature in that area. Dale and Krueger alsoupdate the analysis of an earlier paper and find that reputation, proxied by se-lectivity, positively affects labor market outcomes in naive regressions. However,

9

after controlling for selection into selective colleges, the effects fall dramatically,consistent with the existing literature on the topic. Research on this topic is basedon differences between schools, whereas this paper can be viewed as examining thewithin-school variation in reputation created by offering online degree programs.

2.1 Limits of Correspondence Studies

Correspondence studies are an excellent way to uncover the attitudes of em-ployers towards specific employee characteristics. However, a number of caveatsshould be considered. Firstly, callbacks for interviews do not pay bills, and it isnot definitively clear from these studies that fewer callbacks actually translates tolower wages and higher unemployment. Instead, the information transmitted toemployers via the resume may actually improve matching, reducing wasteful andunnecessary interviews that would not result in a job offer. In addition, the revela-tion of some less favored characteristic perhaps allows an applicant to gain accessto an interview they would not have otherwise earned that ultimately leads to agood job offer.

It is not unreasonable to suggest that some employers, due to their own per-sonal experiences and tastes are seeking black, female, older, or homosexual work-ers, or that some employers might take pity (for lack of a better word) on thosewith longer spells of unemployment as they themselves faced a similar spell ofunemployment.

Secondly, given employers review resumes so quickly, the experimental varia-tion may not be as strong as it seems in the study set up. If employers fail to noticethe experimental variation it would limit differences in call back rates betweenapplicants.

Thirdly, applying for jobs posted in newspapers and online is only one way tosecure employment. Social networks and connections, internships, and personalrecommendations may compensate for or exacerbate the effects seen in correspon-dence studies. However, it is not clear that an individual who fares poorly in acorrespondence study couldn’t improve their potential job opportunities via alter-native means.

Lastly, correspondence studies are generally designed to study the impact ofsome immutable characteristic (at least temporarily). While they often come close,

10

they rarely actually study what it is they intend to. Take Bertrand and Mullainathan’spaper as an example. Their paper claims to study the effect of having a Black-sounding name compared to an identical resume with a White-sounding name.Instead the paper studies the effect of having a Black-sounding name, reporting itwithout alteration (Jamal Jones could easily present himself on his resume as Jay orJ. Jones) and having a resume that does not truly reflect changes that an employermay expect to see with that alteration.

If resumes from otherwise similar whites and blacks are systematically differ-ent in the population, then an employer receiving a resume that is not like otherBlack resumes means that those differences are a part of the experimental varia-tion. In that case, the effect reported in the paper is the combined effect of havinga Black-sounding name but a resume that doesn’t seem like other resumes fromBlack people.

An example may clarify: suppose that Black people know their resumes areoften discarded based on their name, their neighborhood of residence, or listed ed-ucation (perhaps they went to a traditionally Black high school or college) and thata typical Black applicant takes action to counteract that by working harder on theircover letter, improving their resume’s content, clarity, and format, oversells or in-flates their experience and skills, perhaps obtains a PO Box in a different zip code,or takes some other action intended to counteract the effect of the pre-existing biasagainst them. Suppose also that employers know that Black people do this.

Then, even if names were ignored, a Black worker with similar listed experi-ence and education may receive fewer callbacks than a similar white worker asthe employer does not believe the resume is as accurate or truthful as a resumefrom a White person. Essentially, the same worker on paper is treated differentlynot because of their race but because the employer interprets resumes differentlybased on prior experience with other applicants. Adding to this concern, Bertrandand Mullainathan find that White applicants experience a much higher return toincreased resume quality. This suggests employers could be skeptical of the accu-racy of Black resumes.

Kroft et al.’s paper is subject to a similar critique. The authors are actually re-porting not just the effect of unemployment duration but the the combined effectof being unemployed and being foolhardy enough to not have a good (even ifsomewhat contrived or completely fabricated) explanation for the spell of unem-

11

ployment. Indeed, it is not possible to rule out if the findings of Kroft et al. areonly the effect of telling someone that you have been unemployed for a long time.Employers may be much more receptive to identical resumes that have a brief ex-planation of the reason for the gap in employment. While being little more thancheap talk, addressing the issue shows that the worker is aware that such gapsraise red flags with employers.

Kroft et al. report that of the resumes they found online 75% of the resumes forseemingly unemployed workers listed the month and date when the worker lastworked. Of these, Kroft et al. report that 95% do not explain why they have notworked since the listed date. They appear to take this as evidence that resumesthey submit listing an unexplained period of unemployment will not seem out ofthe ordinary. That is quite a leap.

In particular, it is not clear from the experimental description how the authorsdetermined if the worker was actually unemployed, had left the labor force, or wasnow employed and had simply failed to remove or update their resume recently.The authors also do not know if the publicly available resume on the online jobboard is really the same one they use when applying for jobs. Additionally, theauthors do not have access to the cover letters these workers used when applying.Applicants may have fully explained the reason for the period of unemploymentin those cover letters.

Unfortunately, the paper you are reading is subject to the same and to some ex-tent unavoidable criticism. The estimated effects reported later in the paper shouldbe accurately seen as the impact on callbacks for interview from having an onlinedegree and telling the employer. However, the consequences of this distinctionare less concerning compared to Kroft et al.’s paper. This is because individualscan fabricate a reason for their spell of unemployment in their application and atinterview that may satisfy an employer’s concerns. In contrast, an entire onlinefour-year college degree is not so easily side-stepped. Even if it is not mentionedin the resume, the issue will likely come up during an interview.8 What this meansis that while the effects of unemployment duration on callbacks and eventual em-ployment could be eliminated with a simple one sentence explanation, the effect

8Consider an applicant who lists work experience coincident with their college degree, perhapsin another state. Alternatively, the fact a degree was completed online likely will arise when thecandidate is asked to answer campus or location-specific questions.

12

of online degrees on labor market success is less avoidable.Alternatively, mimicking the concerns with Bertrand and Mullainathan’s ap-

proach, employers may be used to applicants with online degrees taking steps tocompensate for their perceived deficiency via improvements in other areas. Whilethe resumes may appear equivalent to the researcher, employers may have priorsthat vary for these kinds of candidates. This potential concern is considered insection 5.

These caveats, while relevant, do not completely invalidate the correspondencestudy method nor the causal relationships uncovered. Instead, they should beviewed as qualifications, adding a disclaimer that delineates what it is that is be-ing explained and how it might be interpreted differently under alternative cir-cumstances.

3 Experimental Procedure

The procedure to generate resumes does not vary considerably across corre-spondence studies. For authenticity, typically a bank of resumes is created fromreal resumes the researchers find online. These are anonymized so that the originalperson can no longer be identified. The resumes are then sent to real jobs adver-tised online or in newspapers that the resume is qualified for. Critically, before theresumes are sent, they are randomly assigned one of n possible variations in a char-acteristic of interest. The researcher then tracks callbacks for interview. Because therandomization is completely orthogonal, by construction, to the resume’s other re-ported characteristics, differences in callbacks between resumes can be interpretedas a causal relationship. The entire process is detailed in the rest of this section.

As a preliminary, the author had to decide to focus on a small number of po-tential careers/jobs. Other correspondence studies tend to focus only on clerical,retail, and administrative roles to ensure they can apply to many job openings withmultiple resumes. This study is focused on the difference in callbacks for thosewho have online versus traditional bachelor’s degree programs. As a result, cler-ical and retail jobs are not realistic choices. Instead, the author focused on severaljobs suitable for recently graduated degree holders in the business, engineering,and medical professions. This was purely for convenience as there are lots of jobsavailable in these fields. The usual disclaimer that the findings may not generalize

13

to other situations and professions applies.

3.1 Resume Generation

Having decided to focus on jobs in Business, Engineering, and Medicine, stepone was to search a major job hunting website for resumes of recent graduatesof degree programs in business administration, marketing, accounting, nursing,along with software, mechanical, or manufacturing engineering. First, 16 resumesrepresenting males who are recent graduates (obtained their BA/BS in 2012 orlater) were selected: four from each of the areas of interest. All the resumes hadto reflect someone who was currently employed in a job that closely matched theireducational background. That is, a Registered Nurse with a nursing degree hadto be working as a nurse, and a software engineer had to be currently workingin software development or some other information technology related position.Then, 16 female resumes were selected under the same criteria.

The resumes varied in almost every way one can imagine. The individualslived in a variety of locations, had different work experience, attended differentcolleges throughout the US, had various degree titles even within the same field,many listed internships or part-time employment in college, and some used objec-tive statements and listed “headline” keywords while others did not.

The reasons the resumes are limited to currently employed (and therefore hav-ing at least some experience) individuals is to allow a sufficient number of applica-tions to be completed. Openings suitable for recent graduates with no experienceexist but they are rare whereas those requiring one year of experience are relativelyplentiful.

Once the set of 32 resumes were created, they were anonymized by alteringspecifics such as names, state of residence, dates and places of employment, col-lege attended, graduation dates, listed GPAs, and other characteristics. All of thesechanges carefully preserved the overall quality of the resume. The perceived gen-der, the exact title of the degree held, the reported current job title, and overallyears of experience reported on the resume was not altered. The author was care-ful to only report that a student attended a college or university where the degreeobtained was also offered in a completely online format, open to anyone in theUnited States and often beyond. This meant that when the resumes were random-

14

ized they would still reflect an actual real-world option.More high-profile schools than one might first imagine offer these completely

online degree programs. As just one example, Penn State University offers 24 de-gree programs that can be completed online from anywhere in the world. Inter-estingly, their FAQ page emphasizes that the transcript will not be any different tothe transcript of those who completed their degree on-campus.

The names assigned to individuals were used to create appropriate email ad-dresses (generally: first name, middle initial, last name “at” the internet domain,or a slight variation if that was not available).9 These email addresses were subse-quently used to create virtual phone numbers with associated voice-mail services.The outgoing voice-mail message was left as the computerized greeting as is rel-atively common today. The message an employer heard was the same regardlessof resume received and only differed in the individual’s phone number. It said theperson was not available and asked the caller to leave a message after the tone.

Lastly, after this process was complete, the details of each resume (name, loca-tion, years of experience, gender, US News College Rank, listed GPA) were inputinto a spreadsheet. A simple random number between zero and one was assignedto each resume. The resumes were sorted by this number in increasing order. Thefirst 16 in ascending order were assigned to have earned their education online.The remaining resumes were not altered to reflect an online education. The effec-tiveness of this randomization is discussed in section 4.

The fact that a college degree had been obtained online was conveyed to poten-tial employers by just one word following the name of the university or college theindividual graduated from. On a resume this would look like “[Name of Univer-sity or College] - Online.” That is the only difference potential employers wouldsee on the resume. As mentioned earlier, while this may seem a little contrived,in 2010 just 17% of human resource professionals reported never seeing someonestate their degree was earned online.

9As the paper is not focused on racial gaps all of the names would be traditionally associatedwith Caucasian males and females in the US.

15

3.2 Cover Letter Generation

Applications for positions almost always request a cover letter. For each re-sume, the author created a cover letter which varied in content but not organiza-tion or intent. All cover letters contained exactly four paragraphs. The first ex-pressed interest in the advertised position. The second explained the candidate’scurrent role, responsibilities, length of tenure, and expressed a desire to utilize andimprove their skills in a new position. The third paragraph explained why thecandidate would be a good fit for the position. This paragraph was not tailored inany way. Instead, it reminded the reader of the candidate’s education along withtheir excellent technical, analytical, communication, and soft skills as relevant tothe field.10 The final paragraph thanked the employer for their consideration, re-iterated the candidate’s interest in the position, and expressed a desire to discussthe candidate’s suitability for the position at interview. As each of these letters fol-lowed the same structure but varied specifics to match the details of the resume,their effect cannot be separated from a resume-level fixed effect.

3.3 Responding to Jobs

The author dealt with each resume in a systematic order. Using the randomnumbers generated by the computer to assign online and traditional degree statusto each resume, the author used each resume in the order provided by the rank-ing. A coin toss decided that the first resume used would be a traditional resumeand the author would switch between each resume in turn. The author took thecurrent job title listed for that resume (the real job title of the original person theresume was based upon) and searched for jobs with exactly the same job title ad-vertised in the previous two business days. This timeframe was used in order toensure resumes were not submitted long after the firm had received many suitableapplicants. The fictitious resume and associated cover letter was then used to ap-ply to the jobs the search returned. Jobs for which the particular candidate was notminimally qualified were ignored.

In addition, to avoid bias, only jobs which asked for information explicitlyavailable in the existing cover letter and resume were selected. This resulted in

10For example, all nurse cover letters suggested the candidate was kind, caring, considerate, etc.However, software engineers were adept technically and analytically.

16

many abandoned applications as many job application systems require much morethan a resume and cover letter to be submitted. Unfortunately, it is rarely clearwhat will be asked when beginning a job application. Applications often appearto request just a resume and cover letter be uploaded (or the information to bepasted into a firm-specific format) but upon clicking “submit” bring the applicantto another page of questions which often include personality tests and short essaysspecific to the firm, location, industry, or background of the potential applicant. Toavoid the potential for bias from such essays and tests, these applications wereabandoned. In all, around 40% of the job applications the author began had to beabandoned for this reason.

Lastly, no more than 50 jobs were applied to for any one resume as part ofthe pilot phase. More will be completed in the future. There were rarely enoughadvertised jobs to meet the fifty-job pilot phase ceiling. This is because jobs had tobe advertised within two business days, the information the employer requestedhad to be fully contained in only the resume and cover letter, the person’s currentjob title had to be matched exactly, and many applications were not possible as theindividual did not have the required minimal experience.

3.4 Callback monitoring

Only calls and emails requesting an interview were recorded. This is quite easyas each resume has its own email address and an assigned virtual phone number.While each resume reports a postal address the address is entirely fictitious (al-though it appears realistic) and thus any contact via postal mail would be missed.Bertrand and Mullainathan were concerned about this and contacted several hu-man resources managers who suggested this method of interview request was ex-tremely rare. Given that Bertrand and Mullainathan’s experiment was undertakenmore than 12 years ago we can be confident that the potential for bias introducedby requests for interview via postal mail can be ignored.

17

4 Estimation and Data Overview

4.1 Estimation

The study’s design originally intended to send each of the 32 resumes to a totalof 100 jobs. Sixteen of those would convey that the applicant has an online degreeand the other sixteen would convey a traditional face-to-face college education. Asthe assignment to an online degree is done at random, the estimate of δ in an esti-mation of the following form would provide causal estimates of the difference incallback probability between applicants who have online versus traditional degreeprograms:

yi,k = βXi + δDi + εi

In the estimation, yi,k takes on the value of 1 if firm k calls the applicant forinterview. This binary outcome is predicted by person-specific characteristics Xi

and a dummy Di to represent the result of the randomization. In this paper Xi

includes GPA, years of relevant experience, a measure of college selectivity, gender,and a binary indicator for the industry the person is in (business, engineering, ornursing). The dummy Di takes on the value of 1 if the randomization selects thatindividual as having an online degree.

A negative estimate for δ would suggest the likelihood of getting a callback isreduced for online degree holders, even after accounting for other factors. Suchan empirical approach would not be feasible using labor market survey data dueto concerns about endogeneity and omitted variable bias. These concerns cannotbe driving the results seen later in this paper as the randomization of Di sidestepsthat problem by construction.

The approach in this paper is subtly different to the matched-pairs approachof Bertrand and Mullainathan and others. However, it is similar to the Kroft etal. approach in the sense that they use unique resumes which are then assigneda randomized unemployment duration. Four resumes are created, using simplerules for their research assistants to follow, for each MSA-job pair. The actual con-tents of each resume are drawn from a pool of over 1,200 real resumes. There isthe potential that an identical resume was created by the same or different researchassistants and sent to more than one job but the authors do not track this. Instead,they have their assistants record the objective facts of the resumes and note the

18

duration of unemployment it was assigned. Then, the assistant moves on to thenext MSA-job pair, creating another four unique resumes. This labor intensive ap-proach provides rich data but is not feasible for most researchers.11

In contrast, the matched-pairs approach is to create two (or more) versions ofeach resume and then examine the difference in callback rates as a function of thevaried characteristic. Such an approach completely avoids concerns that resultscould be driven by systematic differences between the resumes which are sepa-rated into groups by randomization. For large enough samples true randomiza-tion ensures the estimated δ would be the same but the empirical interpretation isdifferent in a minor way. This paper’s set up allows δ to be interpreted as the dif-ference in the mean callback rate between the group of people randomly assignedto have an online degree rather than a traditional degree. Formally, this means ifthe group of people assigned to online is group O and those assigned to traditionaleducation are group T then

δ = yi∈O − yj∈T

Where yi∈O = Ei∈O[yDi=1] and yj∈T = Ej∈T[yDj=0] and i 6= j. This approachcomputes the mean of all callback rates, conditional on other characteristics, of allthe members of each group. The approach allows for the possibility that differ-ences between the groups could drive the effects if the randomization fails in someway.

If the study was set up as Bertrand and Mullainathan, δ would instead be seenas the average of each difference in callback likelihood for each individual as theyswitch from an online to a traditional degree.

δ = Ei∈I [yDi=1 − yDi=0]

This set up essentially computes two predicted callback rates for each individ-ual as a function of the varied characteristic and then the estimated coefficient rep-resents the average difference across individuals. The two approaches are closelyrelated but will only be empirically identical if the randomization is perfect, leav-ing all covariates completely orthogonal to the randomization, which is only pos-sible asymptotically.

Ultimately, the benefits of the matched pairs approach are clear. First, it is not

11Kroft et al. thank a total of 14 research assistants.

19

subject to any concerns about the randomization. Secondly, the matched pairsapproach significantly economizes on researcher effort. This is because instead ofcreating N resumes, associated email addresses, phone numbers, contact details,and cover letters, and then randomizing across them, the researcher can create anequivalent sample size by merely creating N/M resumes and having M versionsof each resume, where M = 2 is typical.

The only downsides of the matched-pairs approach is that caution must betaken to never send both versions of the resume to the same employer. Mainlythis is to avoid raising suspicion and undermining the outcome of the study. Inaddition, sending multiple different resumes to the same employer creates the po-tential for equilibrium effects with one of the submitted resumes crowding outanother, potentially in a biased manner unless care is taken. The administrativeand clerical burden with these studies is significant for the researcher and havingto check each application has not been sent before would add further steps, andthe potential for bias-inducing error, into the process.12

The study’s current set up was chosen as it would still allow for the study touse a matched-pairs design if the pilot phase dictated it. However, the oppositeapproach - switching from a matched pairs design to the randomization approach- would require some data to be thrown out. The author initially thought thatthe administrative burden and potential errors associated with managing multipleversions of the same resume exceeded the cost of preparing N rather than N/2 re-sumes and identities. This turns out not to be the case. The marginal cost of addingor removing the word “online” and sending more applications - while being care-ful not to send the same resume to the same employer twice - is much smallerthan creating a unique new resume, email address, cover letter, and phone num-ber. That now seems obvious and illustrates the value of flexible research designsand pilot phases.

4.2 Description of the Pilot Phase Data

While a bank of 32 unique resumes was created, the pilot phase uses just 15 ofthese. The pilot phase also had a hard ceiling of 50 jobs per resume. As detailed

12Even if dates are restricted when searching for available openings, advertisements are oftenre-posted a few days later.

20

in section 3, it was often the case that not enough jobs matched the search criteriathat had not been applied to with another resume in the same field. At the time ofwriting, the author had used these 15 resumes to apply to an average of around 25jobs per resume for a total of 380.

This version of this paper reports only the callback rates for applications thathad been sent at least 10 business days ago to ensure adequate time is allowed forcallbacks. Almost 100 more resumes have already been sent to employers but theyare too recent to include in this version of the paper. As many were sent in the past3-4 business days callback rates are currently very low.

In total, 8.9% of all applications received callbacks. While that may seem low,it is higher than many correspondence studies. For example, Kroft et al. have anoverall response rate of just 4.7%, while Bertrand and Mullainathan have an essen-tially comparable callback rate of 8.05%. The higher overall call back rate makessome sense as the study does not send more than one resume to the same employerminimizing crowd out effects. Also, the resumes are extremely well-matched tothe jobs. The applicant always has a degree in the field, a GPA greater than 3.0, iscurrently working in exactly the same job title, and meets all the minimum require-ments for the posted opening. Table 1 in the next sub-section provides summarystatistics. These statistics explain how the randomization actually fared and guidesthe selection of a preferred empirical specification in section 5.

4.3 Did the Randomization Work?

As mentioned in section 3, the assignment of an online or traditional degreeto a resume was determined by random numbers generated in a spreadsheet pro-gram. Table 1 highlights the summary statistics for the main dependent variableand potential important covariates both overall and by the randomization.13

In the table, we can see that callbacks for resumes with traditional degrees av-erage at 11.8% whereas online degrees are at 5.9%. These are raw differences thatdo not account for any differences in individual characteristics among the resumesin each group. Notice that the randomization did leave some differences. Whilethe proportion male and female and the GPAs of the two groups seems balanced,the groups assigned to having a traditional degree tend to be from less selective

13Of course, the callback rates are after the randomization.

21

schools but have on average more work experience. Note that the reported figurefor selectivity represents an average acceptance rate of schools. Higher numbersindicate less selectivity.

Table 1: Summary Statistics for All, Online, and Traditional Types of Degrees

The table presents group counts along with summary statistics for Callbacks, Gender, GPA, CollegeSelectivity as measured by US News’ Reports of Acceptance Rates, and Years of Experience.

Overall, it appears the randomization was successful. As the study design willpivot towards the same matched-pairs approach of other correspondence studies,this table will not be required in the next version of this paper as the group charac-teristics will be the same by construction. For now, it remains important to controlfor these differences to ensure the observed callback rates are not being driven bydifferences in group characteristics.

22

5 Empirical Findings

As detailed in section 4, the paper estimates the difference between callbackrates for two groups: those whose resumes convey that they have an online degreeand those with a traditional face-to-face degree. The estimating equation is of theform yi,k = βXi + δDi + εi. The specifics of this are described at the beginning ofsection 4. To aid intuition, the empirical approach is equivalent to a difference-in-difference estimation where the randomization of the treatment ensures that thepre-existing difference between treatment and control groups is zero by construc-tion.

The correspondence study set up, with multiple observations for a number ofindividuals, results in data that forms an unconventional panel data-set. It is onethat has repeated observations for each individual but no time component. Asthe data can be considered as a panel a fixed or random effects specification canbe estimated. However, typically, if we have reason to believe that differencesacross entities have some influence on the dependent variable then a random effectsmodel should be used.14 Random effects estimations assume that the error term isnot correlated with the independent variables. This allows for values that are fixedacross all observations for each individual to play a role as explanatory variables.Random effects specifications typically cause concerns about omitted variable biasbut that is less of a concern here as there are no “missing” variables.

However, upon estimating a random effects model a Breusch-Pagan LagrangeMultiplier (LM) test is typically completed. This test is used to see if treating thedata as a panel is worthwhile. In particular, the null hypothesis in the LM testis that variances across entities is zero. That is, there is no significant differenceacross units - no “panel effect.” Based upon the data from the pilot phase, theBreusch-Pagan null hypothesis cannot be rejected. The panel-probit random ef-fects estimations (reflecting similar findings to Table 2 as suggested by the test)and the LM test statistics are reported in the Appendix.

The LM test results suggests that treating the data as cross-sectional, using OLS,and clustering standard errors at the person ID level would be appropriate. Table2 reports the empirical estimates of the coefficient of interest using various OLS

14See the wonderful notes of Princeton’s Oscar Torres-Reyna athttp://www.princeton.edu/~otorres/Panel101.pdf.

23

Probit specifications. The outcome of interest is whether or not a callback was re-ceived and a probit specification is ideal as the outcome takes on only the valueszero and one. The table adds controls with each estimation. The preferred specifi-cation in column 5 includes controls for sex, experience, college rank, career, andGPA, all factors likely to effect callback rates.

Table 2: Probit Estimates using OLS - Callback Rates

Standard errors are clustered at the Person/Resume ID level.

While the effect of an online degree is negative in all specifications, it is not untilschool selectivity is added that the effect becomes significantly different from zero.As one of the issues with the randomization was that selectivity was higher forthe schools of those taking online degrees, it is reasonable to observe that control-ling for this separately increases the statistical significance of the effect of onlineeducation.

Table 2 reports raw probit estimates without intuitive interpretations. Table 3reports the post-estimation marginal effects from the preferred specification (Col-umn 5). The table suggests that after all covariates are considered, the probabilityof receiving a callback is about 4.46% for someone with an online degree. In con-trast, it is over 3.5 times higher at 16.05% for someone with a traditional degree.15

15Note that the marginal effects are provided to aid intuition. There are well-known issues

24

To give these figures some context Bertrand and Mullainathan found Whites were1.5 times more likely to receive a callback for interview as Blacks, and Kroft et al.found that someone who is just one month unemployed is about 1.8 times morelikely to receive a callback for interview relative to someone who has been unem-ployed for eight months.

Overall, after controlling for covariates the data strongly suggest that callbacksfor interview vary considerably for traditional versus online degrees.

Table 3: Marginal Effects on Probability of Callback of reporting Online Degree

Table reports marginal effects of Online Degree. These estimates should be interpreted with cautionas the margins command in Stata struggles to account well for factor variables in the computationof marginal effects.

5.1 A Clear Causal Relationship?

As mentioned in section 2, one potential concern with audit studies is that re-sults are driven not by the experimental variation but by how the resumes compareto their subjective competition. This was illustrated by showing Bertrand and Mul-lainathan’s findings could be the result of changing an applicants name to signalrace but then changing nothing else. If identical Black and White resumes are inter-preted differently, perhaps because employers have learned that one of the groups

with computing marginal effects in Stata when the estimation involves a large number of dummyvariables. In essence, the problem is that the effects crudely consider the effect on the average ofvariables that have no average interpretation such as “gender”. In the data male is 1 and femaleis zero and the estimation reports the difference between 0 to 1, female to male. The marginaleffects set the value of gender to its average value in the data, which is a meaningless number. Thecommand does the same for the dummy variables for career.

25

tends to oversell or fabricate their experience and skills more than another, thenthe effects Bertrand and Mullainathan find could be due to the equilibrium effectscaused by other resumes.

That is, it may be that employers don’t callback Blacks because they expecta Black applicant to actually overstate their abilities in order to combat expectedracism that may not actually exist. The signal of ability the employer takes fromeach resume is then different not because the employer is racist but because ofthe actions of applicants that the researcher cannot control. As Bertrand and Mul-lainathan find that employers respond only slightly more to higher-quality Blackresumes, this “employer skepticism” explanation is a potential concern with theirfindings.

A similar concern arises in this paper. An employer may be used to seeing aperson who has a degree earned online having other compensating characteristics.When they do not see this, they infer something about the candidate’s ability thatthe researcher is not able to control. This problem is caused by holding all elseequal when the changes made should not result in all else remaining equal. Inparticular, mimicking Bertrand and Mullainathan, this concern would appear as alower return to resume quality (as measured by years of reported experience) forresumes listing an online degree. The logic is that experience and education aresubstitutes. If employers expect to see online degree holders having compensatingcharacteristics such as more experience, then their callback rates as a function ofexperience will be lower.

Examining this concern requires a difference-in-difference approach. Table 4reports the estimates from the following estimation;

yi,k = βXi + λDi + γExpi + δDi ∗ Expi + εi

where Expi is the difference between the actual and required experience for thejob-applicant pair, and all else is as described in section 4.

The table only reports the findings using the preferred specification from Table2. Estimates using alternate specifications are similar. The coefficient of interest isthe interaction between having an online degree (Di = 1) and more experience.Similarly to Bertrand and Mullainathan, a negative estimate for δ suggests thereis a concern that the online degree resumes are viewed in a categorically differentmanner. In a sense, the firms are reacting to more than just the study’s experimen-

26

tal variation and this may be because their experience with online degree holderschanges how they view the rest of the resume.

Table 4: Probit Diff-in-Diff Estimates using OLS - Callback Rates (by Experience)

The table only reports the findings using the preferred specification from Table 2. Estimates usingalternate specifications are similar.

The negative estimate on the diff-in-diff interaction term suggests employersare responding less to additional years of experience for online degree holders.This raises a concern - but does not immediately imply - that the findings in Table2 could be caused by more than just the simple experimental variation. If this wasthe case, the results are being driven not by having an online degree, but by havingan online degree and a resume that does not have what others with online degreestend to have, such as greater experience or other compensating characteristics. Un-fortunately, this is a limitation of correspondence studies. One way to control forthis effect would be to use difference-in-differences. A researcher would need ac-cess to many typical resumes for those with and without online degrees. Theycould then use those real resumes and compare their callback rates to the experi-

27

mental resumes within each group to gain some control for the way the two typesof resumes are interpreted by employers. However, gaining access to the actual re-sumes of a typical job applicant would be challenging. It would be tempting to justuse the resumes posted online that listed online degrees, but the resumes postedonline are likely not representative of the actual distribution of applicants for jobs.

Without a solution to this potential concern, the effects reported in this papershould properly be viewed as the effect on callbacks of both having an online ed-ucation and doing nothing to your resume and background to mitigate the effectsthat it may cause in the labor market.

The estimates suggest that callback rates will be lower for those who pursuean online education but who do not equip themselves with other compensatingcharacteristics to counteract the effect of that decision.

5.2 Next Steps

With a successful pilot completed, the next step is to continue data collectionin order to complete the paper. As mentioned in section 4 it seems best to pursuethe typical matched-pairs strategy, both for experimental validity and for the sakeof economizing on scarce resources. This is easily done by continuing to sendapplications for each of the 32 resumes already created. Once sufficient time haspassed, the randomization can be reversed and the altered resumes used again toapply for another 50 jobs each (or more if necessary). This will create a total of morethan 3,000 observations. As the estimated effect is already large and statisticallysignificant, the purpose of this exercise is to ensure that the results are not drivenby variation across groups that remains even after the randomization.

6 Conclusion

To date, studies of the potential move to online education have focused on is-sues of learning outcomes. Under the assumption that costs are lower, it appearsthe literature has decided that if learning outcomes can be shown to be at least asgood, then online education will become the norm. To these authors, the future ofhigher education is a function purely of production costs and learning outcomes.There is no space at the table for emergent and endogenous market forces to work

28

their usual magic.By way of contrast, this paper considers that the future of higher education

may be determined via an interaction between producers (universities and col-leges) and consumers (students and perhaps their families). To do so, the paperassumes labor market outcomes matter to students and that how they are affectedby changes in the medium of instruction is a critical determinant of the future ofhigher education.

To examine the relationship between online degrees and labor market out-comes, the paper presents the findings of a correspondence study. It comparesone metric of labor market success, callbacks for interview, for individuals withonline degrees relative to callbacks for those with traditional degrees. Despite arelatively small data set, the empirical estimates suggest that employers are skep-tical of online degree programs. The reduction in callbacks is more than twice thesize of the gap in callbacks found by authors who examined the effects of race andunemployment duration. The estimates suggest the movement of college educa-tion to coffee shops and parents’ basements is by no means a foregone conclusion.

An important caveat is that the estimates suggest that callback rates will belower for those who pursue an online education and do not take any steps to coun-teract that decision. At the same time, this still implies employers are not yet readyto consider individuals with online degrees as being as attractive as those with tra-ditional degrees. Indeed, they may never do so.

An additional but somewhat moot caveat is that it is not clear what aspect ofa traditional college education employers are responding favorably to. They maybelieve human capital formation is diminished in online programs relative to tra-ditional degrees, they may believe the individual will be less socially adept, areinferring some socioeconomic characteristics, or they may feel a traditional collegeeducation gives students something more than just grades written on a piece ofpaper. While understanding why students with online degrees fare worse is im-portant, it is not the main focus of this paper. This paper is agnostic about why andonly cares about if labor market outcomes are affected by how a degree was earned.Until labor market outcomes are comparable, demand for traditional face-to-facelearning from a professor in a classroom setting on existing college campuses willlikely remain quite high. Moreover, allowing for an even broader conception ofhow market forces might play out, a reduction in the cost of completing some

29

introductory college classes may simply increase demand for more advanced orgraduate-level classes which may be less amenable to routinized online instruc-tion.

Ultimately, the interaction between producers and consumers of college educa-tion, and not learning outcomes or administrative costs alone, will determine howcollege education will be delivered in the future.

30

References

Allen, I. E. and Seaman, J. (2014). Grade Change: Tracking Online Education in theUnited States. Technical report, Babson Survey Research Group.

Ary, E. J. and Brune, C. W. (2011). A comparison of student learning outcomesin traditional and online personal finance courses. MERLOT Journal of OnlineLearning and Teaching, 7(4):465–474.

Bennett, D. S., Padgham, G. L., McCarty, C. S., and Carter, M. S. (2007). Teachingprinciples of economics: Internet vs. traditional classroom instruction. Journal ofEconomics and Economic Education Research, 8(1):21–31.

Bertrand, M., Duflo, E., and Mullainathan, S. (2004). How much should we trustdifferences-in-differences estimates? Quarterly Journal of Economics, 119(1):249–275.

Bowen, W. G., Chingos, M. M., Lack, K. A., and Nygren, T. I. (2014). Interactivelearning online at public universities: Evidence from a six-campus randomizedtrial. Journal of Policy Analysis and Management, 33(1):94–111.

Dale, S. B. and Krueger, A. B. (2014). Estimating the effects of college characteristicsover the career using administrative earnings data. Journal of Human Resources,49(2):323–358.

Daymount, T. and Blau, G. (2008). Student performance in online and traditionalsections of an undergraduate management course. Journal of Behavioral and Ap-plied Management, 9(3):275–294.

Driscoll, A., Jicha, K., Hunt, A. N., Tichavsky, L., and Thompson., G. (2012). Canonline courses deliver in-class results? a comparison of student performanceand satisfaction in an online versus a face-to-face introductory sociology course.Teaching Sociology, 40(4):312–331.

Enriquez, A. (2010). Assessing the effectiveness of synchronous content delivery inan online introductory circuits analysis course. Proceedings of the annual conferenceof the American Society for Engineering Education.

31

Figlio, D., Rush, M., and Yin, L. (2013). Is it live or is it internet? Experimentalestimates of the effects of online instruction on student learning. Journal of LaborEconomics, 31(4):763–784.

Hernandez-Julian, R. and Peters, C. (2012). Does the medium matter? online ver-sus paper coursework. Southern Economic Journal, 78(4):1333–1345.

Kroft, K., Lange, F., and Notowidigdo, M. J. (2013). Duration dependence andlabor market conditions: Evidence from a field experiment. The Quarterly Journalof Economics, 128(3):1123–1167.

Lack, K. A. (2013). Current status of research on online learning in postsecondaryeducation. Technical report, ITHAKA S+R.

Lahey, J. N. (2008). Age, women, and hiring: An experimental study. The Journal ofHuman Resources, 43(1):30–56.

Neumark, D. (1996). Sex discrimination in restaurant hiring: An audit study. Quar-terly Journal of Economics, 111(3):915–941.

Pager, D. (2003). The mark of a criminal record. American Journal of Sociology,108(5):937–975.

Rechlin, A. M. and Kraiger, K. (2012). The effect of degree characteristics on hir-ing outcomes for i-o psychologists. The Industrial-Organizational Psychologist,49(4):37–47.

Rich, A. J. and Dereshiwsky, M. I. (2011). Assessing the comparative effectivenessof teaching undergraduate intermediate accounting in the online classroom for-mat. Journal of College Teaching & Learning.

32

Appendix

As mentioned in section 5, the data could be considered as a panel dataset withthe person ID as the group variable indicator. Repeating the analysis of Table 2using a random effects panel data probit specification, gives the estimates in Table5. Using the preferred specification in column five as the basis for the test, theBreusch-Pagan Lagrange Multiplier test statistic does not recommend rejecting thenull of no panel effect (χ2 = .13 with a p-value of .36).

Failure to reject suggests that treating the data as cross-sectional and using OLSis the most appropriate estimation approach. This is visually suggested by thesimilarity between the point estimates from the panel probit and the estimatesfrom the OLS probit with clustered standard errors. We can see in the table belowthat the panel probit specification has slightly larger standard errors but the pointestimates are essentially the same.

Table 5: Probit Estimates using OLS - Callback Rates

33