Month: September 2012

Trend Reporting in International Education: WES Survey Results Can Mislead

by: Anthony O’Donnell and Aleksandar Popovski

In its recent report, “Not All International Students Are the Same: Understanding Segments, Mapping Behavior” (“the WES Report”), World Education Services (WES) proposes a model for segmenting international applicant pools based on a student’s financial resources and levels of academic preparedness.

The WES Report [covered here on The PIE News] is based on survey data taken from 1,600 international students during their application for credential evaluation, one of the services offered by WES.

Although 2,500 students started the survey, 36% “dropped out when asked about their experience with agents,” resulting in the 1,600 results used for the report.  The previous two sentences hint at three common statistical biases briefly acknowledged by WES in the section “Data Limitations.” These biases make most of the claims in the report statistically invalid.

The first limitation is sampling bias.  Ideally, we would like to use reports such as the WES Report to guide substantive policy decisions. To do so, the means of gathering data must conform to the basic statistical requirement of random sampling. If we perform random sampling, the data we collect should be representative of the population we want to study, and should not over-represent, under-represent, or distort the “real” population in question.

WES does not randomly draw from the population of international students; it draws from its own clients. With the data collected, we cannot answer questions about prospective international applicants in general but only answer questions about prospective international applicants who also use WES services.  We cannot make reliable statistical inferences about the overall population of international students based on this data.

Another data limitation is self selection bias, where some respondents are more likely to take the survey than others.  Since the survey was conducted in English, only students with a working knowledge of English could have taken it.  Hence, the group that volunteered for the survey is not representative of all potential students.

Students with low English proficiency may have been excluded from the analysis because they could not complete the survey on language grounds. Incidentally, these students would need the most help to navigate the complicated process for applying to a U.S. university, and are most likely to utilize agent services.

The third limitation is that of missing data. With missing data, WES’ claim that one-sixth of respondents used agents cannot be a reason to conclude that “the use of agents might not be as widespread as previously indicated”. As noted above, 36%, or 900 respondents, dropped out of the survey when asked about their experience with agents. WES notes that those students may have dropped out because they “perceived agent-related questions to be sensitive.”

The only way for missing data to not be an issue is if only one-sixth of the 900 that dropped had used an agent. However, it is possible that anywhere from 0 to 900 used agents. Of the 2,500 original respondents, from 10% to 46% could have used agents.  This, coupled with the possible omission of weak English speakers from the survey (self-selection bias), casts serious doubt on WES’ claim that agents are sparingly used by international students.

Having addressed the statistical issues in the WES Report, we turn to the claim that 62% of agent users “are not fully prepared to tackle the academic challenges of an (sic) U.S. education.” This claim is only valid under random sampling. Without random sampling, we cannot conclude that there are proportionately more academically prepared students among non-agent users than among agent users. Missing data compounds this problem, because of the 900 students who did not answer the survey there may have been a large proportion of highly-qualified agent users. Considering the many problems posed by statistical biases, we should discard this claim.

Finally, we should be cautious of any study that discusses academic preparedness from an a priori perspective. Whether a student is academically prepared to tackle the academic challenges of a U.S. education may be more a function of the admissions/academic standards of universities and less a function of the academic quality of students. What is a good student for some universities may not be a good student for others. Given that U.S. HEI’s form a wide spectrum of institutions from community colleges to big research centers, students may find a place at U.S. HEI’s with huge variations in academic preparedness.

WES presents an interesting strategy for international market analysis. However, given the statistical deficiencies with the survey, the conclusions drawn are of limited value for purposes of policy making in the area of international recruitment.

Aleksandar Popovski (popovski@binghamton.edu) is Assistant Dean of Admissions and Recruitment for the Graduate School at Binghamton University. Anthony O’Donnell (anthony.odonnell@binghamton.edu) is a graduate assistant and data analyst at Binghamton University.

What Study Abroad in the UK Did for Me

In 2004, while pursuing a degree in Texas, I took advantage of my school’s helpful and well-run study abroad office to take the chance to go to England.

I applied to the University of East Anglia, and was not only accepted but given a small scholarship, a stipend of 500 pounds, which made it much more possible for me to take this leap, and also set a positive tone that made me feel exceptionally welcome.

To subsidize all the travel, I decided to pursue summer employment in London before my fall semester began. I enrolled in an exchange program called BUNAC that sets American students up with British work visas (and vice-versa).

For the first couple of weeks, my employment situation was grim. My friend and roommate who had gone ahead of me “found a great deal on rent,” subletting from a Chinese grad student living in what turned out to be the garage of a council flat, though we didn’t know it until we took down a poster, revealing one wall to be a pair of outward-opening garage doors. No wonder it was so drafty.

Meanwhile, we were working for a rather shady catering company that hired other transient workers from around the globe – terrible work, but we made some good friends. Still, I wanted more. Being a writing major, I found the name of a publisher in the BUNAC directory that had offered internships before, got interviewed, got the job (due to my knowledge of Latin – probably the first hire on that basis since the Middle Ages) and set to work finding out-of-print titles for their new classics line.

Once fall came around, I packed up my bags again and took the train to Norwich. I studied creative writing there – UEA is a hallowed place for that subject in the UK, with alumni like Ian McEwan and Kazuo Ishiguro, and I eagerly soaked up the atmosphere of artistic seriousness, along with many a Snakebite.

Which is not to say I didn’t work hard academically during that semester; my grades were transferred back as pass/fail, but I got the equivalent of all As, meaning I probably took classes more seriously than necessary, if anything. My papers were extremely well-received by tutors and professors, and I came away with the impression that humanities education in the UK was both less demanding and more enriching than in the US – much more emphasis on simply taking your own time to read and write.

If I could do it over again I’d spend slightly less at the pub, and travel more within the British Isles rather than blowing my scholarship money on Ryanair jaunts to the continent, but I have no real regrets. The highlight of my whole experience was probably cooking a turkey dinner to introduce my British associates to the glories of an American Thanksgiving. Which goes to show: the benefits of international study even extend to those who stay home, and meet weird people like me from faraway lands.

Integrating with the people of a host country can be hard, even if, as in the case of my transatlantic adventure, you’re “divided by a common language.” I won’t lie: I was often homesick. The recent Open Doors report from the Institute of International Education shows the health and vibrancy of study abroad programs, but surveys have shown problems with integration that reinforce the importance of building these bonds. So for those of you engaged in this field: be active in encouraging social assimilation, and don’t let any of us fall through the cracks! I want everyone to have as great an experience as I did.

An experienced writer on all things related to higher education and business, Amanda Watson spends her days covering the latest stories on various topics such as online mba rankings, web entrepreneurship, and social media marketing. You can contact Amanda at watsonamanda.48@gmail.com.