‘Indian students should work on oral communication skills’: ETS

(Graphics by Abhishek Mitra)

In the list of TOEFL top 5 countries with the highest number of test takers, India has climbed from the fifth rank in 2019 to second in 2022. In 2019, China ranked at the top, followed by the USA, Japan, South Korea and India respectively. The trend remained the same in 2020, but in 2021 India rose to the third spot after China and the US, and then to the second spot in 2022.

READ | Facing accommodation challenges in a foreign country? Here’s what a student can do

For GRE, in the list of GRE top 5 countries with the highest number of test takers, India was at the third spot in 2019 but has retained its second spot since 2020, with the US at the top. Cities such as Hyderabad, Mumbai, Bengaluru, Pune and Chennai had the majority test takers in 2019 and 2020. In 2021, Guntur and Vijayawada entered the top spots at second and fourth rank, in addition to Hyderabad at first, Mumbai at third and Pune at fifth. The trend remained the same in 2022.

Some excerpts…

What are some of the main changes brought in GRE and TOEFL by ETS?

Just over the last couple of months, we have introduced changes to both TOEFL and GRE, and these changes were driven mostly by the feedback we have received from the stakeholders. Our two major constraints are the institutions, the universities or colleges that accept these assessments, and then the test takers themselves. The stakeholders demand a valid, reliable and fair assessment. Students wish to have a test that is relatively straightforward and helps them showcase their skills.

ALSO READ | Students, experts question validity of IELTS, TOEFL, PTE

Also keeping in mind that test taking can be a stressful experience, we decided to shorten the test time for GRE and TOEFL. The time for TOEFL has been reduced to just under two hours now, from close to three hours. It’s one of the shortest assessments among the major assessments in the world.

TOEFL will now have a new, more modern ‘writing for an academic discussion’ task, which will replace the previous independent writing task. Also, the reading section has been shortened and all unscored test questions have also been removed. Even the registration process has been simplified. Test takers can create an account and register for an available TOEFL test date quickly.

TOEFL by ETS: Cities with highest test takers. (Graphic by Abhishek Mitra)

We have also been able to make sure that the test-taker experience from a registration perspective is smoother and more streamlined as well. We looked at some of the question types that were there and made sure that it is reflective of what is going to be the student experience because the student experience inside the university has also evolved.

Further, the test takers will be able to see their official score release date upon completion of the test, in addition to receiving real-time notification of changes to their score status.

Similarly, the test duration for the GRE was also reduced by nearly half. For GRE, we also streamlined operations so that the test takers can now get their results within eight to 10 business days. This has been done keeping in mind the feedback we got from the marketplace. With the advances in research methodologies, we can measure the same skills with a smaller number of questions.

What took ETS so long to bring in these changes?

In every organisation, there is a process that needs to be followed. At ETS, we collect a lot of data for internal assessment of our operations, and through that process, we noticed a few changes that we could bring in our tests.

Historically, in GRE and TOEFL, we used to have a break in between (which has now been removed). Also, there were unmarked questions that did not count towards the final score. Those questions were used as sample items for us to think about what kinds of items are working well. We have been doing such analysis of items in the past. What has changed in the past nine to 12 months is the emergence of Artificial Intelligence and generative AI, through which we can analyse a lot of data in a short period of time. It can help an assessment company in several ways such as generating questions because model languages have been trained through the items that we have been testing over the years.

ALSO READ | Ahmedabad, Hyderabad top English test-takers cities in India: Duolingo English Test

It also helps in scoring, and with advances in research methodologies, we are also able to reduce the number of questions. We have used AI to see that if earlier we were using 20 questions to assess a student in the speaking section, through AI we are now designing more complex questions that can assess that in 10 questions.

We were able to do that because all these years, we had been collecting data on what works, and what doesn’t work, and with the help of advanced research methodologies and technology, we were able to do that.

Since the number of questions and test time has been reduced, will the quality of these tests suffer?

Absolutely not. We are the world’s most permanent preeminent testing and measurement organisation, and our assessments are accepted by the most number of education institutions around the world. No institution is going to accept anything that doesn’t measure what we say and stand behind what we are saying. If a test measures quantitative ability, we make sure that it measures quantitative ability, and does so in a way that is free from most challenges around liability, fairness, biases, etc, that we try to minimise as much as possible.

There have been reported cases of cheating in the past, especially after Covid ETS has introduced ‘at-home’ test services. Keeping these security lapses in mind, what are the key points that ETS is adopting for TOEFL and GRE to improve security?

Just like any other sector, there is a small portion of bad actors in any industry, and this is not immune from that. However, the good news is that it is a very small portion. Having said that, we realise that even if it’s a small portion, there are big implications, both from a reputational standpoint and also from a standpoint of what we stand behind. When we say that somebody is good at a certain skill, we better be sure that it is that person and the skill that they are talking about.

To ensure this, the first thing we have done is that we have internally set up a department called the ‘office of testing integrity’. What that essentially means is that every exam that is conducted is different. Every exam that is conducted goes through a whole review process. Historically, these things used to be manual, due to which scores used to take time. Now, we are able to do it much faster, because we are applying technology. We are analysing how different item types performed, is it within the norm that we expect it to perform. If there is any big deviation from the norm, we look at whether the deviation is because of any cheating or irregularity. This is the job of the ‘office of testing integrity’.

We have also appointed a chief security officer, and we are probably one of the few (if not only) to do this. This is a testing security officer, not physical security. The job of this official and the team is to ensure that we continue to both investigate but also invest in path breaking technologies that can help us (in this area). The first aim is to completely prevent such incidents, and we do realise that such incidents can happen sometimes. In those cases, we want to ensure that our team is able to detect and remediate quickly. We will directly cancel the score, if it is found that somebody used any unfair means. We don’t want somebody with unfair means to take advantage because we work on behalf of 99 percent of the test takers who are honest test takers.

(Graphics by Abhishek Mitra)

Another thing is that we keep investing in advanced technologies such as facial recognition, voice recognition that can help us detect whether these things are happening or not. Especially when candidates take ‘at-home’ tests, we want to make sure that they don’t have any programmes running in the background, and there is nobody present physically or through a computer software to help.

Q. A lot of Indian candidates resort to coaching centres for TOEFL and GRE preparation. Do you think these competitive exams are encouraging coaching culture?

Rohit Sharma: It depends on where the person is coming from, in terms of educational background. TOEFL is used for both undergraduate and graduate level admissions, and the GRE is largely for graduate level studies. If someone has grown up in an English speaking environment, not just at school but also at home, then it is easier for students to prepare for TOEFL without coaching. TOEFL is about measuring your English language proficiency to be able to operate effectively in an all English speaking environment in an academic context. So for the students who may be going to an English medium school but may not be a part of a household that generally converses in English, then coaching centres might be useful for them.

The basic way of understanding this is that if you don’t practice anything, you will become rusty, be it math, English, Hindi, Punjabi, or anything else. There’s nothing wrong with taking that extra help as sometimes some people require more help than others.

Indians in top colleges | ‘Having a published research paper made my application stronger,’ says University College London student

Our study materials are largely designated to give potential test takers, both a flavour of the types of questions as well as make them familiar with what test taking experience could be like in the formal setting. Till now, we have been in the language assessment business, but we are now also working on providing targeted feedback on how candidates can improve. For example in the speaking section, we will use technology to give constructive feedback such as what the candidate do well, what area needs improvement such as ‘your pace was fine but pronunciation of these words was wrong’, and more. We want to be seen as a partner in candidates’ success.

How is ETS using Artificial Intelligence (AI) to improve content?

Rohit Sharma: We have so many students that take the assessments, we are starting to mine the data that can help us provide tailor made suggestions to candidates from around the world. For example, if a candidate comes to the TOEFL website and enters their background information, we will be able to suggest tips and areas to them such as ‘students from Indian origin usually do well in speaking, or practice more writing questions’ etc. That particular candidate may or may not fall in that category, but at least they know the general statistic/ pattern.

So we are actually leveraging AI to build those models that can be personalized. In this case, the suggestion of course lies around your ethnicity being an Indian, but over time, it could be so many things such as location, educational qualification, profession etc.

What are the skills that Indian students should improve?

Rohit Sharma: Indian candidates are usually very hardworking, analytical, and good with numbers and technology. I think one of the areas where we don’t rank as high, especially when you are in your formative years and early mid-teens to late teens, or early 20s is more around oral communication in a foreign language. Due to this, what happens is that when you are out in a situation for the first time where you need to converse in English completely, one can get shy or awkward. This is mainly with those students who do not grow up in an English-speaking household.

Having said that, I would also like to add that this is a natural thing as it would be the same scenario for someone from the West who learns Hindi in school but doesn’t speak it at home. While I grew up in India and studied English at school, I was a bit shy when I reached the US and had to converse in English all the time. It took me some time and I eventually got comfortable.

Parents should encourage their children to develop oral communication in a foreign language, practice it more in comfortable settings, try to be a part of some volunteer project where people are being assigned to lead a team and they have to communicate.

Practice is the key to success for anything, so it is important to practice, and also ensure that you benchmark yourself as to where you’re starting from so that you can see whether you’re improving or not.