The first fully-automated speaking assessment with immediate score delivery
We leverage innovative research in language assessment methodology, advanced speech analytics, and artificial intelligence to produce today’s most accurate, efficient and economical solution
The TrueNorth Speaking Assessment has 3 sections that collect 3 distinct types of data. It begins with a language background survey where we collect information such as test-taker’s overall language experience and her previous exposure to the language being assessed. Part 1 of the assessment utilizes a process called elicited imitation, which is a fancy way to say listen and repeat. Elicited imitation is a research-proven method to efficiently get a strong and reliable prediction of oral language proficiency. It measures how efficiently the test-taker’s brain can process or “chunk” language information. Higher proficiency in the language enables a test-taker to process longer and more complex sentences. The video below provides a brief explanation of this process.
The second part of the assessment follows a question-answer format, collecting spontaneous responses from the test-taker. Because part 1 can be rated automatically, we use the part 1 data to determine the difficulty level for the questions asked in part 2. For each question, test-takers have 30 seconds to read the prompt and prepare to respond, and then up to 60 seconds to respond. After part 2, the system analyzes all of the data to determine and present a score for the test-taker. We provide a numerical score on the TrueNorth scale, and also provide score predictions for the ACTFL, CEFR, and other relevant scales.
The TrueNorth scoring process utilizes speech recognition and natural language processing, advanced psychometrics, and machine learning. Each test-taker audio response is processed through a previously-calibrated speech recognition engine. Working with speech recognition partners such as Carnegie Speech and IBM Watson, we have trained recognition engines specifically for our prompts with a large calibration data set. Each response provides dozens of individual data points, such as the quality of the sounds and words, the start and end time of each sound and word, including pauses, vocabulary frequency and difficulty, etc.
Using these data, a polytomous partial credit score (ex. 0-3) is determined for each item. We use a psychometric approach based on Item Response Theory to determine the ability level of each test-taker. This approach takes into account the difficulty for each item and the scores from other test-takers in the entire dataset to determine the test-taker’s ability.
As we collect more data, we are beginning to incorporate more and more data into our scoring algorithm and test-taking process. Using the language background survey, we can adjust the test-taking and scoring process based on that information of the test-takers, such as their native language. Additionally, as we collect more data, we are using machine learning processes to identify which particular sources of data from the open response section lead to stronger language ability predictions and incorporating them to improve our process.
Elicited Imitation has been shown to correlate highly with other standard measures of oral proficiency in multiple languages, achieving correlations as high as r = .94, p = < .05. It achieves such high correlations because it is assumed to be reconstructive in nature, that is the participant must process and reconstruct the prompt—not just rotely repeat it. The participant cannot accurately reconstruct and thus reproduce longer, more complex items if the participant lacks the proper grammatical and lexical competence in his or her interlanguage.
In addition to the languages listed below, we currently have 8 languages in development:
Hire and retain the best talent by standardizing your screening process
Decrease turnover by hiring better-qualified employees more quickly
Place students quickly and accurately with automated, adaptive language testing
Attract and retain more customers through accurate placement and tracking
Contact us today for a free demonstration on how you can improve the language assessments for your hiring and screening processes