東華大學圖書館 |

Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context./
作者:	Karatay, Yasin.
面頁冊數:	1 online resource (216 pages)
附註:	Source: Dissertations Abstracts International, Volume: 84-04, Section: A.
Contained By:	Dissertations Abstracts International84-04A.
標題:	Linguistics. -
電子資源:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29165842click for full text (PQDT)
ISBN:	9798351419596

Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context.
Karatay, Yasin.

Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context. - 1 online resource (216 pages)

Source: Dissertations Abstracts International, Volume: 84-04, Section: A.

Thesis (Ph.D.)--Iowa State University, 2022.

Includes bibliographical references

The assessment of oral proficiency in English for Specific Purposes (ESP) contexts remains a central challenge for researchers, test developers, and classroom teachers (Knoch & Macqueen, 2020). This problem is apparent especially in Tourism and Hotel Management (THM) programs, where students are tested mostly based on grammar, vocabulary, and listening through multiple-choice items because of several logistical issues such as crowded classrooms and resource intensive nature of administering an oral communication assessment (Hsu, 2014). One way to solve this problem is to use authentic role-play tasks that can predict the target language performance of a test-taker in a simulated context (Douglas, 2000; Manias & McNamara, 2016) by using spoken dialog systems (SDSs) in which computers act as conversational agents. However, there have been limited applications of SDSs for assessing oral communication skills in ESP assessment so far.To fill this gap, the current dissertation study aimed to develop a prototype of SDS-mediated Tourism English oral communication assessment and investigate the level of support for the first four inferences (domain description, evaluation, generalization, and explanation) in the interpretation/use argument for the computer-based Tourism English oral communication assessment (CTEA). Based on evidence-centered design (ECD) model (Mislevy, Steinberg, &, Almond, 2003) as the guiding framework for test development, three counter service scenarios were designed and developed using a specialized SDS in which computer acts as a hotel guest and examinees act as a receptionist. A total of 30 EFL students majoring in a THM program at a Turkish state university, 12 domain experts, and four raters participated in the study. Employing a mixed-method multiphase research design (Creswell & Plano Clark, 2012), the study collected various types of data such as students' CTEA test scores, their Tourism English midterm grammar and vocabulary scores, Tourism English course instructors' evaluations of the students, stimulated recalls, individual semi-structured interviews, and responses on post-test surveys and post rating surveys.The four inferences were well supported by the evidence collected from the data analysis. The domain description inference was supported by domain experts' overall positive opinions about the authenticity and effectiveness of the CTEA tasks in eliciting oral discourse from test-takers. The evaluation inference was backed by the findings obtained from the students' interview and survey data that suggested an overall positive perception of the CTEA. The students also thought that the CTEA allowed them to demonstrate their oral communication skills in simulated hospitality-related situations and underlined a potentially positive impact on their learning processes in a future CTEA integration into the Tourism English assessment. The evaluation inference was also supported by the additional findings obtained from all four raters who reported that the SDS was consistent across all test-takers and thought that the SDS could potentially eliminate the effect of partner variability on their performance.The generalization inference was backed by the findings of the G-theory analysis on 360 composite scores assigned to 30 students' CTEA performances on three different task prompts. G-study findings revealed that the score variance was 66.5% for person and there was minimal rater severity, meaning that two-thirds of the score variance was due to the test-takers' Tourism English oral communication ability. Additionally, D-study findings yielded a substantially high dependability index (Φ= .90) for the current test design and an acceptable level of dependability (Φ ≥ .70) was obtained in all testing scenarios, except for one rater and one task scenario. However, the assumption for the appropriateness of test task specifications for producing parallel tasks was partially supported because of the technical difficulty and resource intensive nature of developing parallel SDS-based test tasks.Finally, the explanation inference was supported by the results of the concurrent correlational study between the CTEA scores and external measures (grammar scores, r=.62; vocabulary scores, r=.67; and teacher evaluations, r=.75). This inference was also supported by the stimulated recall data findings that pointed to an overall potential of CTEA in providing opportunities for test-takers to demonstrate a variety of strategies to complete the tasks.The study contributes to knowledge about practical assessment of ESP, specifically Tourism English, oral communication at universities by presenting significant implications on the design and use of an SDS-based role-play tasks in ESP assessment, as well as learning and teaching processes.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2023

Mode of access: World Wide Web

ISBN: 9798351419596Subjects--Topical Terms:

524476
Linguistics.
Subjects--Index Terms:

Computer assisted language testingIndex Terms--Genre/Form:

542853
Electronic books.

Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context.
LDR:06298nmm a2200409K 4500 001 2354420
005 20230414084752.5
006 m o d
007 cr mn ---uuuuu
008 241011s2022 xx obm 000 0 eng d
020 $a 9798351419596
035 $a (MiAaPQ)AAI29165842
035 $a AAI29165842
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Karatay, Yasin. $3 3694773
245 1 0 $a Development and Validation of Spoken Dialog System-Based Oral Communication Tasks in an Esp Context.
264 0 $c 2022
300 $a 1 online resource (216 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 84-04, Section: A.
500 $a Advisor: Chapelle, Carol.
502 $a Thesis (Ph.D.)--Iowa State University, 2022.
504 $a Includes bibliographical references
520 $a The assessment of oral proficiency in English for Specific Purposes (ESP) contexts remains a central challenge for researchers, test developers, and classroom teachers (Knoch & Macqueen, 2020). This problem is apparent especially in Tourism and Hotel Management (THM) programs, where students are tested mostly based on grammar, vocabulary, and listening through multiple-choice items because of several logistical issues such as crowded classrooms and resource intensive nature of administering an oral communication assessment (Hsu, 2014). One way to solve this problem is to use authentic role-play tasks that can predict the target language performance of a test-taker in a simulated context (Douglas, 2000; Manias & McNamara, 2016) by using spoken dialog systems (SDSs) in which computers act as conversational agents. However, there have been limited applications of SDSs for assessing oral communication skills in ESP assessment so far.To fill this gap, the current dissertation study aimed to develop a prototype of SDS-mediated Tourism English oral communication assessment and investigate the level of support for the first four inferences (domain description, evaluation, generalization, and explanation) in the interpretation/use argument for the computer-based Tourism English oral communication assessment (CTEA). Based on evidence-centered design (ECD) model (Mislevy, Steinberg, &, Almond, 2003) as the guiding framework for test development, three counter service scenarios were designed and developed using a specialized SDS in which computer acts as a hotel guest and examinees act as a receptionist. A total of 30 EFL students majoring in a THM program at a Turkish state university, 12 domain experts, and four raters participated in the study. Employing a mixed-method multiphase research design (Creswell & Plano Clark, 2012), the study collected various types of data such as students' CTEA test scores, their Tourism English midterm grammar and vocabulary scores, Tourism English course instructors' evaluations of the students, stimulated recalls, individual semi-structured interviews, and responses on post-test surveys and post rating surveys.The four inferences were well supported by the evidence collected from the data analysis. The domain description inference was supported by domain experts' overall positive opinions about the authenticity and effectiveness of the CTEA tasks in eliciting oral discourse from test-takers. The evaluation inference was backed by the findings obtained from the students' interview and survey data that suggested an overall positive perception of the CTEA. The students also thought that the CTEA allowed them to demonstrate their oral communication skills in simulated hospitality-related situations and underlined a potentially positive impact on their learning processes in a future CTEA integration into the Tourism English assessment. The evaluation inference was also supported by the additional findings obtained from all four raters who reported that the SDS was consistent across all test-takers and thought that the SDS could potentially eliminate the effect of partner variability on their performance.The generalization inference was backed by the findings of the G-theory analysis on 360 composite scores assigned to 30 students' CTEA performances on three different task prompts. G-study findings revealed that the score variance was 66.5% for person and there was minimal rater severity, meaning that two-thirds of the score variance was due to the test-takers' Tourism English oral communication ability. Additionally, D-study findings yielded a substantially high dependability index (Φ= .90) for the current test design and an acceptable level of dependability (Φ ≥ .70) was obtained in all testing scenarios, except for one rater and one task scenario. However, the assumption for the appropriateness of test task specifications for producing parallel tasks was partially supported because of the technical difficulty and resource intensive nature of developing parallel SDS-based test tasks.Finally, the explanation inference was supported by the results of the concurrent correlational study between the CTEA scores and external measures (grammar scores, r=.62; vocabulary scores, r=.67; and teacher evaluations, r=.75). This inference was also supported by the stimulated recall data findings that pointed to an overall potential of CTEA in providing opportunities for test-takers to demonstrate a variety of strategies to complete the tasks.The study contributes to knowledge about practical assessment of ESP, specifically Tourism English, oral communication at universities by presenting significant implications on the design and use of an SDS-based role-play tasks in ESP assessment, as well as learning and teaching processes.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2023
538 $a Mode of access: World Wide Web
650 4 $a Linguistics. $3 524476
650 4 $a Educational technology. $3 517670
650 4 $a Educational tests & measurements. $3 3168483
653 $a Computer assisted language testing
653 $a English for specific purposes
653 $a Language assessment
653 $a Oral communication
653 $a Spoken dialog systems
653 $a Validity argument
655 7 $a Electronic books. $2 lcsh $3 542853
690 $a 0290
690 $a 0710
690 $a 0288
710 2 $a ProQuest Information and Learning Co. $3 783688
710 2 $a Iowa State University. $b English. $3 1020984
773 0 $t Dissertations Abstracts International $g 84-04A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=29165842 $z click for full text (PQDT)