Assessment of Inter-Language Pragmatic Competence: Review of Current Assessment Tools

Key Words: data elicitation, pragmatic assessment, interlanguage pragmatics, foreign language education, pragmatics, interlanguage pragmatics

?

CHAPTER 1: INTRODUCTION

This research reviews the literature on assessing pragmatic competence in foreign language learning. This review aims to assess and evaluate this under-researched issue. The resources reviewed went through a quality check to ensure relevance to the English-speaking population and the world at large. This introductory chapter will introduce diverse data types in pragmatics. The chapter will discuss various kinds of pragmatics analytical research because there are diverse modalities for pragmatics data, for instance, spoken and written language. This type of research data is grouped into four scalar dimensions; constraints on the interactants and the allowable contributions, fictionality or factuality of the language under observation, research interference in the production of the data, and locating data according to the researcher's focus. The problem statement review of the pragmatic assessment data elicitation technique is stated. The second chapter is the literature review, which aims to define Interlanguage Pragmatics (ILP) and the associated assessment methods. In this chapter, diverse methods assessing interlanguage pragmatics are reviewed. The purpose is to compare and contrast pragmatic assessment tools. The chapter concludes with a discussion of the limitations of pragmatic assessment research, the research gap, and research recommendations.

The third chapter is the methodology which discusses the methods chosen for the study. The methodology is qualitative, while the method of collecting data is a literature review. The chosen sources were published between 2001 to 2022 and from 1980 to 1999. The sources were retrieved from Science Direct, Journals of Pragmatics and Language Studies, Pubmed, and MedScience. The inclusion criteria were that studies should be in English, not older than 42 years, have human subjects, have quantifiable information on teaching and assessment, studies conducted in various language settings, and not be limited to the pragmatics of English language assessment. The research was also limited to types of interlanguage pragmatics assessments meaning that studies on the teaching of pragmatics were excluded. The chapters end by listing forms and methods of data collection in pragmatic assessment as elicited, oral, written, and natural. The fourth chapter is about the results, where 40 extracted titles and abstracts were narrowed to 25 that met the inclusion criteria. The purpose is to results presentation, and synthesize. The fifth chapter is the discussion and research summary. The chapter aims to answer the research questions and compare the results against the literature.

Research Questions (rephrase when done):

●?????? What are the current pragmatic assessment techniques

●?????? How do current elicitation techniques compare in terms of validity and authenticity?

Evolution of language teaching and testing to include pragmatics

Pragmatic refers to the practical and appropriate use of language to comprehend ideas or for social interactions (Rad, 2014). The assessment of pragmatics in a spoken language is crucial in evaluating interlanguage communications after learning a second language. In education, foreign language teaching has experienced expansive developments. These developments have also changed priorities that shape language teaching practices. Language teaching aims to enable learners to master a language with attention to grammar, writing, and reading. Assessment procedure in grammar education focuses on assessment procedures to ensure the students acquire linguistic knowledge.

Many theories study the communicative side of language and its linguistic features. Searle's (1969) arguments on the theory of Speech Act Theory, Canale and Swain's (1980) model of communicative competence, Leech's (1983) model for distinguishing pragmalinguistics and sociopragmatics followed by Bachman's (1990) theory of pragmatic competence. These theories have paved the way for interactive features of teaching language. The significance of communicative skills has positively impacted the focus on foreign language education. With the help of the mentioned theories, the attention has shifted from linguistic to communicative aspects in teaching foreign languages. Hence, interactional skills are prioritized over structural competence developments in a target language.

?

CHAPTER 2: LITERATURE REVIEW

Assessing pragmatic competence

There are limited studies about assessing pragmatic competencies compared to studies about factors affecting pragmatic developments and speech act production. Oller (1979) proposed a pragmatic proficiency test while investigating two problems incurred when designing the rests. The first problem is time limitations because test makers produce multiple tests within a specific time frame, while it is difficult for learners to process a language within the challenges of time limitations. The second challenge relates to the meaningfulness of conversations in the tests because test makers should provide tests with a language that helps them to produce practical tests. Oller (1979), after analyzing these problems, suggests that there should be meaningful language for examiners to work effectively.

There is a high level of reliability and validity in programmatic assessments that foreign language scholars have adopted. Hudson, Detmer, and Brown's (1992, 1995) study designed and developed six prototype assessment instruments to analyze power differences' influence on social distance, interlocutors, and ranking imposition on pragmatic productions. These scholars developed various discourse tests containing open-ended questions both in the form of written and oral versions focusing on Written Discourse Completion Tasks (WDCT), Multiple-Choice Discourse Completion Tasks (MDCT), Oral Discourse Completion Tasks (ODCT), Discourse Role-play Talks (DRPT), Discourse Self- Assessment Tests (DSAT) and Role-play Self-Assessments (RPSA). The aim was to assess learners' performance based on certain circumstances. Commenting on these tools, Brown (2001) argues that variations offered through these tools are significant in designing language tests. Jianda (2006) re-designed the Discourse Completion Task (DCT) while conducting a Chinese EFL quantitative study where they added multiple-choice items to assess the offline pragma linguistic productions of the participants. The type of DCT met the reliability and validity criteria for the Chinese context in assessing participants' interleague pragmatic skills and knowledge. It was concluded that there are variations between higher achievers and low achievers, considering that English proficiency differences were not observed.

Types of data gathered from production instruments

Researchers using different data collection tools have focused on assessing pragmatic competencies. Two categories of data are gathered from production instruments (Félix-Brasdefer, 2007). These are Natural discourse data and Elicited data.

Natural discourse data

The first category is natural discourse data which refers to the sources of social interactions observed in a natural setting which are then recorded using audio- or videotaped recordings. In the case of elicited data, researchers manage social and situational variables such as gender, level of imposition, age, and power. The most common method of data elicitation is DCT, where participants write situational description information (Martínez-Flor & Usó-Juan, 2011). There are merits of DCT, including that situational and social variables are controlled, and it is possible to collect a considerable amount of data within a short time. However, according to (Golato, 2003), this method is criticized based on reliability, validity, and naturalness in reflecting honest communication. In addition, WDCTs are metapragmatic because they require the participants to comprehend what they think can be situationally effective responses within imaginary and possible situations.

The naturally occurring speech is a valid measure of actual languages because it shows ND discourse features, as argued by (Tran, 2004). This method, however, limits the researcher from controlling social and contextual variables. Fusterv (2013) laments that this lack of control challenges the replication of the same scenario in similar research. It also makes it hard to compare speech samples from various groups and individuals in a systematic approach. Natural data is unsystematic because the structure of the entire conversation is not captured, such as dispreferred responses and non-conventionally indirect exchanges (Beebe and Cummings 1996; Félix-Brasdefer, 2007). It is challenging and time-consuming to control sociolinguistic variables such as ethnic group, sex, age, socioeconomic status, and educational level of the participants. Accurate data may be used to examine pragmatic behavior, but the naturalistic corpus does not produce adequate data to assess higher frequencies of pragmatic features in the questionnaire (Félix-Brasdefer, 2007). To avoid these limitations, effective data collection methods have been developed to evaluate learners' interlanguage pragmatic competence, such as rating assessment tests, Written Discourse Completion Tasks (WDCT), recordings of natural discourses, verbal protocols, and Multiple-Choice Questionnaires (MCQ).

Role Play (RP) is a popular data-gathering tool for studying speech actions (Reiter, 2000; Rental, 1981; VonCanon, 2005). RP simulates social interactions in which players adopt and perform specific roles within specified scenarios (Tran, 2006). Open and closed RPs are the two main categories. The participant and the interlocutor take a single turn in a closed RP, but in an open RP, both parties may take as many rounds as necessary to accomplish the goal. RR is used to collect data in IPL research because it elicits spoken data where interlocutors assume roles under a predefined experimental condition (Félix-Brasdefer, 2010). Variations between closed and open-ended RR depend on the variety and the number of productions involved (Kasper & Roever, 2005). The closed role-play pays attention to a single participant in responding to situation analysis using specific instructions, as argued by (Martínez-Flor & Usó-Juan, 2011), while the open RR several discourses and turns phrases because the interlocutors need the order to end or to continue the interaction. In WDCT, contextual variables such as situation, power, and distances of the participants can be altered to assess their impacts regarding how learners express communicative acts. Role-playing has similar merits to WDCT, such as oral production presentation. It also has identical demerits related to the naturalness and validity of data (Golato, 2003). This author suggests that researchers should consider issues related to transcription because they are time-consuming. However, despite the limitations, RR is regarded as an authentic language, and ethnographic rather than written production approach, as with DCT.

Elicited data

The second category is the elicited data collection methods. In pragmatics, the DCT has been for a long time used to gather information. In this method, participants are presented with a variety of hypothetical situations and asked to provide written responses based on their expectations of how they would respond in those circumstances. This technique can collect a substantial amount of data in a short amount of time, and it has found widespread application in the fields of interlanguage pragmatics (Barron, 2000, 2003; Hoffman-Hicks, 1999; Kondo, 1997; Matsumura, 2001; Rodriguez, 2001) and cross-cultural pragmatics (Ballesteros-Marin, 2002; Mir, 1993; Pinto, 2002). Fuster (2013) argued that DCTs gather what people think they should say rather than what they do. The participants are given time to deliberate about their answers, making them less impromptu than when speaking.

DCT allows sensitive and accurate evaluation of changes in listening to and reading discourse (Rad, 2014). DCT is a one-dimensional evaluation method and is not advised for a comprehensive pragmatic examination (Eslami & Mirzaei, 2014). Role Play, as opposed to DCTs, is a more accurate indicator of natural language use (Fuster, 2013). DCT is a standardized narrative processing test with strong psychometric qualities that probes both overt and covert understanding of narrative content.

Fuster (2013) argues that the RP had seven times as many alerts as ND. The author demonstrates a property of the RP data alone, in which greetings and alerts co-occur in the same opening phase. Opening tactics were utilized more often in the former (RP: 91% vs. ND: 48%) when comparing RP and ND encounters. This finding demonstrates that many genuine conversations began immediately with a request, without any introductory period. It might have resulted from how the talks were recorded. The verbal report data from the (Woodfield, 2008) study provide evidence that even when research participants complied with the researcher's requests, they might have negative opinions of the task design. According to the study, participants had trouble with a task meant to elicit a request when another speech act would have been more appropriate

ODCT data is lengthier, creates more alerts, support motions, and internal modifiers, and shows a more casual manner. The WDCT information is concise and written in an official style. The DCTs elicit hypothetical language, not a speech occurrence's social aspect. DCTs provide sufficient information for establishing the semantic formulas used in a speech act and the acceptable range of linguistic behavior in a specific language (Cyluk, 2013). However, in WDCT, participants need more clarification about which variation to use than in ODCT. More instances of combining formal (written) and informal (spoken) varieties are found in WDCT. Formal and informal language devices are not mutually exclusive categories because both sound reduction and singular/plural verbs end in action, for instance, considering the difference between the words "benshin" and "beneshinid" (Eslami & Mirzaei, 2014). There is a difference between the written and spoken forms of language on WDCT, with the former being more formal.

Eslami & Mirzaei (2014) suggests that the distinction between written and oral DCT data is pronounced in languages where the two modes of communication vary significantly. These distinctions manifest in the kinds of alerters and internal modifiers used. For example, the courtesy marker "please" (lotfan, informal/spoken) is more common in ODCT than it is in WDCT (khahesh mikonam, written/formal). WDCT includes blatant examples of blending different writing and speaking styles. Academics such as (Rintell & Mitchell, 1989; Sasaki, 1998) argue that surveys and RP assess the same concept, and this may be the case for languages like English, where the differences between the written and spoken forms are less pronounced.

?Each Persian variation has its distinct characteristics, and the results of WDCT and ODCT reveal various facets of these variants for Persian speakers (Eslami & Mirzaei, 2014). Eslami & Mirzaei (2014) demonstrate that native Persian speakers' performance differs when using data elicitation strategies. Examining how elicitation design aspects could influence Persian language students' performance data would be fascinating. To determine what parts of task design may have impacted participant performance, it is necessary to compare real-world information with verbal and written DCT results.

Data elicitation techniques have varying effects on different types of speech. Additional distinctions between WDCT and ODCT are expressed by examining more face-threatening activities, such as refusals. How these two elicitation instruments affect data in high power and high distance conditions compared to other situational configurations requires further into the situational characteristics. In addition, neither formality nor the ability to switch styles should be seen as absolutes but as ranges on a continuum (Eslami & Mirzaei, 2014). There are issues with the elicitation instrument that need to be addressed in pragmatics research studies that rely only on performance data (Woodfield, 2008). Nonetheless, WDCTs may be helpful research tools for extracting offline practical information from L2 students.

?

CHAPTER 3: METHODOLOGY

Literature review refers to the critical evaluation, summarizing, and evaluation of the selected materials and, at the same time, relating them to the area of research (Labaree, 2009). The literature review focuses on examining what has been done in the years of inclusivity, discussing the finding, and identifying the gaps within the same study area. Limitations of the sample articles or studies are also addressed to ensure that the study being conducted will not fall into the same problems. Grouping of articles helps in placing them under the same topics and helps in the evaluation of each article. The main idea of employing a literature review is to summarize and evaluate what each article or study found in each case.

The literature review process helps in shaping the problems being investigated by the current research. Labaree (2009) reports that summarizing the previous studies helps to bring forward essential points from the previous studies. In a similar study, Levy and Ellis (2006) argue that a literature review helps a researcher to substantiate their problem statement and provide a solid theoretical background in the area of study. Additionally, a literature review helps frame an appropriate methodology to employ in the current study and the research goals (Levy & Ellis, 2006). Therefore, a research-based paper must use a literature review to communicate the past findings and, at the same time, help understand what is needed in the study being conducted.

In selecting relevant articles, keywords such as pragmatics, pragmatic assessment, foreign language education, data elicitation, and interlanguage pragmatics were targeted. The articles had to mention at least 4 of the keywords in order to be included for use in the work. Using keywords helps narrow down the desired articles and eliminate the others. The abstract was also included in the preliminary steps before an article or study was considered for use in the current project.

CHAPTER 4: RESULTS

The researcher extracted 40 titles, and abstracts were narrowed to 25 that met the inclusion criteria. The results are summarized in the table below.

Task

Component of pragmatics targeted

Format

Conversation simulations (a) (Bardovi-Harlig et al., 2010)

Responding to turns; use of conventional expressions

Aural input/oral production

Conversation simulations (b) (Bardovi-Harlig, Mossman, &Vellenga, in press)

Responding to turns; Academic discussion; use of pragmatic routines?

Aural input/oral production

Written exchange (Kuha, 1997)

Responding to computer-generated turns, speech act (originally complaints), multiple turns

Written

Conversational Implicature Task (Taguchi, 2005, 2008)

Interpretation of conversational implicature; listening comprehension (needed for conversation); the processing of prosodic information

Aural presentation with written/read interpretation?

Speech Act interpretation task (Koike, 1989, 1996)

Intended (illocutionary) and perceived (perlocutionary) force of utterances; grammar and pragmatics

Audio and Audio-Visual presentation with written/read interpretation

Pragmatic acceptability judgment tasks (BardoviHarlig&D?rnyei, 1998)

Sociopragmatic knowledge; pragmalinguistics; interaction of grammar and pragmatics

Audio-visual presentation with written/read interpretation (two presentations)

Prediction task (based on Koike, 1996 and Bardovi-Harlig, 2009)

Sociopragmatic knowledge (which contexts require what speech act), independent of the linguistic form (pragmalinguistics)

Written, audio, or audio-visual input

Aural multiple choice task (Teng & Fei, 2013)

Sociopragmatic and pragmalinguistic knowledge

Aural presentation of distractors

?

Research shows that RP and production surveys show more realistic responses (Billmyer & Varghese, 2000; Félix-Brasdefer, 2007; Kuriscak, 2006; Márquez Reiter, 2000; Márquez Reiter, Rainey, & Fulcher, 2005; Woodfield, 2008). They give enough information about the required context components because role-playing situations are proposed as a valuable tool in pragmatics research (such as a description of the setting, purpose, age, and gender of participants, and their relationship). Speech act behavior may be elicited because social parameters (such as gender, age, social distance, social power, and the degree of imposition) can be controlled, and the interaction elements of speech acts can be examined via RP. Conversational data from groups of individuals are gathered using RP. For both native and non-native speakers, increasing situational triggers significantly influences request length and complexity. Production surveys have limited social variables (social distance, dominance, and imposition) and are inadequate to elicit as complete a response as biological data.

When assessing the pragma linguistic aspects of speech, RP is better than raw data since it minimizes confusing social contexts (such as social distance, power, sex, and degree of education). For scientific research, RP is an artificial interaction that mimics human social behavior. The realistic encounters, when all kinds of internal lexical mitigators are employed, the RP speakers use politeness markers (such as por favor) while speaking to one another (such as bonico [cutie]) (Fuster, 2013). The politeness marker is the overall favorite among participants, and while just 13% of talks use this term, 46% of RP interactions rely on it.

RP and raw data both guide different facets of speech act behavior. However, the higher the number of tasks, the more effect it has across methodologies for comparing pragmatic data (Cohen 2004). Since neither RP nor the natural approach can recreate the precise context of the recorded speech, documenting is not advised (Félix-Brasdefer, 2007). Speakers from various cultural backgrounds are sensitive to varying aspects of the context related to multiparty interaction, such as the type of addressee (single or not) and the type of beneficiary (Pulaczewska, 2013). Multiparty interaction should be considered when selecting testing situations for RP and written simulations of speech. Distinct from group contact or interpersonal collaboration, communication between a Speaker and Hearer occurs outside of a group setting, and people who speak different languages have different experiences with different events.

?

?

CHAPTER 5: DISCUSSION, CONCLUSION & RECOMMENDATIONS

There are long-established pragmatic assessment formats such as multiple-choice DCTs, oral DCTs, written DCTs, RR, and self-assessments. Still, they have been shown to have varying degrees of reliability, validity, and applicability. The questionnaire mode of data collection has both strengths and weaknesses. This data-gathering strategy should be assessed in light of the questions posed (Cyluk, 2013). However, its trustworthiness and validity are disputed, and the tools should be used wisely, clearly understanding their abilities and limitations. Without any ridiculous expectations that any instrument might be able to solve all the problems in linguistics, all methods can be used to improve the understanding of language and language use. Researchers and instructors can utilize such technologies because they are beneficial in their study and teaching settings. There is a need to shift from pragmatics assessment research that employs closed laboratory instruments because they are accessible to a selected group of academics (Labben, 2016). Learners' pragmatic competence in particular second/foreign language circumstances, such as those involving countries with limited resources, should be assessed for instructional and research purposes.

Contrary to the findings of other studies such as (Bardovi-Harlig & Hartford, 1993; Martinez-Flor, 2013; Rintell & Mitchell, 1989), the results of this study show that requests elicited from native Persian speakers using the two different elicitation techniques (WDCT and ODCT) are significant. This research criticizes the reliability of questionnaires because responses are generated in a test-like setting rather than in a natural one. The method is also criticized because it elicits the respondent's spoken word in a written medium (Eslami & Mirzaei, 2014). ODCT data represent real-world scenarios better compared to questionnaires.

This study suggests that there is still a need for consideration when making essential judgments based on test results. It is impossible to achieve perfection in reliability, validity, and applicability at once; instead, researchers need to find a medium between reliability, validity, and usability that works for the specific context in which a particular test will be used (Bachman, 2005), and this means that researchers need to focus on the context in which a test will be used (Bachman & Palmer, 2010; Bardovi-Harlig & Shin, 2014). The focus will ensure the creation, selection, usage, and assessment of a practical assessment instrument. Educators should focus on communicative knowledge to enhance students' pragmatic competencies in learning a foreign language. There should be new regulations for material and curriculum design guides, especially for foreign language programs. These regulations ensure the development of learners' communicative or pragmatic skills. There are variations between higher and low achievers, considering that English proficiency differences are challenging to observe. Hence, in measuring pragmatics produced, researchers should deploy various data collection tools.

??

?

?

References

Agnieszka Cyluk. (2012). Discourse Completion Task: Its Validity and Reliability in???????

Research Projects on Speech Acts. Anglica. An International Journal of English Studies, 22(2), 100–111.

Agnieszka Cyluk. (2012a). Discourse Completion Task: Its Validity and Reliability in????? Research Projects on Speech Acts. Anglica. An International Journal of English????????? Studies, 22(2), 100–111.

Félix-Brasdefer, J. C. (2007). Pragmatic development in the Spanish as an FL classroom: A???????? cross-sectional study of learner requests. Intercultural Pragmatics, 4(2).? https://doi.org/10.1515/ip.2007.013

House, J. (2018). Authentic vs. elicited data and qualitative vs. quantitative research methods?????? in pragmatics: Overcoming two non-fruitful dichotomies. A system, 75, 4–12.??????????? https://doi.org/10.1016/j.system.2018.03.014

Kanik, M. (2013). Reverse discourse completion task as an assessment tool for intercultural????????? competence. Studies in Second Language Learning and Teaching, 3(4), 621.?????? https://doi.org/10.14746/ssllt.2013.3.4.8

Kathleen Bardovi-Harlig, & Sun-Young Shin. (2014). Expanding Traditional Testing??????? Measures with Tasks from L2 Pragmatics Research. International Journal of???????? Language Testing, 4(1), 26–49.

Labaree, R. V. (2009). Research guides: Organizing your social sciences research paper: 5.????????? The literature review.

Labben, A. (2022). Reconsidering the development of the discourse completion test in????? interlanguage pragmatics. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA), 69–91. https://doi.org/10.1075/prag.26.1.04lab?????????

Levy, Y., & Ellis, T. J. (2006). A systems approach to conduct an effective literature review???????? in support of information systems research. Informing Science, 9.

Takkac, T., Yagiz, & Han. (2017). Bridging the Gap between Linguistic and Pragmatic???? Assessment in Turkish Contexts. PASAA Journal (PASAA) Thailand., Volume?????? 54(July-December 2017), 1–24. https://files.eric.ed.gov/fulltext/EJ1171224.pdf

Woodfield, H. (2008). Problematising Discourse Completion Tasks: Voices from Verbal? Report. Evaluation &Amp; Research in Education, 21(1), 43–69.?????????? https://doi.org/10.2167/eri413.0

Xu, L., & Wannaruk, A. (2015). Reliability and Validity of WDCT in Testing Interlanguage??????? Pragmatic Competence for EFL Learners. Journal of Language Teaching and Research, 6(6), 1206. https://doi.org/10.17507/jltr.0606.07

Zohreh R. Eslami, & Azizullah Mirzaei. (2014b). Speech Act Data Collection in a Non-Western Context: Oral and Written DCTs in the Persian Language. International??????? Journal of Language Testing, 4(1), 137–154.