JUSThink Alignment Analysis

doi_dedup___::c054151b6a8c4f41c7acf160651a6503 2022-10-13T00:15:44+0000 2022-10-13T07:44:29.152Z

oai:zenodo.org:4675070 50|od______2659::3801993ea8f970cfc991277160edf277 oai:zenodo.org:6974562 50|od______2659::9c87ff4a5e7710052b873088e7265072 10.5281/zenodo.4675069 10.5281/zenodo.4675070 10.5281/zenodo.6974562 10.5281/zenodo.4675069 JUSThink Alignment Analysis Norman, Utku Dinkar, Tanvi Bruno, Barbara Clavel, Chloé 2022-08-08 1. Description This repository contains tools to automatically analyse how participants align their use of task-specific referents in their dialogue and actions for a collaborative learning activity, and how it relates to the task success (i.e. their learning outcomes and task performance). As a use case, it processes data from a collaborative problem solving activity named JUSThink [1, 2], i.e. JUSThink Dialogue and Actions Corpus data set that is available from the Zenodo Repository, DOI: 10.5281/zenodo.4627104, and reproduces the results and figures in [3]. In brief: JUSThink Dialogue and Actions Corpus contains transcripts, event logs, and test responses of children aged 9 through 12, as they participate in the JUSThink activity [1, 2] in pairs of two, to solve a problem on graphs together. The JUSThink activity and its study is first described in [1], and elaborated with findings concerning the link between children's learning, performance in the activity, and perception of self, the other and the robot in [2]. Alignment analysis in our work [3] studies the participants' use of expressions that are related to the task at hand, their follow up actions of these expressions, and how it links to task success. Changes in Release v1.1.0: updated with the publication information, finalized paper structure, research questions and hypotheses as in the published article: U. Norman*, T. Dinkar*, B. Bruno, and C. Clavel, "Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do," Dialogue &amp; Discourse, 13(2), 1–48. *Contributed equally to this work. 10.5210/dad.2022.201. Full Changelog: https://github.com/chili-epfl/justhink-alignment-analysis/compare/v1.0.0...v1.1.0 2. Publications If you use this work in an academic context, please cite the following publications: Norman*, U., Dinkar*, T., Bruno, B., &amp; Clavel, C. (2022). Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do. Dialogue &amp; Discourse, 13(2), 1–48. *Contributed equally to this work. https://doi.org/10.5210/dad.2022.201 Norman, U., Dinkar, T., Bruno, B., &amp; Clavel, C. (2021). JUSThink Alignment Analysis. In Dialogue &amp; Discourse (v1.1.0, Vol. 13, Number 2, pp. 1–48). Zenodo. https://doi.org/10.5281/zenodo.6974562 3. Content The tools provided in this repository consists of 7 Jupyter Notebooks written in Python 3, and two additional external tools utilised by the notebooks. 3.1. Jupyter Notebooks We highlight that the notebooks up until the last (i.e. to test the hypotheses (tools/7_test_the_hypotheses.ipynb)) present a general pipeline to process event logs, test responses and transcripts to extract measures of task performance, learning outcomes, and measures of alignment. Extract task performance (and other features) from the logs (tools/1_extract_performance_and_other_features_from_logs.ipynb): Extracts various measures of task behaviour from the logs, at varying granularities of the activity (i.e. the whole corpus, task, attempt, and turn levels). In later notebooks, we focus on one of the features to estimate the task performance of a team: (minimum) error. Extract learning outcomes from the test responses (tools/2_extract_learning_gain_from_test_responses.ipynb): Extracts measures of learning outcomes from the responses to the pre-test and the post-test. In later notebooks, we focus on one of the features to estimate the learning outcome of a team: relative learning gain [4] Select and visualise a subset of teams for transcription (tools/3_visualise_transcribed_teams.ipynb): Visualises the transcribed teams among the other teams in the feature space spanned by task performance and learning outcome, as well as the distribution of their number of attempts and turns. Extract routines from transcripts (tools/4_extract_routines_from_transcripts.ipynb) (uses dialign to extract routines): Extracts routines of referring expressions that are "fixed", i.e. become shared or established amongst interlocutors. Combine transcripts with logs (tools/5_construct_the_corpus_by_combining_transcripts_with_logs.ipynb): Merges transcripts with event logs to have a combined dialogue and actions corpus, to be processed e.g. to detect follow-up actions. Recognise instructions and detect follow-up actions (tools/6_recognise_instructions_detect_follow-up_actions.ipynb): Extracts verbalised instruction such as "connect Mount Basel to Montreux", and pairs them with the follow-up action that may match (e.g. if the other connects Basel to Montreux) or mismatch (e.g. if the other connects Basel to Neuchatel) with the instruction. Test the hypotheses in [3] (tools/7_test_the_hypotheses.ipynb) (uses effsize to estimate effect size, specifically Cliff's Delta): Considers each research questions and hypotheses studied in [3] and generates the results in [3]. 3.2. External Tools dialign tool to extract routines, specifically Release 1.0 from dialign-1.0.zip: It extracts routine expressions that are "shared" among the participants from transcripts. It is used as an external module (in accordance with its CeCILL-B License, see License). effsize tool to compute estimators of effect size. We specifically use it to compute Cliff's Delta, which quantifies the amount difference between two groups of observations, by computing the Cliff's Delta statistic. It is taken from project DABEST (see License). 4. Research Questions and Hypotheses in [3] RQ1 Lexical alignment: How do the interlocutors use expressions related to the task? Is this associated with task success? H1.1: Task-specific referents become routine early for more successful teams. H1.2: Hesitation phenomena are more likely to occur in the vicinity of priming and establishment of task-specific referents for more successful teams. RQ2 Behavioural alignment: How do the interlocutors follow up these expressions with actions? Is this associated with task success? H2.1: Instructions are more likely to be followed by a corresponding action early in the dialogue for more successful teams. H2.2: When instructions are followed by a corresponding or a different action, the action is more likely to be in the vicinity of information management phenomena for more successful teams. The RQs and Hs are addressed in the notebook for testing the hypotheses (i.e. tools/7_test_the_hypotheses.ipynb). Acknowledgements This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 765955. Namely, the ANIMATAS Project. License The whole package is under MIT License, see the LICENSE file. Classes under the tools/effsize package were taken from project DABEST, Copyright 2016-2020 Joses W. Ho. These classes are licensed under the BSD 3-Clause Clear License. See tools/effsize/LICENSE file for additional details. Classes under the tools/dialign-1.0 package were taken from project dialign. These classes are licensed under the CeCILL-B License. This package is used as an "external module", see tools/dialign-1.0/LICENSE.txt for additional details. {"references": ["[1] J. Nasir, U. Norman, B. Bruno, and P. Dillenbourg, \"You Tell, I Do, and We Swap until we Connect All the Gold Mines!,\" ERCIM News, vol. 2020, no. 120, 2020, [Online]. Available: https://ercim-news.ercim.eu/en120/special/you-tell-i-do-and-we-swap-until-we-connect-all-the-gold-mines", "[2] J. Nasir*, U. Norman*, B. Bruno, and P. Dillenbourg, \"When Positive Perception of the Robot Has No Effect on Learning,\" in 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Aug. 2020, pp. 313\u2013320, doi: 10.1109/RO-MAN47096.2020.9223343", "[3] U. Norman*, T. Dinkar*, B. Bruno, and C. Clavel, \"Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do,\" Dialogue &amp; Discourse, vol. 13, no. 2, pp. 1\u201348, Aug. 2022, doi: 10.5210/dad.2022.201.", "[4] M. Sangin, G. Molinari, M.-A. N\u00fcssli, and P. Dillenbourg, \"Facilitating peer knowledge modeling: Effects of a knowledge awareness tool on collaborative learning outcomes and processes,\"\" Computers in Human Behavior, vol. 27, no. 3, pp. 1059\u20131067, May 2011, doi: 10.1016/j.chb.2010.05.032."]} alignment situated dialogue collaborative learning spontaneous speech disfluency mutual understanding 2021-04-09 2022-08-08 Zenodo true false 0.8 dedup-result-decisiontree-v3 doi_dedup___::ae235765bbc422195a6c9f632b2d77eb 2104.04429 arXiv 2022-08-05 Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do 10.48550/arxiv.2104.04429 10.5210/dad.2022.201 corda__h2020::c4515ebef538a734cf11f795347f5dac Advancing intuitive human-machine interaction with human-like social capabilities for education in schools 765955 ec__________::EC::H2020 ec__________::EC::H2020::MSCA-ITN-ETN ANIMATAS doi_dedup___::0a6314b0ed275d915f5b57a259375691 2021-03-22 Zenodo 10.5281/zenodo.4627104 JUSThink Dialogue and Actions Corpus 10.5281/zenodo.4627103 Zenodo 10.5281/zenodo.4675070 JUSThink Alignment Analysis 2021-04-09 2022-08-08 Zenodo 10.5281/zenodo.6974562 JUSThink Alignment Analysis (v1.1.0) JUSThink Alignment Analysis (v1.1.0) 2022-08-08 Zenodo 10.5281/zenodo.4675069 2022-08-08 10.5281/zenodo.4675069 https://opensource.org/licenses/MIT https://doi.org/10.5281/zenodo.4675069 2022-08-08 10.5281/zenodo.6974562 https://opensource.org/licenses/MIT https://doi.org/10.5281/zenodo.6974562 2021-04-09 10.5281/zenodo.4675070 https://opensource.org/licenses/MIT https://doi.org/10.5281/zenodo.4675070