+
+ 1. Description
+
+ This repository contains tools to automatically analyse how
+ participants align their use of task-specific referents in their
+ dialogue and actions for a collaborative learning activity, and how
+ it relates to the task success (i.e. their learning
+ outcomes and task performance).
+ As a use case, it processes data from a collaborative problem solving
+ activity named JUSThink [1, 2], i.e.
+ JUSThink Dialogue and Actions Corpus data set that is available from the
+ Zenodo Repository, DOI: 10.5281/zenodo.4627104, and reproduces the results and figures
+ in [3].
+ In brief:
+
+ - JUSThink Dialogue and Actions Corpus contains
+ transcripts, event logs, and test responses of children aged 9
+ through 12, as they participate in the JUSThink activity [1, 2]
+ in pairs of two, to solve a problem on graphs together.
+ - The JUSThink activity and its study is first
+ described in [1], and elaborated with findings concerning the link
+ between children's learning, performance in the activity, and
+ perception of self, the other and the robot in [2].
+ - Alignment analysis in our work [3] studies the participants' use of
+ expressions that are related to the task at hand, their follow up
+ actions of these expressions, and how it links to task success.
+
+
+ 2. Publications
+
+ If you use this work in an academic context, please cite the following
+ publications:
+
+ -
+
Norman*, U., Dinkar*, T., Bruno, B., & Clavel, C. (2022).
+ Studying Alignment in a Collaborative Learning Activity via
+ Automatic Methods: The Link Between What We Say and Do. Dialogue
+ & Discourse, 13(2), 1 - ;48. *Contributed equally to this
+ work. https://doi.org/10.5210/dad.2022.201
+
+ -
+
Norman, U., Dinkar, T., Bruno, B., & Clavel, C. (2021).
+ JUSThink Alignment Analysis. In Dialogue & Discourse
+ (v1.0.0, Vol. 13, Number 2, pp. 1 - ;48). Zenodo. https://doi.org/10.5281/zenodo.4675070
+
+
+
+ 3. Content
+
+ The tools provided in this repository consists of 7 Jupyter Notebooks
+ written in Python 3, and two additional external tools utilised by the
+ notebooks.
+
+ 3.1. Jupyter Notebooks
+
+ We highlight that the notebooks up until the last (i.e. to test the
+ hypotheses (tools/7_test_the_hypotheses.ipynb)) present a general
+ pipeline to process event logs, test responses and transcripts to
+ extract measures of task performance, learning outcomes, and measures of
+ alignment.
+
+ - Extract task performance (and other features) from the logs
+ (tools/1_extract_performance_and_other_features_from_logs.ipynb):
+ Extracts various measures of task behaviour from the logs, at
+ varying granularities of the activity (i.e. the whole corpus, task,
+ attempt, and turn levels). In later notebooks, we focus on one of
+ the features to estimate the task performance of a team: (minimum)
+ error.
+ - Extract learning outcomes from the test responses
+ (tools/2_extract_learning_gain_from_test_responses.ipynb): Extracts
+ measures of learning outcomes from the responses to the pre-test and
+ the post-test. In later notebooks, we focus on one of the features
+ to estimate the learning outcome of a team: relative learning gain
+ [4]
+ - Select and visualise a subset of teams for
+ transcription
+ (tools/3_visualise_transcribed_teams.ipynb): Visualises the
+ transcribed teams among the other teams in the feature space spanned
+ by task performance and learning outcome, as well as the
+ distribution of their number of attempts and turns.
+ - Extract routines from transcripts
+ (tools/4_extract_routines_from_transcripts.ipynb) (uses dialign to
+ extract routines): Extracts routines of referring expressions that
+ are "fixed", i.e. become shared or established amongst
+ interlocutors.
+ - Combine transcripts with logs
+ (tools/5_construct_the_corpus_by_combining_transcripts_with_logs.ipynb):
+ Merges transcripts with event logs to have a combined dialogue and
+ actions corpus, to be processed e.g. to detect follow-up
+ actions.
+ - Recognise instructions and detect follow-up actions
+ (tools/6_recognise_instructions_detect_follow-up_actions.ipynb):
+ Extracts verbalised instruction such as "connect Mount Basel to
+ Montreux", and pairs them with the follow-up action that may
+ match (e.g. if the other connects Basel to Montreux) or
+ mismatch (e.g. if the other connects Basel to
+ Neuchatel) with the instruction.
+ - Test the hypotheses in [3] (tools/7_test_the_hypotheses.ipynb) (uses
+ effsize to estimate effect size, specifically
+ Cliff's Delta): Considers each research questions and hypotheses
+ studied in [3] and generates the results in [3].
+
+
+ 3.2. External Tools
+
+
+ - dialign
+ tool to extract routines, specifically Release 1.0 from dialign-1.0.zip:\n It extracts routine expressions that are
+ "shared" among the participants from transcripts. \n It is
+ used as an external module (in accordance with its CeCILL-B License,
+ see License).
+ - effsize tool to compute estimators of effect
+ size.\n We specifically use it to compute Cliff's Delta, which
+ quantifies the amount difference between two groups of observations,
+ by computing the Cliff's Delta statistic.\n It is taken from
+ project DABEST (see License).
+
+
+ 4. Research Questions and Hypotheses in [3]
+
+
+ - RQ1 Lexical alignment: How do the interlocutors
+ use expressions related to the task? Is this associated
+ with task success?
+ - H1.1: Task-specific referents become
+ routine early for more successful teams.
+ - H1.2: Hesitation phenomena are more likely
+ to occur in the vicinity of priming and establishment of
+ task-specific referents for more successful teams.
+
+
+ - RQ2 Behavioural alignment: How do the interlocutors
+ follow up these expressions with actions? Is this
+ associated with task success?
+ - H2.1: Instructions are more likely to be
+ followed by a corresponding action early in the dialogue for
+ more successful teams.
+ - H2.2: When instructions are followed by a
+ corresponding or a different action, the action is more
+ likely to be in the vicinity of information management
+ phenomena for more successful teams.
+
+
+
+ The RQs and Hs are addressed in the notebook for testing the hypotheses
+ (i.e. tools/7_test_the_hypotheses.ipynb).
+
+ Acknowledgements
+
+ This project has received funding from the European Union's Horizon
+ 2020 research and innovation programme under grant agreement No 765955.
+ Namely, the ANIMATAS Project.
+
+ License
+
+ The whole package is under MIT License, see the LICENSE
+ file.
+ Classes under the tools/effsize package were taken from
+ project DABEST, Copyright 2016-2020 Joses W. Ho.
+ These classes are licensed under the BSD 3-Clause Clear License. See
+ tools/effsize/LICENSE file for additional
+ details.
+ Classes under the tools/dialign-1.0 package were taken
+ from project dialign. These classes are licensed under the
+ CeCILL-B License. This package is used as an "external
+ module", see tools/dialign-1.0/LICENSE.txt for
+ additional details.
+
+