Published on October 11, 2023 b y Tristan Cui and Jeff Wang, Monash University on British Journal of Education and Technology.

Abstract

This study investigates the impact of Perusall, a social annotation tool, on an online postgraduate course conducted over two semesters at an Australian university. We examine the connection between students' pre-class engagement and learning outcomes, utilizing both secondary data from Perusall platform and primary data through a survey. The findings indicate that pre-class social annotations have a positive impact on students' performance on post-class assessments. Notably, English as an Additional Language students with low English proficiency achieve comparable results in Perusall as those with high English proficiency. Additionally, the study identifies key aspects of social annotation that students highly value, providing insights for future implementation. Overall, this study highlights the potential of social annotation tools like Perusall to improve pre-class engagement and enhance learning outcomes.

Practitioner notes

What is already known about this topic

• Student engagement with pre-class activities is critical to their learning outcomes.
• The integration of social annotation tools for completing pre-class reading has become progressively prevalent among teachers.
• The use of social annotation tools for reading and annotating has been found to have a positive influence on students' motivation, interactions, and academic performance.

What this paper adds

• Online social annotation on lecture content has a short-term positive impact on subsequent assessments, indicating that it is an effective way to engage students in pre-class learning and improve their understanding and retention of course material.
• English as an Additional Language (EAL) students of varying levels of English proficiency achieve comparable results in social annotation assignments, highlighting the potential of this tool to provide an equitable learning experience for all students regardless of their language abilities.
• Social annotation activities promote a strong social presence with cohesive communication, positively impacting online learning's cognitive presence. This highlights the significance of collaborative learning and social interaction in online education, and the potential of social annotation tools to create an engaging and supportive learning environment.
• Students prefer receiving feedback from teachers within a Community of Inquiry (CoI) rather than through traditional modes of direct instruction or conversation initiation, highlighting the importance of teaching presence in refining their cognitive and social presences, and ultimately enhancing the quality of their learning experience.

Implications for practice and/or policy

• This study provides practical suggestions for designing pre-class learning activities that build upon pre-class lessons, helping teachers to optimize students' learning experiences and outcomes.
• It sheds light on using inclusive and collaborative assessments as an effective way to enhance engagement and performance of EAL students.
• To cultivate a productive CoI within online educational contexts, teachers need to prioritize personalized feedback to students.

INTRODUCTION

Online learning has become increasingly prevalent in higher education due to its flexibility and convenience (Bhagat et al., 2019). However, no one had foreseen the sudden and quick shift to online learning due to the COVID-19 pandemic that began in early 2020. In a matter of weeks, face-to-face classes were transferred to online spaces. It raised concerns about students' motivation, engagement and efficacy during this difficult time.

The Community of Inquiry (CoI) framework is a valuable tool for enhancing online learning experience, emphasizing the importance of creating a supportive and collaborative learning environment (Akyol & Garrison, 2013; Garrison et al., 2000, 2001). CoI includes three dimensions of presence: cognitive presence, social presence, and teaching presence. CoI provides “a collaborative-constructivist process model” (Castellanos-Reyes, 2020, p. 557) for evaluating the quality of online teaching. It offers a useful framework for designing online classes and building online communities. In particular, online communities can enhance students' engagement, motivation to learn and quality of learning (Fiock, 2020). To establish presence and foster online communities, various tools can be employed, including assessment design, learning activities, feedback to students and class announcements, among others. Online collaboration is a particularly valuable tool (Fiock, 2020) for facilitating interactions and information exchanges among students and between teachers and students (Rovai, 2002; Tallent-Runnels et al., 2006).

Discussion forums, an example of online collaboration, have been widely used to promote higher-order thinking and collaborative learning skills among students (Harman & Koohang, 2005). Students can better understand the lessons after active discussion (Bonwell & Eison, 1991). Online forums allow students and teachers to interact flexibly via a learner-centred, social constructivist approach (Stacey, 2002; Turcotte & Laferrière, 2004). Furthermore, online discussion forums facilitate English as an Additional Language (EAL) students to overcome language barriers and engage with peers in an encouraging and supportive environment (Biesenbach-Lucas, 2003). This is particularly important in classes with high numbers of international students.

In addition to online discussion forums, teachers have increasingly turned collaborative annotation systems to foster social learning communities (Miller et al., 2018). Those systems are built on the foundation of online discussion forums and digital annotation tools (DATs). DATs have been utilized by the teachers for reading tasks to assist students to read prescribed texts (Nor et al., 2013; Thoms et al., 2017; Thoms & Poole, 2017; Tseng et al., 2015). Studies show that DATs can help students with reading comprehension (Nor et al., 2013; Tseng et al., 2015). Perusall (www.perusall.com), a free collaborative DAT, has gained popularity in recent years. It offers a combined platform for DATs and discussion forums, enabling students to annotate digital content and engage with peers' annotations. Perusall also provides automated evaluation of annotations, analytics and a “confusion report” to identify areas where students need support. Studies have shown that Perusall facilitates co-construction of knowledge and scaffolding of learning during reading and annotation activities (Miller et al., 2016; Tian, 2019).

The present study was conducted in a postgraduate course completed by 313 students across two semesters at a large Australian university. Most students are international students who have little prior knowledge of the subject. The cohort sample is representative of the enrolment patterns observed in numerous postgraduate business courses offered at Australian universities. The course was offered in a blended mode before the pandemic, with online lessons and face-to-face tutorials, and was shifted to synchronous online tutorials in early 2020. Perusall was introduced with the aim of enhancing students' pre-tutorial engagement. Weekly tasks included annotating lecture slides, applying theories to real-world situations, raising questions and interacting with fellow students on the platform. This collaborative learning environment was designed to improve comprehension collectively. This research aims to examine the effectiveness of Perusall by addressing the following research questions:

• RQ1. Is there a correlation between pre-class annotation performance on Perusall and post-class assessment outcomes?
• RQ2. Does the English proficiency of EAL students influence their participation and performance on Perusall?
• RQ3. Which aspects of Perusall do students perceive as the most valuable in their learning experience?

LITERATURE REVIEW

Community of Inquiry

To describe the essential elements of online learning and teaching, the CoI framework has been developed for designing online courses to support critical community inquiry by students and teachers (Garrison et al., 2000). It is rooted in the social constructivism theory (Akyol et al., 2009; Akyol & Garrison, 2011), which states that learning is enhanced when students interact with others in a sociocultural context (Oldfather et al., 2003). CoI emphasizes that students are more likely to succeed when their teachers and peers are present and supportive (d'Alessio et al., 2019). CoI framework contains three interdependent dimensions of presence: cognitive presence, teaching presence and social presence (Garrison et al., 2000).

Originally presented as “the most basic to success in higher education” (Garrison et al., 2000, p. 89), cognitive presence refers to students' ability to construct meaning through critical discussion and reflection in a CoI (Garrison, 2011). Cognitive presence is vital for community members to achieve the ostensible goal of higher education: critical thinking (Garrison et al., 2000). Garrison et al. (2001) posited that the cognitive presence includes four phases: (1) a trigger event or communication, where a problem is defined; (2) exploration, where information, knowledge and clarification are explored; (3) integration, where the students construct the meaning of the acquired information and knowledge; (4) resolution, where learners apply the idea or hypothesis as a solution. Then, depending on whether the idea is confirmed or the application is successful, the inquiry process may continue. Researchers have found that cognitive presence is connected to online students' satisfaction (Joo et al., 2011), learning outcomes (Akyol & Garrison, 2011), teaching presence and social presence (Kozan & Richardson, 2014).

Teaching presence is related to the way that teachers design the course, facilitate discussion and direct the class to help students achieve meaningful and worthwhile learning outcomes in a CoI (Garrison, 2011). It contains three components: (1) designing and organizing courses; (2) facilitating the discourse; (3) providing direct instruction. Teachers' active intervention is vital to creating student-centred learning communities in which students and teachers are equal participants (Hilliard & Stewart, 2019). This is when the online learning course becomes a useful and instructional learning resource (Anderson et al., 2001). Teaching presence is related to the perceived learning of students (Arbaugh, 2007), assignment grades (Shea et al., 2011) and student satisfaction (Miller et al., 2014). It is closely related to both cognitive presence and social presence (Kozan & Richardson, 2014) and contributes significantly to the higher levels of cognitive presence (Garrison & Arbaugh, 2007).

Social presence is defined as the students' ability to relate to the group, have purposeful communication and form personal relationships with others in the course (Garrison, 2011). It comprises three categories of elements: (1) interpersonal communication (eg, affective expression, self-disclosure, use of humour); (2) open communication (eg, quarrelling with others, referring to others, questioning, complementing, expressing agreement); (3) cohesive communication (vocatives, addressing group inclusively, salutations) (Rourke et al., 2001). With a high level of social presence, students bring their enthusiasm and experiences into the class and see the whole group as working towards a common goal (Hilliard & Stewart, 2019). Social presence develops trust and fosters a sense of belonging, freedom of expression and cohesiveness in the learning community (Law et al., 2019). A high level of social presence is important for students' perceived learning (Richardson & Swan, 2003), deep and meaningful learning (Rourke et al., 2001) and satisfaction with the course being studied (Richardson & Swan, 2003). Other studies suggest that it holds greater significance than teaching and cognitive presences (Armellini & De Stefani, 2016).

In sum, CoI requires all three types of presence, which interconnect with each other to create a desirable learning environment. Cognitive presence mediates between social and teaching presence and can be applied to students' online learning experience (Kozan, 2016). It suggests that interactions are of the utmost importance in the learning process (Kozan & Caskurlu, 2018). Therefore, this research investigates how to facilitate interactions and promote collaborative learning through Perusall, a combination of an online annotation tool and a discussion forum.

Discussion forum

Discussion forums, also known as “discussion boards” or “message boards,” are an integral part of online learning and a key feature of major online learning platforms (Harman & Koohang, 2005). It has been found that discussion forums are effective in enabling students and teachers to communicate and share knowledge as well as analyse problems (Bradshaw & Hinton, 2004). Students also better understand the learning resources after their participation in discussion (Sorcinelli, 1991). According to the social constructivist theory of learning with technology (Brown & Campione, 1996), conversations among students and between students and teachers are vital to the learning process. The collaborative and social learning settings provided by discussion forums are important for the creation of a CoI.

The asynchronous nature of online forums is another reason for their widespread use in online learning. It provides a place in which the students and teachers can interact when they are free and where it is convenient for them. Moreover, education literature shows that online asynchronous discussions give students additional time to complement and construct their conversations (Gorski et al., 2000). It has also been found that students participate more actively in online debates than in traditional classrooms (Smith et al., 2000). In addition to better participation, students also achieve higher quality of knowledge construction and academic performance (Romero et al., 2013).

Teachers must effectively incorporate online discussion forums into students' holistic learning experience to maximize its usefulness. For example, they need to integrate online discussions into course assessments. Research shows that students may need to be given credits as an incentive to participate in online discussions (Burkett et al., 2004). Teachers have called for alternative asynchronous discussion tools that can meet varied learning goals and facilitate complex learning (Gao et al., 2013). Integrating online discussion forums into DATs can be an effective approach to facilitating student learning (Biro, 2021).

Digital annotation tool

Individuals use annotations as personal records of their reading and interpretations of the material (Nor et al., 2013). As an effective reading tool, annotating improves student comprehension of the reading materials (Simpson & Nist, 1990). With the emergence of online learning, it has become important to adopt DATs to support students' online reading. The design of DATs often includes personalized reading and collaborative learning, such as sharing and interactions within the learning community (Nor et al., 2013). Apart from students highlighting important text, underlining contents they do not understand and writing down their thoughts as well as their questions, they can also read other students' annotations and discuss their ideas with each other. Therefore, students are encouraged to participate in an inviting and interactive environment (Greenhow et al., 2019) and thus enhance their understanding of assigned readings.

Several studies have been conducted to examine the effectiveness of DAT. In the K-12 context, Yang et al. (2013) find that collaborative annotation improves Chinese students' reading skills and fosters higher-level cognitive abilities in primary school. Lu and Deng (2013) suggest that it is a useful tool to support the reading-to-argue process in secondary school. At the university level, its positive impact on student motivation, engagement, learning and community building has been investigated in subjects including statistics (Nokelainen et al., 2005), second language Spanish courses (Thoms & Poole, 2017), teacher education (Nor et al., 2013), EAL courses (Tseng et al., 2015), second language Chinese courses (Thoms et al., 2017), physics (Miller et al., 2016) and politics (Clarke, 2019). With the pandemic dramatically altering course delivery in the tertiary education sector, researchers have been investigating social annotation tools, such as Perusall, in various fields, including philosophy (Biro, 2021), pre-service teacher education (Nel & Marais, 2021) and engineering (d'Entremont & Eyking, 2021).

In summary, DAT has a positive impact on students' motivation, interactions and academic performance. Building upon existing studies, we aim to investigate the effectiveness of collaborative annotations on lecture slides in stimulating students' reflection on their learning after watching lecture videos. Using the CoI framework, we will assess the impact of a collaborative annotation assessment task integrated with online lectures.

METHODS

This study was conducted in 2021 across two semesters of an online post-graduate course in a large Australian university. Every week during the semester (12 weeks in total), students were required to study online lectures and then submit annotations on lecture slides via the Perusall platform. Instructions and examples of the annotations were provided to the students. Students were randomly assigned to groups of 20 individuals, allowing everyone to interact and view each other's annotations. Through the use of Perusall, students engaged in creating personal annotations, responding to their peers' annotations, raising and answering questions, and exchanging their perspectives and ideas. The Perusall exercise was mandatory and due one day before the tutorial. Students' annotations were evaluated by Perusall's algorithm and accounted for 20% of the final grade. Subsequently, students attended weekly synchronized online tutorials. Following each tutorial, they submitted essay answers (30% of final grade) that assessed their comprehension of key theories and concepts by applying them to real-life scenarios.

Students applied annotations to lecture slides as an assessment and learning activity associated with online lecture videos. At the end of each semester, we administered a survey to gain an understanding of students' experience with Perusall. The survey measured student experience on a five-point Likert scale (from strongly disagree (1) to strongly agree (5)). Students were also given an opportunity to provide qualitative feedback such as what they benefited from Perusall activities and where could be improved. The open-ended questions allowed us to obtain more insights into students' unique experiences.

In addition, demographic information including students' gender, age and how they met the English requirement at the time of admission were collected from the university database. Finally, data on students' engagement and grades from Perusall, as well as other assessment grades, were collected from the Perusal platform and the Learning Management System (LMS), respectively.

Participants

In total, 313 post-graduate students who completed the assessments in semester 1 and semester 2 of 2021 participated in this study. Among these participants, 297 were international students (female: 168, male: 129), and 16 were domestic students (female: 7, male: 9). Notably, approximately 80% of the international students (239) were from China. All participants were enrolled full-time, and the average age of the cohort was 23.82, ranging from 19 to 34 years.

For data analysis pertaining to RQ1 and RQ2, both pre-class annotations and post-class assessments from all participants were utilized. In addressing RQ3, a questionnaire was administered to gauge the students' learning experiences with Perusall at the end of the semester. A total of 77 students (24.60%) completed the questionnaire, comprising 74 international students and 3 domestic students. Within this sample, 48 were female, 28 were male and 1 student chose not to disclose their gender. No incentives were offered to the participants. The findings from the questionnaire were subsequently employed to address RQ3.

Data analysis

To address RQ1, we conducted multiple regression analysis to examine the relationship between the quality of students' weekly pre-class annotations in Perusall and their performance on post-class assessments. For RQ2, we used independent t-tests to compare the participation of EAL students on Perusall and their performance on assessments across different English proficiency levels. To address RQ3, we used a combination of qualitative and quantitative methods. One-sample t-tests were used to investigate students' experience with online annotations. The qualitative data from open-ended survey questions were thematically coded using the inductive approach (Braun & Clarke, 2012). Through this analysis, we identified the specific elements of online collaborative annotation activities that students found most valuable.

RESULTS

The descriptive statistics and correlations of the variables used in this study are presented in Table 1. The dependent variable is post-class assessment performance, and the independent variables are pre-class annotation grades, number of annotations and ratio of follow-up annotations.

Figure 1 displays students' weekly participation rate in Perusall. On average, 91 percent of students annotated the lecture slides from Week 2 to Week 12, considering that Week 1 was a trial task and not assessed.

RQ1. Is there a correlation between pre-class annotation performance on Perusall and post-class assessment outcomes?

To address RQ1, we developed a multiple regression model to predict students' weekly post-class essay performance using three predictor variables: pre-class Perusall annotation grades, the number of annotations made by students per week and the ratio of annotations engaged in discussion with peers. The regression model is statistically significant: F(3, 309) = 75.62, p < 0.001 and the combined predictor variables account for 41.8% of the variance in students' weekly post-class essay performance (Table 2). In Table 2, the unstandardized coefficients (second column) indicate the partial effects of the predictors on students' weekly post-class essay performance, while the standardized coefficients (third column) show the relative weights of each variable when compared with each other. The results in Table 2 reveal that the average Perusall grades, the number of annotations per week and the ratio of follow-up annotations per week are all significant factors that influence students' weekly post-class essay performance.

In particular, Perusall grades are the most significant factor affecting students' average weekly post-class essay performance. The standardized coefficient for average Perusall grades is 0.55 (p < 0.001), indicating that a unit increase in Perusall grades corresponds to 0.55-unit increase in students' weekly post-class essay performance, while holding other variables constant. Additionally, the number of annotations per week also has a significant and positive relationship with students' weekly post-class essay performance (β = 0.12, p < 0.05). One unit increase in annotations per week corresponds to 0.12-unit increase in students' weekly post-class essay performance. By contrast, ratio of follow-up annotations has a negative relationship with students' post-class essay performance (β = −0.09, p < 0.05). One unit increase in ratio of follow-up annotations per week corresponds to 0.09 unit decrease in students' weekly post-class essay performance.

The regression results suggest that students with higher grades and more annotations on Perusall performed significantly better in their weekly post-class assessments. However, students who wrote a higher ratio of follow-up annotations to other students' annotations performed worse in their post-class assessment.

RQ2. Does the English proficiency of EAL students influence their participation and performance on Perusall?

EAL students must meet the English entry requirement when they are admitted by the University. Most students in this course undertook the Master of Business degree. The EAL students were considered meeting the English requirement if they completed their secondary or undergraduate education in English, or received the minimum score established for the English language tests. For example, they must score 6.5 overall on the International English Language Testing System (IELTS), with minimum band scores: Listening 6.0, Reading 6.0, Writing 6.0 and Speaking 6.0. If they fail to meet the minimum standard but receive a score above a lower standard (eg, 5.5 overall in IELTS, with minimum band scores: Listening 5.0, Reading 5.0, Writing 5.0 and Speaking 5.0), they can be admitted to a postgraduate course on condition that they complete the University's English bridging program.

Based on their English test results and how they met the English entry requirement, international EAL students were divided into two groups. Students who had IELTS score between 5.5 and 6.4 were put into Group A (ie, low English proficiency students). Students who had IELTS scores 6.5 or higher, or completed secondary or undergraduate study in English, were put into Group B (ie, high English proficiency students). Independent samples t-tests were used to compare Perusall activities and assessments results between these two groups of students. The results were presented in Tables 3 and 4.

As shown in Table 3, Group A (low English proficiency) students wrote more words per annotation than students in Group B (high English proficiency) (t = 3.61, p < 0.001). Specifically, Group A students wrote 86 words per annotation whereas Group B students wrote 75 words. There is no significant difference on other Perusall activities between the high and low English Proficiency students.

The best 10 weekly Perusall and post-class assessment results were used to calculate the student final grade. The independent-samples t-test results in Table 4 show that Group A students received lower marks on the individual proposal (t = −2.41, p < 0.01) than Group B students. Mean score for low English Proficiency students = 13.68, and for high English Proficiency students = 14.32. However, there is no statistical difference in their weekly Perusall performance and weekly essay questions.

RQ3. Which aspects of Perusall do students perceive as the most valuable in their learning experience?

Figure 2 depicts the survey results of students' experience with Perusall (N = 77). An analysis using the one-sample t-test revealed that the mean scores for the following items significantly differ from the middle-value 3 at a significance level of p < 0.05:

1. Using Perusall helped students better understand the lesson's content (M = 3.92) and prepare for the weekly assessments (M = 3.85).
2. Students made great efforts into making annotations (M = 3.90) and producing quality comments (M = 4.20).
3. Students found Perusall easy to use (M = 4.05).

The thematic findings from the students' open-ended answers reveal what students value most in their Perusall activities: (1) cognitive benefits; (2) social interactions with other students; (3) receiving teaching staff's feedback; (4) user-friendly and flexible interface.

Firstly, students believed that their cognitive processes were enhanced through critical reflection on their own understandings or by learning from the perspectives of other students. Here are some selected quotes:

This led me to think more critically about the themes and concepts of the lessons.

When I listen to the lecture, it triggers my thought process and I can share that on Perusall.

Students give various applications of the concepts and help each other to get better understanding of the materials.

Secondly, students enjoyed social interactions with other students, engaging in conversations and expressing their viewpoints. They believed that such interactions enhanced their education, as illustrated by the following quotes:

The best thing I like is to communicate with classmates and teachers on Perusall, because it makes me understand the lesson much better. Specifically, I can ask what I don't understand wherever I have problems, and classmates give me some help on Perusall.

You can share your views with students anytime, anywhere.

Furthermore, students appreciated the activities because they could receive “feedback from teachers on my comments” and Perusall enabled them to “interact between students and professors.” However, students suggested that greater teacher presence would improve the learning activity. They mentioned that it was sometimes difficult to determine the accuracy of other students' annotations. As one student stated: “It would be better if the teacher can highlight any excellent responses each week.”

Lastly, students highly appreciated the user-friendliness and flexibility of using Perusall. The platform's ease of use allowed them to focus on their own learning, and students praised that “Perusall's operation is very convenient, even for the first time I use it.” The asynchronous and interactive features provided them with the flexibility they needed. One student wrote:

Sometimes we miss out on important ideas when we are in a time restricted learning session. I feel Perusall has the potential to become a place where each student can engage with a topic to their capacity and time is not an obstacle for engagement.

On the other hand, students also reported facing challenges during their experience. They mentioned the quality of their peers' annotations, teachers' facilitation and technical difficulties, as shown in the following quotes:

Some of them just copy and paste definitions, and are not interested in engaging in relevant discussions.

It would be good if our professors can correct or label the wrong/misleading annotations.

Sometimes I cannot select the knowledge points that I want to choose.

It needs to be an app which can provide notification on phone to show what's happening.

Table 5 shows how the qualitative data were thematically coded using an inductive approach (Braun & Clarke, 2012). The “Example” column in Table 5 presents students' opinions obtained from the survey, while the “Code” column displays the first-level codes applied to each quote, along with the respective counts of coded responses. Finally, the “Theme” column encapsulates the aggregate-level theoretical and practical insights derived from the findings. We found that 42% responses focused on cognitive benefits. Additionally, 39% of the replies pertained to social interactions, with only 8% discussing teacher presence and 11% addressing user-friendly technology. Regarding the challenges encountered while using Perusall, the majority of students (57%) commented on either technical difficulties or expressed a desire for new features. Additionally, 23% of students advocated for more effective teacher facilitation, while 20% mentioned concerns about the quality of other students' annotations.

TABLE 5. Findings from open-ended questions.ExampleCodeThemeWe can give various applications of concepts and help each other to get better understanding of the materialsLearning by reading other annotations (N = 17)Cognitive benefitsIt helps me understand each topic more deeplyGenerally improving understanding (N = 10)I can learn beyond the material by doing my own research (eg, examples supporting the theory)Better learning by doing my own work (N = 5)Opportunity to interact with the classmates in the virtual environmentDiscussions with other students (N = 21)Social interactionsI can share my idea with my classmatesExpress my own voice (N = 9)It could be a good way to interact between students and professorsReceiving feedback from academics (N = 6)Teacher presenceEasy to annotate and useEasy to use and interactive (N = 7)User-friendly technologySometimes we miss out on important ideas when we are in a time restricted learning session. I feel Perusall has the potential to become a place where each student can engage with a topic to their capacity and time is not an obstacle for engagementFlexibility (N = 1)Some students just write several words or one sentence to finish their Perusall assignment. Actually, I want to have more deep communication with our classmatesQuality of students' annotations (N = 6)ChallengesIt would be better that the teacher can highlight any excellent responses in each weekTeacher's facilitation (N = 7)It has got a slow reaction between each pageTechnical issues/function improvement (N = 17)

DISCUSSION

This research examines students' engagement with a pre-class social annotation tool and links it with their performance in other assessments. It shows that the use of the social annotation tool resulted in a high rate of completion for pre-class learning activities. Between Week 2 and Week 12, when the annotations were assessed, an average of 91% of students completed the tasks. This number aligns with the high participation rate (90%–95%) reported in previous research on the use of Perusall for pre-class text reading assignments (Miller et al., 2018) and the participation rate (90%–100%) reported in pre-class video watching assignments on Perusall (Sigmon & Bodek, 2022).

Group conversations constituted a significant portion of students' annotations. Replies to other annotations accounted for almost half (49%) of all the annotations made. As demonstrated by CoI research, the Perusall assignment provided opportunities for student-to-student interactions and fostered open channels of communication (Fiock, 2020; Richardson et al., 2012). In addition, the social presence facilitated by student interactions encourages peers to participate in pre-class learning activities.

RQ1. Is there a correlation between pre-class annotation performance on Perusall and post-class assessment outcomes?

Regarding the association between students' online annotations and their performance on subsequent assessments, we found that students who produced better quality annotations and made more of them before class tended to perform better on the post-class essay questions. This short-term positive effect of Perusall on student academic performance is a novel finding. The result also indicates that students are more likely to perform poorly on the weekly post-class assessment if they write a greater proportion of follow-up annotations with others. One explanation is that high-achieving students begin working on Perusall early in the assessment period and complete their annotations before others. They are likely to have greater sustained attention and comprehension compared to those with low self-regulated learning skills (Chen & Huang, 2014). Thus, these students are more likely to provide new annotations in the documents rather than merely responding to existing ones.

Our findings suggest that designing pre-class activities is critical to students’ success in online learning. Social interactions during annotation activities improve students’ engagement and understanding of learning materials in a way that traditional individual learning cannot (Thoms et al., 2017). A pre-class learning environment that fosters connectedness and support among students and teachers can enhance students’ engagement with activities and enhance learning outcomes.

RQ2. Does the English proficiency of EAL students influence their participation and performance on Perusall?

We find that EAL students with low English proficiency wrote longer annotations (measured by words) than EAL students with high English proficiency. EAL students with low English proficiency received slightly higher grades on Perusall than their high-proficiency peers, but the difference was not statistically significant. Consistent with previous literature, students with low English proficiency received lower grades in the major individual assessment (Eddey & Baumann, 2010; Hartnett et al., 2004; He & Banham, 2009). This finding suggests that online collaborative annotation, as a short and low-pressure assessment, could be a comfortable and adaptable platform for students with low English proficiency to succeed. This could promote inclusivity and create a level playing field for disadvantaged students.

RQ3. Which aspects of Perusall do students perceive as the most valuable in their learning experience?

The survey results provide further insights into how students perceive and value specific learning experiences within online annotation activities. The themes that emerged from the students’ answers align with CoI's cognitive, social and teaching presences.

The cognitive benefits have been recognized by students. They reported that annotating lecture slides after watching lecture videos helped them comprehend the material better and prepare for assessments. There are two reasons for this enhancement: (1) students can learn from other students' annotations; (2) when students engage in discussions or respond to others' questions, they take an active approach and subsequently achieve a better understanding of the learning materials.

Facilitating social presence, Perusall enables students to study course material collaboratively rather than individually. Interpersonal, open and cohesive communications establish a strong social presence in a collaborative learning environment (Fiorella & Mayer, 2013). It is known that such social presence has a positive impact on online learners' cognitive presence (Huang et al., 2019), which is associated with perceived and actual learning outcomes (Akyol & Garrison, 2011). This encourages students to make greater efforts and produce high-quality work. Furthermore, students highly valued demonstrating and communicating their own comprehension and viewpoints with others. Online platform interactions can serve as a crucial tool to prevent disengagement and feelings of isolation among students, particularly during challenging times such as the pandemic. Lastly, when asynchronous social annotations tools are easy to use, students can engage in the interactive social learning process conveniently at their own pace. As a result, social annotation tools have a unique advantage over other DATs that do not allow meaningful user interaction.

However, the teaching presence may need to be enhanced in this collaborative learning, as students expressed a need for increased teacher presence in Perusall. As recommended by the CoI framework (Garrison, 2011), teachers should become involved, facilitate discourse and provide instructions to make online learning resources more useful and instructional. One obstacle that hinders instructors from actively participating and responding to students' annotations is the significantly increased workload, particularly for larger cohorts of 200 students, generating around 2000 annotations per week, with a minimum of 7 annotations per assignment (Some students made more than 20 annotations in a week!). To address students' questions and concerns, the chief examiner for the course (one of the authors) reviewed the confusion report generated by Perusall and selected relevant questions for discussion during tutorials as a compromise.

Overall, we found that the collaborative online annotation activities can facilitate CoI. First, in terms of cognitive presence, students constructed their knowledge and understanding of pre-class learning through critical reflection and discussion on the online annotation platform. This cognitive presence was related to the students' perceived learning and satisfaction. Second, social presence was built through students' discussions and interactions, helping to develop social presence. It fostered a sense of group coherence among the students, which in turn had a positive impact on their experience of the learning activities. Finally, teaching presence is most effective when students can observe their teachers' responses to discussions and inquiries. Students prefer to see teachers respond to their annotations rather than providing direct instruction or initiating the conversation. With teaching presence, students' knowledge construction process in the cognitive presence can be reinforced, and their open communication in the social presence can be more conducive to learning. Hence, teaching presence serves as a catalytic agent to improve the levels of cognitive and social presence.

Although there are several benefits to the online annotation platform, we also identified some challenges associated with it. Students reported instances of their peers attempting to manipulate the scoring system by either rephrasing or copying contents from elsewhere, which did not contribute to a better understanding of the lesson material. Therefore, teachers need to consider manual auditing or adjusting student scores. In the future, Perusall should notify instructors of suspicious annotations (eg, copied from the Internet) so that they can follow up and take necessary actions. In addition, students reported technical glitches and expressed expectations for new functions. Social annotation tools should continue to improve to facilitate even smoother experiences for students to engage in, both on computers and mobile devices.

Perusall's efficacy is optimized when students log in multiple times throughout the assignment period. This allows them to engage with others' questions and responses while also reviewing the feedback on their own annotations. If students complete all of their annotations in one take, their opportunities for interaction with others are limited. Our suggestion for future design of social annotation learning activities is to motivate students to engage with the platform at various intervals and increase their social presence.

CONCLUSION

This study shows that online collaborative annotation activities can effectively engage students in pre-class online learning activities. The quality of students' pre-class annotations has a positive influence on their post-class assignment performance. Notably, EAL students with different levels of English language proficiency performed similarly in collaborative annotation assignments, indicating that online collaborative annotation activities offer an inclusive environment for students with low English proficiency to thrive. In addition, students highlighted several positive aspects of their online annotation experiences. According to these findings, the pedagogical influence of teacher presence is highlighted by the students in the augmentation of their cognitive presence and the facilitation of open communication in the social presence. Therefore, teachers are encouraged to facilitate, promote and participate in high-quality communication by engaging students with online learning resources.

Moving forward, teachers should consider designing assessment tasks on collaborative platforms to create a more inclusive environment for EAL students. By doing so, teachers can enhance students' learning experiences and promote their engagement with online learning resources. Overall, this study highlights the benefits of online collaborative annotation activities in promoting engagement and enhancing learning outcomes.

LIMITATIONS AND FUTURE RESEARCH

Despite the promising results, this research has several limitations that must be acknowledged. The course design and the small cohort size limit the generalizability of the findings. Also, the high value of grades (20%) assigned to the annotation activities might influence how students engage in the pre-class learning. Therefore, future research can consider implementing a control group with a larger sample size to validate the impact of online annotation participation on student learning outcomes. Furthermore, this study did not evaluate the students' prior knowledge level, which may have influenced their learning performance in the course. In addition, there was no analysis of the types of annotations made by the students, which could provide further insights into their learning process. Future research can investigate the content of student annotations to better understand their contribution to learning outcomes.

ACKNOWLEDGEMENT

Open access publishing facilitated by Monash University, as part of the Wiley - Monash University agreement via the Council of Australian University Librarians.

FUNDING INFORMATION

No funding was received in connection to this research.

CONFLICT OF INTEREST STATEMENT

There are no conflicts of interest in this study.

ETHICS STATEMENT

This research was approved by Monash University ethics committee (approval number 16702), prior to the commencement of the data collection.

‍