Like many college faculty, when the COVID shutdown hit during Spring 2020, I was unsure how best to transform my courses for a suddenly fully online format. When my colleagues Michelle Reidel and Heather Scott suggested Perusall, I used the disruption, like many others, as an opportunity to try something new. The Perusall platform worked well, and most of m students were enthusiastic about the chance to complete some of their coursework in an online social media setting. Of course, this was not true for all students, particularly those who had been avoiding reading. Nonetheless, by the end of 2020, most of my courses presented most of their content on Perusall.

Four years later, some students are not responding as enthusiastically. Perusall is now much more common on our campus, especially in my college’s teacher-training programs. Although our College of Education accounts for only 12% of our total university enrollment,¹ our college disproportionally accounts for 20-25% of total Perusall use on campus.²

Both informally and via official course evaluations, my students, especially my undergraduates, have been increasingly complaining about having to spend what seems to them an inordinate amount of time working through and responding to materials on Perusall. Thus, the question arose: how much time are my students actually and actively spending on my courses’ Perusall sites? And, when added to the time³ needed for other course assignments, am I expecting too much?

Our university’s credit hour policy states, “there is an expectation that students spend a minimum of two hours on coursework outside of class for every hour spent in class.”⁴ This means that students in my face-to-face undergraduate courses, all scheduled for 2.75 hours of class time per week, should expect approximately 5.5 hours of outside work per week. Meanwhile, as I interpret this policy, my asynchronous graduate courses, which do not have any mandatory class meetings,⁵ should require approximately 8.25 hours of total work time. Generally, I would like my students’ outside-of-class time to be approximately evenly split between reading (and other Perusall content) and assignment work. This yields an expectation of 2.75 hours of Perusall time per week for undergraduates and 4.13 hours for graduate students.

With IRB approval, I pursued a quick study of all my courses that included a significant Perusall component from Summer 2020 to Fall 2023. Using data stripped of any identifying information, I compiled and compared, overall and course by course, my individual students’ final grades with Perusall’s count of their total active time on my course websites.

In order to establish some reliability for analysis, my first question was, how consistent or inconsistent have my Perusall expectations and my student’s performance been since 2020? I have repeatedly modified the role of Perusall in my courses by adding or subtracting materials, altering the scoring algorithm, and/or changing the weight of Perusall in the final grade. The following graph⁶ shows that it took until the beginning of 2022 for me to settle into a set of structures and expectations that yielded relatively consistent levels of student active time across semesters.

This visual demonstrates that it took me more than two years to settle into a relatively consistent level of expectations for my students’ active time on Perusall. Thus, the remaining data analysis will focus on the last two calendar years (2022 and 2023), starting with Spring 2022 (2022-1). This reduces my sample size to n=299 students across 16 course sections, as described in this table:

Note that only 27% of my recent teaching load is with undergraduates.¹⁰ Also, many of my courses (both Language Education Policy/Sociolinguistics and Cultural Diversity) serve a combined population of graduate and undergraduate students. The graduate student versions of these courses' syllabi require additional and higher-level materials on Perusall and additional and/or more complex assessments.

Looking across all sections and semesters, the following graphs demonstrate that during 2022 and 2023, the active time spent on Perusall generally correlates with my graduate and undergraduate students’ final grades.¹¹

However, the question remains: how much active time were most of my students spending on Perusall each week? As the following bar graphs show, the answers vary substantially between my graduate and undergraduate students.

Among my graduate students, 62% have spent 1.5 to 3.0 hours per week actively engaged in Perusall. In addition, though, a long tail above 3 hours per week includes almost 30% of my graduate students. The graduate student average is 2.8 hours per week and 3.0 hours for ‘A’ students. Both figures are substantially lower than my previously stated goal of just over 4 hours for graduate students. However, this gap is partially mitigated by the long tail of higher amounts
of time.

Among my undergraduates, 70% fell between 1 to 2.5 hours of active time, with only 6% putting in more than 3 hours per week. Their average is 2.0 hours per week and 2.3 hours for ‘A’ students. Again, both are also lower than my previously stated goal of about 2.75 hours for undergraduates.

Thus, for both groups of students, these statistics support an assertion that my students are not spending an excessive amount of time working with the assigned Perusall materials. In fact, it appears that, for both groups, I could be justified in increasing their Perusall workload by approximately 20% for undergraduates and 33% for graduate students to bring their average active time up to my goals.

A side note, given the relatively small number of high-end outliers, this data does not provide evidence that many students are using hardware or software ‘mouse jigglers’¹² that try to deceive the algorithm into thinking someone is moving the mouse and reading the screen – and/or – it indicates that the Perusall algorithm is effective in removing such outliers from its active time calculations.

Before pursuing any modifications to my assigned Perusall workloads, it is critical to see if there is a gap when comparing the data from specific courses. However, the enrollment numbers (only 15 students) in my Language Education Policy/Sociolinguistics course are, prima facie, too small to justify any strong claims about the course’s expectations for active time on Perusall.¹³ Since this only leaves undergraduate data from my Cultural Diversity course, this also eliminates any cross-course comparisons of undergraduate workloads. This leaves only one possible comparison at this time.

Comparing my two graduate courses with substantial enrollment numbers, Children’s Literature students averaged only 2.6 hours and 2.8 hours for an ‘A.’ Meanwhile, Cultural Diversity students averaged 3.0 hours, with 3.3 hours for an ‘A.’¹⁴ Again, these figures are well below my earlier stated goal of about 4 hours per week for graduate students. The following graphs provide more details regarding this gap.

Thus, going forward, this analysis of and reflection on my students’ active time on Perusall yields the following action steps:
• For the moment, I will not worry about ‘jigglers’ being able to game the Perusall scoring algorithm successfully.
• While explaining my expectations for Perusall, I will likely share a summary of this data with my undergraduate students at the start of next semester.
• I will use this data, as appropriate, to reply to students complaining of spending excessive amounts of time completing the Perusall elements of my courses.
• Since most of my courses fall short of my active time goals for my students, I will consider raising the Perusall workload in most of my classes, especially my graduate courses. (Excepting Language Education Policy/Sociolinguistics. See endnotes 8 and 13.)
• To make my expectations more consistent across courses, I will add more materials (most likely at least one more exemplary children’s picture storybook) to each week of my graduate Children’s Literature course.

I hope that this model of how to examine my students’ Perusall data can help others critically consider their expectations for their students. In addition, I would suggest to the Perusall.com team that adding more direct access to this type of data and some simple analysis tools would benefit everyone who is using Perusall in their courses.

Written by Dr. Scott A. Beck, Georgia Southern University

________________________________

Endnotes

¹ https://em.georgiasouthern.edu/ir/enrollment_interactive/
² Personal correspondence with Annette Ramos, Georgia Southern University Learning Technology Support office.
³ Note that this study does not address the available data for each student’s number of comments because the essential data regarding the average quality of those comments is not easily extracted from Perusall.
⁴ Emphasis added. https://digitalcommons.georgiasouthern.edu/cgi/viewcontent.cgi?article=1004&context=provost
⁵ All include bi-weekly, optional, extra credit Zoom discussions.
⁶ Note that the dotted lines indicate interpolated data for semesters when none of my students earned a low grade in the course. These numbers are small because I strongly encourage students on the path to a ‘C’ or lower to drop
my courses before the deadline for withdrawal without academic penalty. See also endnote 11.
⁷ Georgia Southern’s Banner/Wings Enrollment management website is not publicly accessible.
⁸ Due to low enrollment during Spring 2023, my College of Education Language Education Policy course was combined with also low-enrolled sections of graduate and undergraduate Sociolinguistics, normally taught in the College of Arts and Humanities. This merger of the two courses’ worth of content is likely responsible for the high active times noted in endnote 13.
⁹ Most semesters, when I teach Cultural Diversity, I combine my graduate and undergraduate sections. My graduate students bring maturity, timeliness, and classroom experience to the conversations. Meanwhile, my undergraduates bring more iconoclastic and progressive perspectives to the course materials. This challenges both groups to achieve more than they might otherwise.
¹⁰ This has not always been true because, like many others, the majority of my courses during my first decade as a faculty member were for undergraduates.
¹¹ This is expected since their Perusall scores are a substantial element of their course assessments. Note though, that the small numbers of students earning low grades, explained in endnote 6, yields some anomalous data. This correlation was also found by Virginia Clinton-Lisell in her October 2023 article in Research in Learning Technology. She states, “only active reading time reliably predicted course grade when controlling for all positively associated variables.” https://blog.perusall.com/social-annotation-students-perceptions
¹² For more information about ‘mouse jigglers’ see: https://www.timedoctor.com/blog/how-to-detect-mouse-jiggler/
¹³ The averages for the Spring 2023 combined Language Education Policy/Sociolinguistics are 4.6 and 4.2 hours for graduate and undergraduate students, respectively. These numbers are substantially higher than my other courses, indicating a possible need to reduce the course’s reading load, especially for undergraduates. See also endnote 8.
¹⁴ This is problematic because Children’s Literature is a 7000-level course, while graduate Cultural Diversity is a 6000-level course.

‍