The latest PiP evaluation "strand report" has recently been published (WP7:38 Impact and process evaluation - Piloting of C-CAP: Evaluation of impact and implications for system and process development). This strand of the evaluation attempted to understand the impact of the Class and Course Approval Pilot (C-CAP) system within specific stakeholder groups and sought to understand the extent to which C-CAP was considered to support process improvements. As the report explains, many of the findings were positive and the data collected supported many of the findings from previous evaluative strands. System perceptions were generally positive across stakeholder groups. C-CAP was found to:
- clarify expectations
- support simplicity in the drafting process
- improve process transparency and visibility
- enhance process control, and
- solve a variety of document management and workflow issues.
C-CAP's renewed focus on those aspects of curriculum design that are integral to good pedagogical practice and to high academic quality standards has reduced the bureaucratic burden on academics and relieved strain on the academic review process. This has simplified academic review and has delivered a system that is less likely to stifle innovation. A central repository of curriculum designs as a place to curate "knowledge assets" was also considered to support a number of key academic quality processes, as well as better enabling responsive curriculum design. Data supported the view that C-CAP functioned as a platform from which to disseminate explicit and tacit curriculum design practice, which would maximise the value of institutional knowledge assets by enabling the re-use of curriculum designs.
However, among some of the more problematic findings of this strand of evaluation was evidence of systems resistance among some academics. Tech-supported approaches to curriculum design and approval are generally predicated upon the assumption that the most difficult part of stakeholder buy-in is persuading them of the benefits, whether those benefits are manifest in a more efficient approval system or better support for academic quality; but rarely is system acceptance itself discussed as a "live" issue. It always appears to be assumed that the system itself – and by "the system" we are referring to the "tech-support" in "tech-supported curriculum design and approval" – will be automatically accepted by stakeholders and that it is the consequences of the system, not the system itself, that requires our careful attention. Previous evaluation activity corroborates this general view and focuses our advocacy efforts; yet, despite a generally well received system, a small but vocal minority of academics demonstrate system resistance. Why? The answer lies in MS Word.
During the latest strand of evaluation activity a preference among some academics for adhering to MS Word as the basis for the curriculum design and approval process was expressed. As the strand report documents, this perspective appears to have strong links to previous working practices and the comfort many have for everyday software applications. And this preference among academic participants for using MS Word was not merely theoretical but was also observed during C-CAP piloting. For example, C-CAP enables the upload of attachments at specific sections of C-CAP. Depending on the context, these attachments may contain detailed financial forecasts or additional academic information (i.e. information not embedded within the system). Piloting of C-CAP revealed unexpected system use behaviour whereby certain sections of course and class proposals were left incomplete; instead, the requested curriculum information was contained in a number of separate MS Word attachments, uploaded at various sections of the proposal. The consequence of such system behaviour for the approval processes C-CAP supports is catastrophic. Important curriculum information or data cannot be captured in a structured manner, thereby compromising subsequent information extraction and reuse and subverting the underlying process. This, according to M. Lynne Markus (a seminal academic in the area of information systems research), is the classic characterisation of "system resistance", i.e. "…when a person engages in behaviour that may result in the disruption or removal of a system that is interdependently used by others as well as that person".
The issues surrounding collaborative working with single-user applications (such as MS Word) has been extensively reviewed in the literature, particularly by researchers focusing on their use within business and organisational contexts. For example, some researchers summarise the problems intrinsic to collaborative working with single user applications, highlighting a lack of collaborative transparency (e.g. understanding the activities of others to avoid duplication of – or neglecting - work) and version control as particular issues. Others note the inefficient and "error prone" use of MS Word and email in the collaborative design of ethics protocols. Whilst better integration of single-user applications is now offered by document management and sharing platforms (e.g. MS SharePoint), the information and data contained within any uploaded documents often lacks structure and therefore evades most types of extraction or computation. C-CAP has been expressly designed to resolve the information management issues that commonly result from unstructured information, poor version control and inadequate collaborative working mechanisms.
Information systems resistance has been investigated by scholars for decades and is often cited as the principal cause of many system implementation failures. System resistance is generally viewed using a series of theoretical perspectives:
- People-orientated theory: The people-orientated theory suggests that system resistance is provoked by certain internal factors peculiar to the groups or individuals exposed to the system. There is evidence, for example, that user characteristics (e.g. age, gender, etc.) as well as the differing values and belief systems held by users can influence the acceptance of a new system or technology.
- System-orientated theory: The system-orientated theory assumes that resistance is created externally by the system or its design. For example, previous research has noted that resistance often increases when users are presented with a technically deficient system, and our previous heuristic evaluation and user acceptance testing of C-CAP was partly to address subsequent user resistance issues.
- Interaction theory: The interaction perspective attributes the cause of resistance to be within a complex web of factors relating to people and systems and the way in which they both interact. Fundamental to this view is that systems can assume wildly different social and political meanings and their consequences are interpreted differently depending on the user’s role or position. Lynne Markus provides indicative examples of the interaction perspective in practice, such as those systems that are resisted on the basis that they centralise control over data within an organisations that otherwise demonstrate decentralised structures, or systems that change the balance of power within an organisation such that it is resisted by those who lose it.
The love some academics appear to have for the continuation of an MS Word/email based system is clearly an issue that should not be overstated; it has emerged as an issue for only a minority of academic participants throughout the entire PiP evaluation programme. User acceptance testing has also confirmed that, by and large, system perceptions are positive. It is nevertheless a significant minority and - as the latest strand report explains - appears to be borne out by the interaction perspective of system resistance, in which the system is perceived to benefit administrators at the expense of academic freedom. This view appears to be corroborated by anecdotal observations of academic quality processes at a number of faculties whereby it was not uncommon for incomplete or substandard curriculum designs to be submitted for faculty consideration. Designs often followed no particular template, omitted key information (e.g. number of student contact hours, resource implications, constructive alignment, etc.), and were left for academic quality teams to "sanitise". Due process was also occasionally subverted at the behest of senior academics.
The design process under the previous state therefore afforded some academics significant freedom in the curriculum design process, and this freedom no longer exists in the new state. C-CAP seeks to standardise curriculum designs and centralise data. C-CAP also seeks to improve process transparency, visibility, monitoring and control which, in turn, also renders process subversion more difficult (although, as we noted above, not impossible!). Yet, the perceptions of some participants clearly related more to people-orientated theories of system resistance; a comfort with familiar working practices and with single-use applications (e.g. If it had just been Word documents, I would have had my various bits and pieces up on the screen), personal attachment to physical documents (e.g. Call me a Luddite but I like having documents) or a lack of training or IT literacy (e.g. I'd like to be able to lift stuff out so that when I create course materials I can easily lift bits and pieces out).
The purpose of this lengthy parable is simply to illuminate what, for many projects in the Institutional Approaches to Curriculum Design Programme, is the thin end of the wedge. The emphasis on enhancing curriculum design, improving the efficacy of approval processes, creating mechanisms for responsive design, effecting organisational change, and so on; these ambitions are completely futile if the fundamental vehicle for achieving them is resisted, or worse, rejected by its stakeholders. The unspoken assumption that stakeholders will generally accept whatever technical widget is used to deliver tech-supported curriculum design and approval is misguided and - given its influence on whether system implementation will be successful or not - is extremely dangerous. The potentially apocalyptic scenario for PiP might be that a significant minority of academics resist or subvert C-CAP, thus disrupting the majority of other stakeholders whose sub-processes depend on it, or deluging overworked academic quality teams with MS Word documents that academics have instructed them to input into C-CAP on their behalf. Such an apocalypse will be averted because the PiP team are now cognisant of system resistance issues. Pre-emptive measures have already been taken and a user acceptance strategy is currently in development. Indeed, it is worth noting that researchers propose a number of system acceptance strategies.
This blog post was originally entitled, “Resistance is futile or Rage against the machine?”; but this painted an erroneous and dichotomous picture, in which the stakeholders and the system implementers were railing against each other while engaging in the blame game of "who-is-responsible-for-hindering-progress-in-curriculum-design". And that's precisely the point. It's nobody's fault because the truth is, as we have seen, the question of systems resistance is far more complicated than that.