Multimodal design, learning and cultures of recognition

https://doi.org/10.1016/j.iheduc.2011.12.003Get rights and content

Abstract

In this article, a design-oriented, multimodal understanding of learning will be outlined. There seems to be a need for a new conceptualisation of learning in an era characterised by an increasing virtual space, blended media and new communicative patterns. This means a broader understanding of learning, and a theoretical understanding of transformational and interactive processes, where meaning-making, the role of agency, and what is seen as proper representations and as signs of learning in a cultural context are central issues.

Introduction

The digitised media have in many senses, and rather rapidly, changed communicative patterns and access to information. However, not only single, factual statements are easy to find. Also more elaborated overviews, descriptions and definitions in different fields of knowledge are easily accessible. Furthermore, people can be in constant connection with each other, and use digitised media for low intensive small talk. It is possible to share whatever opinion with others, with people that might be unknown. And it is also possible to produce music, films and reportages, and likewise share this with “the world” by using open channels like YouTube.

These changes mean that information “of value” comes from many sources, and the ways to “learn” new things differ a lot from the experiences of older generations and their ways of communicating and sharing information. It changes the role of schooling, and it challenges the trust in school-knowledge. What is learnt outside school to an increasing degree also affects the learning in schools (Stocklmayer, Rennie, & Gilbert, 2010). To understand this new communicative pattern, it is not enough to rely on verbal text only, may they be written or oral. Other modes also come into play to handle information, share experiences, as well as to learn new things. The multi-modal character of communication therefore has to be understood.

In many instances, planning for learning has, to a large degree, been the same as the understanding of learning outcomes. This view is also challenged today, not the least since so much learning takes place in so many different arenas and contexts (Stocklmayer et al., 2010). To plan for, or design, educational settings is but one aspect. The other aspect is about the kind of learning that de facto takes place, and how that is recognised within the culture of given context like a school level or the like.

It is also the case that the planning for teaching in many countries has been a centralised practice, leaving the professional teacher to fulfil the goals. However, today teachers, like many other professionals, are asked to take part not only in planning processes but also in assessment practices. They have become designers. Pupils/students are asked to take a greater responsibility for their own learning. They too have become designers, of their own learning practices.

An established understanding of design is to see it as the forming of ideas, concepts and patterns for the shaping of new products. One could say that the basic idea is to “think of” a need for a product and then to design the conditions and the prototype for that very product, to make that product to happen or come true. In this understanding, form precedes function. However, to use this understanding of design in relation to a social phenomenon such as learning – and to look at learning results (only) as a consequence of the design of for example school buildings, learning resources or classroom activities – is to make a categorical mistake.

Another way of thinking of design is to emphasise design much more as interaction design processes. Here, one not only focuses on products but also on, for example, social processes at different work places, and emphasis is laid on the making of products together with users, for “quality-in-use” (Ehn & Löwgren, 1997). Aspects like transparency, user control and playability, social-action space and personal connectedness are then put to the fore (Bagnara and Crompton Smith, 2006, Löwgren, 2007, Löwgren and Stolterman, 2004). In some schools, one can notice an increased planning with the pupils/students concerning some aspects of the work.

Here we will also discuss interactive design, and by that we mean those kinds of processes, which users themselves are engaged in, with each other and with the professional designer (which in our case could be the teacher, Selander & Kress, 2010). This also comes close to design by use. Here, re-design is the central aspect of how products, artefacts or processes can be used in a new way, related to new situations and new problems or demands. One could also say that in this case function precedes form, where people use whatever form to solve a functional problem (Brandes, Stich, & Wender, 2009). And this aspect seems to be crucial also in understanding how learning is going on in schools or at work places. Pupils/students use not only pre-produced learning resources such as textbooks. They may also check other sources for information, for example on Wikipedia, although this source of information is neither highly valued, nor authorised, by experts. Therefore, it is perhaps more important than ever before to think about how information is selected and presented. This aspect was never important in a school using only ‘authorised’ textbooks.

We would like to draw upon all three design-perspectives to understand education and learning. But we would also underline that the re-design, or inter-active design, is a central aspect for a theory of learning. Here acting, the engagement, and the doing in making choices during a process to transform given information and given representation into new representations are emphasised. One could also talk about contact zones (in German “Kontaktzonen”) in the virtual space where negotiations are performed in the age of new media (Clemens, 2010).

From a design-theoretic and multimodal point of view, not only professionals are designers. Rather, everyone engaged in communication – and learning – is a designer/re-designer. Design is the planning of something new to happen, either seen from the perspective of the designer-as-producer, or from the designer-as-user, point of view. Design is about shaping products, but also about shaping social interaction. Design is a way to configure both communicative resources and social interaction (Kress & van Leeuwen, 2001).

An increasing interest for design perspectives in learning could be noticed during the past decade (Kempe and West, 2010, Rostvall and Selander, 2008, Selander, 2008a, Selander, 2008b, Selander and Kress, 2010, Sheridan and Rowsell, 2010, Wasson and Ludvigsen, 2003). Sometimes the orientation is in terms of instructive-oriented approaches to classroom work (Gagnon & Collay, 2001). From our point of view, the emphasis is rather on a theoretic understanding of how one can analyse teaching- and learning activities as communicative, sign-making processes. Here, the transformation of information and the production of new representations of the world are central, as well as the understanding of cultures of recognition, assessment practice and, as a consequence, what is seen as signs-of-learning.

A design-theoretic approach to learning focuses on learning as communication and emphasises the role of materiality, which simultaneously creates and defines learning conditions — as in teacher education, school buildings, school textbooks and ‘serious’ games, assessments and tests, or for example in teacher's planning and allocation of time. It also includes the kind of resources that are at hand (and are allowed to use) when learners learn to master something, as well as the processes designed by the teacher (Wiggins & McTighe, 1998).

This further emphasises an understanding of designs in learning as a central aspect to understand learning as meaning-making activities and engagement. It is about what takes place when human beings learn and how possible learning paths, including all kinds of choices and decisions, are constructed (Insulander, 2010, Kempe and West, 2010, Kjällander, 2011, Sheridan and Rowsell, 2010). The time aspect, finally, is as important as the interactive aspect of communication and the role of materiality. Designs in learning focuses transformational processes, where decisions and sequences of possible actions are created over time.

Signs, and configurations of signs, are like rhizomes, not linear and with great varieties of possible meanings (Deleuze & Guattari, 2008/1988). But this does not mean that any interpretation is meaningful or relevant. The existence of a concrete garden path is a real reason for the direction of rhizomic growth. Signs are embedded in social relations, organising principles, reading paths and signifying practices. What a combination of signs “stands for” is in this sense not arbitrary. Signs are parts of structuring social practices (Kress, 2010).

To learn something in a school context is to be capable of using signs according to established conventions, usually expressed as ‘sets of standards’. Being able to show “how” one understands something is a key issue. Thus, what is here discussed as cultures of recognition is related to those signs of learning that are seen as, and accepted as, knowledge and learning in the specific context (Selander & Kress, 2010). The role of cultures of recognitions and assessment will be commented upon at the end of this article.

The idea of meaning-making has been entirely commonsensical. As human beings we know words, and most of us know how to put them into an acceptable order to form sentences, and most of us even know how to make longer texts out of these sentences. The process is so routinised that we hardly reflect on what we do. If we do it well we are regarded as reasonably competent. At times we become aware that we have made a wrong choice, whether of words, or of syntax, and before we know it we are in trouble, bigger or smaller. We become aware that we had choices, of words or syntax, and we did not make the right choice. The ubiquity of the new media has given us the means – not really available to us even 30 years ago – of producing meaning and disseminating these meanings to a very large group of people in various ways. That makes certain choices problematic.

Our sketch aims at unsettling this long established commonsense of meaning-making, in several directions: by relooking at the idea of the agency of the learner; in looking at the means for making meaning beyond those which have traditionally been acknowledged; and by looking at the effect of the digital media in that. We do wish to emphasise that in our view the social is prior to the technological, which we see both as the product of social work and, as a technology, a means of furthering social arrangements.

The new media have other effects: they provide greater facilities for producing texts, where by ‘text’ is here meant any semiotic entity which has a sense of (social) completeness. The text may be very small, such as the sign No Smoking on the wall of a room; or very large, such as a film; or somewhere in between, such as a website. Often the No Smoking sign is accompanied by an image: usually as a quasi traffic sign, in which a diagonal red line crosses through a smoking cigarette. Choice has become extended: namely a choice whether to use just one ‘mode’, writing, or two modes, ‘writing’ and ‘image’, where the one seems to reinforce the other. Within each of the two modes there is still choice: for instance: “we thank you for not smoking” and with the image there is a choice too: an image could be shown of someone stubbing out a cigarette. To make a sign that says “We thank you for not smoking” and accompany that by the crossed through cigarette begins to be more than mere reinforcement: the overly nice, genteel written comment backed in fact by the force of the law. To write “we thank you for not smoking” and show an image of a cigarette being stubbed out, is either confused or sarcastic or downright nasty. Each choice is the expression – the making material – of different meanings.

The availability of many modes – as in the seeming simple No Smoking + image example – also moves us beyond “competent use”. Something quite new is now taking place: the conjunction of the genteel written comment with the brutal image. That is made: it was not there before for us to use. Design has moved into the foreground, and competence is there, of course, but it is backgrounded.

With the use of digital media, the choices of modes available to use in meaning-making multiply: most – we would say all – representations are multimodal. That ranges from seemingly trivial matters such as choice of font to point size of letters; to less trivial seeming matters such as choice of layout; of colour and image – moving and or still; of speech and writing; and so on. In a social semiotic approach to multimodality, choice is the basis of and the expression of meaning: not to have chosen those other possibilities – de-saturated pale colours rather than fully saturated deep colours – becomes laden with significance; and that applies to the choice of font as well.

Choice from the availability of modes opens a quite other facility. Modes have differing ‘affordances’, different potentials for making meaning. The logic of time – of modes that entail movement: speech, dance, action, and gesture – offers different potentials epistemologically and ontologically to modes bound by the logic of space: (still) image, writing (in some ways), colour, layout, 3D objects, etc. This offers a vast potential and also represents a trap for the unwary: the kinds of meaning possible through the logic of time differ profoundly from those offered by the logic of space (Kress, 2010). ‘Multimodal ensembles’ offer the possibilities of both; and that demands clarity in rhetoric and design.

Choice derives from and rests on the ‘interest’ of the maker of the sign, who is at the same time the maker of the meaning. ‘Interest’, the availability of means for making meaning – modes for instance – and the contingencies of the social situation in which the sign-maker finds herself or himself, lead to the making of the sign that is made: and in that way it allows us to connect the form of the sign with the social givens of the making of the sign.

The move from ‘use’ and ‘competence’ to ‘making’ and ‘design’ gives rise to a quite different sense of agency. “Use” often refers to an instrumental approach, and competence to a more or less pre-defined and fixed set of “achieved” qualities. “Meaning-making”, “making” and “design”, on the other hand, refer to the very process of engagement, transformations and sign making to explore the world and take part and communicate with others in a certain context. And as has been said, the potentials of the new media have a large role in permitting what are the social factors – of changes in distribution of power, for instance – to be expressed more intensely.

Agency is present in all interactions, of whatever kind, even when it is seemingly not visible. Social theories and social ideologies will either foreground agency as such, or, selectively, the agency of some. Traditional theories of communication have emphasised the agency of the maker of the message – the ‘sender’ – at the expense of the ‘receiver’. The latter was consigned to the action of competent decoding. Contemporary social ideologies and social givens are beginning to allow us to see the agency of both the maker of the initial sign, which then acts as a prompt for someone who engages with it, to interpret (it or) part of it in the light of her or his interest and of her or his semiotic resources. In so doing he or she makes a new sign from the initial sign: and an apt theory of communication – with a different sense of agency – is a theory which holds that communication has happened when there has been interpretation (Kress, 2010, Kress and van Leeuwen, 2001). Or as the semiotically oriented philosopher Cassirer (1996) phrased it: “mankind is capable of form” (Cassirer, 1996:46). Mankind produces (motivated) signs, symbols and narratives — as producers and interpreters, and as learners (Jewitt, 2006, Stewart, 2001, Wagoner, 2010, Zittoun, 2006). These signs, symbols and narratives are representations of ways of understanding the world (Chaib et al., 2011, Kress and van Leeuwen, 2001).

This approach distributes power and agency differently; both the initial maker of the sign, and the re-maker of a new sign from that – taking the initial sign as a prompt for interpretation – are sign-makers. Each makes selections and shapes of the sign according to principles based on their interest. This makes meaning-making into an ongoing process; and at each stage the social enters in the guise of the interests of the sign-makers. In that way, signs and complex signs such as texts are traces of the social and can be read – hypothetically – in that way. In that way, social action as interaction is the generative basis of meaning: an ongoing, ceaseless chain of rhetorically motivated selection, (re-) designed transformation/interpretation.

At each point, individuals act and interact. At times the interaction is with others, in groups which may be communities of some standing or with other individuals where there is less ‘connection’. At times the interaction is by an individual with aspects of the socially shaped world, such as when I am in a city new to me and I wander along, gazing at the street-scape and the different kind of life in the streets. At times the interaction is ‘internal’ — still social though, with socially made resources internally; at times the internal interaction becomes visible, or audible even, to bystanders, who look and wonder.

Here too the new media have brought a profound change through the affordances of speaking, writing, arguing back in outwardly visible and disseminable semiotic entities — texts. The possibility to argue back — in video, in writing, in speech, in multimodal ensembles has changed the potentials for the effects of what was always inner agency into outward manifestations.

Where competence offered one form of participation, limited by the rules of competent use within tight social conventions, design (based on a prior rhetorical assessment) offers the possibility to an agent to project her or his wishes, desires, proposals, forward into an imagined social future. That is a profound difference, for identity as for learning.

What might be called traditional theories of teaching and learning – the ordering of the two terms is important – assumed an authoritative holder of knowledge – the teacher – and a learner who had to ‘acquire’ that knowledge. Metrics of success measured the degree to which acquisition had been achieved. These metrics were set up in terms, which acknowledged the power difference between teacher and learner; moreover they were set up in terms of the canonical forms in which knowledge had been ‘coded’ — both in terms of modes and in terms of genres.

Learning is now seen as an instance of communication. This leads to the re-valuing of the agency of both the ‘sender’, in traditional terms, and the ‘receiver’, now seen as an interpreter of the message taken as a prompt. That requires entirely new metrics for the understanding of the semiotic work – the learning – of the learner. As an agent who interprets according to principles of his or her own, acquisition is no longer a relevant or plausible metric. Instead, the principles which underlay and guided the learner's engagement with the prompt supplied by the teacher, now come to the fore: these supply the new metric for understanding the learner's semiotic work.

With the change in the agency of the learner comes therefore a need to value the work performed by the worker. The teacher/assessor's task is to look at the interpretation of the prompt, to understand the principles of selection, and the other principles of transformation used in the interpretation, as indication of the resources available to and used by the learner. The change in agency marks a different social relation of teacher and learner: and it will lead to a reassessment of the teacher's role. Assessment is no longer only about evaluating in relation to certain standards, but far more a question of “feed up”, “feed back” and “feed forward” to facilitate the learning in a broader sense (Björklund Boistrup, 2010, Black and Wiliam, 2009).

This social change will have its effects on what will be new cultures of recognition: a recognition that in communication, and in interaction as communication generally, there are two makers of signs, two meaning-makers, both of whom have a claim to have their semiotic work fully recognised.

The other aspect of recognition lies in the domain of representation. At all times there will be ‘recognised’, ‘canonical’ forms of representation. In the traditional school, these were dominantly writing (and speech), numerical means, and certain forms of images, as in geometry for instance. In a world in which it is recognised that meaning exists in many forms, and not only in those which have hitherto been regarded as ‘canonical’, a new requirement is to develop means of recognition of the different modes – and genres – through and in which learner/interpreters express the meanings which they have made in the transformative – and transductive (see Kress, 2010) – work of interpretation.

This is of course a question of power, in two distinct ways: is the agentive work of learners as interpreters acknowledged and recognised? And is recognition given to non-canonical forms of representation, whether in modes or in genres, through which learners give expression to and materialise their meanings as interpretations. One question therefore is: who takes the power onto themselves to accord or to withhold recognition? Cultures of recognition are effects of the use of social power. The question “What gets recognised and by whom?” will become means of determining distributions and locations of power.

We said that this leaves a question about the role of teachers. In a society, which has a culture that accords appropriate recognition to semiotic work, the role of the teacher changes. The responsibility of the teacher continues to be to make available to the young forms of knowledge in domains which the society regards as crucial. The interpretations of the learners become means of establishing what principles they have brought to bear in their interpretations; and these principles in turn become means for allowing teachers to engage in discussions with learners about their interpretations and then to shape the next prompt in such a way that it is better suited to lead the learner/interpreter in the direction the teacher would want.

This will lead to a culture of recognition of agency and semiotic work, in all modes and genres; not as the final stage of learning, but as a means of establishing what principles underlay the interpretative work of students. This culture of recognition is at the same time a culture of valuation — a valuing of the agency of all learners. A culture of Wertschaetzung.

References (29)

  • P. Ehn et al.

    Design for quality-in-use: Human–computer interaction meets information systems development

  • S. Bagnara et al.

    Theories and practice in interaction design

    (2006)
  • L. Björklund Boistrup

    Assessment discources in mathematics classrooms. A multimodal social semiotic study

    (2010)
  • P. Black et al.

    Developing the theory of formative assessment

    Educational Assessment, Evaluation and Accountability

    (2009)
  • U. Brandes et al.

    Design by use. The everyday metamorphosis of things

    (2009)
  • E. Cassirer

    The philosophy of symbolic forms. Vol. 4. The metaphysics of symbolic forms

    (1996)
  • M. Chaib et al.

    Education, professionalization and social representations. On the transformation of social knowledge

    (2011)
  • I. Clemens

    The global icon of the Indian IT-professional. Contact zones in the age of new media, globalization and world society

  • G. Deleuze et al.

    A thousand plateaus. Capitalism and schizophrenia

    (2008/1988)
  • G.W. Gagnon et al.

    Designing for learning. Six elements in constructivist classrooms

    (2001)
  • E. Insulander

    Tinget, rummet, besökaren. Om meningsskapande på museumDiss

    (2010)
  • C. Jewitt

    Technology, literacy and learning. A multimodal approach

    (2006)
  • A.-L. Kempe et al.

    Design för lärande i musik

    (2010)
  • S. Kjällander

    Designs for learning in an extended digital environment. Case studies of social interaction in the social science classroom

    (2011)
  • Cited by (0)

    View full text