Interactions

Authors:
Jonas Löwgren

---

 

The notion of making seems to be gaining some traction lately. When I talk to people outside academia about interaction design and new media, making often comes up, with references to physical computing, 3D printing, and Maker Faires, and overtones of grassroots activism and yet another this-changes-everything movement. I note a corresponding increase in the interaction design research discourse—but here, the word is used slightly more broadly to include construction, programming, and other craft-like activities forming part of the core of traditional design practice. That is also the sense in which I will be using the word here.

In his influential effort to characterize design as a distinct tradition in relation to the CHI community, Bill Gaver pointed out among many other useful observations that within design, "the practice of making is a route to discovery" [1]. This instinctively rings true to me, and the paper from which the quote is taken may in fact be one of the reasons behind the growing recognition that making is important in interaction design research.

  Insights

However, it is less clear exactly how and why making is seen as important in interaction design research. One strategy, formerly quite dominant, is to blackbox making in favor of its outcome, the prototype, which could be tested empirically to yield the science of a traditional "design-oriented" HCI paper. My sense is there is an increasing interest in unboxing making, but the elaboration of its significance has been tentative thus far. It is my intention here to contribute to this elaboration by suggesting four ways in which making can be significant in interaction design research:

• Making is required for explorative design of non-idiomatic interaction.
• Related to the previous point, making contributes to responsive sketching media.
• In extramural collaboration, making is a convenient co-production nexus.
• Making yields artifactual knowledge.

The first and second points concern the fine-grain details of how making as a design research activity can contribute to knowledge production, whereas the third one indicates some useful possibilities of making in the larger context of organizing research projects. I elaborate on each, using examples drawn from my own work.

The fourth point is slightly different in nature, as it states something quite obvious. Yet, as a community, we seem to find it remarkably hard to come to terms with artifacts as knowledge in the context of design research. More on this later.

  Designing Non-Idiomatic Interaction

For professional designers and design researchers alike, it is a fact that most common sketching techniques for detailed interaction design are more or less off the mark. The key in designing the finer points of interaction is arguably the dynamic experience of user-and-product interplay unfolding over time. Yet we tend to sketch either formal or temporal properties but rarely both at the same time. Drawings, comps, and wireframes capture the static forms of display-based interfaces, but the temporal properties are left implied in the spaces between the screenshots. Bodystorming and generic-prop role play, on the other hand, focus almost exclusively on temporal sequences.

If this is the case, then how is it we can design at all—and how is it our products often end up being pleasant as well as useful? I would argue this is thanks to our sketching literacy and our grasp of interaction design's idiom. When we draw a series of wireframes, we do it against the collective experiential backdrop of user-testing thousands of websites and interacting with millions more. We can feel the interactional qualities of transitioning from one wireframe to the next, the sense of progression and rhythm, even though the sketching techniques themselves actually do not support that kind of reflection. The same holds true for sketching out standard interface widgets, manipulation gestures, and other established interaction elements—in short, when sketching idiomatic interaction.

---

Interaction design research would be well served by preparing Ph.D. candidates to consider disposable programming as a major technique for hi-fi sketching.

---

However, when we aim to explore interaction possibilities outside the established idiom, it turns out the common sketching techniques can feel inadequate. And this is where making with a sketching mindset becomes significant. As an example, I offer the information visualization concept called Pinpoint that I helped design in 2008 (Figure 1).

Briefly, Pinpoint explores resource management in a large multinational organization where every employee could benefit from talking to and working with a large number of colleagues who aren't even on their radar. Data is mined from internal sources and communication logs to generate a radial view of the 60 topically and communicatively closest colleagues, which can then be filtered and browsed so that curiosity can shift to making new contacts within the organization.

This overall concept was established at a certain point in the design process, and we nailed the graphic design easily with static comps. But we were still in the dark as to whether the overall interaction experience would be compelling, or simply confusing. There was no idiomatic knowledge to rely on in figuring out the details of interface behavior upon user clicks and filter selections. Pencil drawings and storyboards did not help us decide how the act of hovering over a potentially interesting unknown colleague and then focusing the radial view on her should unfold, for example. The detailed behaviors of the real-time layout algorithm and the between-states animations could be designed only by trying to make them.

We used Processing (Figure 2) to quickly implement a selection of layout algorithms using the same data and determined that a force-directed layout offered engaging animation behavior and adequate performance. We then wrote a parameterized version of the layout algorithm and tweaked the parameters to find the right balance of swiftness and springiness in the silhouettes' movements as they established their places in a reorganized layout.

For the fine-grain details of interactive responsivity, we switched to Flash's programming language ActionScript and went through similar rapid-fire rounds of experimental coding to find flows and behaviors that seemed coherent and engaging. It is important to note here that the code we wrote in Processing and ActionScript was not intended to be used in the final demonstrator—it didn't consider error handling and edge cases; it was poorly structured and not documented; it was very quickly produced and definitely disposable. In other words, we were sketching in code.

This is a drawn-out way of saying things that may seem self-evident and that have been said before (see, for instance, [2]). However, when I look at the curricula of some interaction design postgraduate programs, I can't help feeling that interaction design research would be well served by preparing Ph.D. candidates to consider disposable programming as a major technique for hi-fi sketching.

  Responsive Sketching Media

At this point, I would like to make a quick digression into the history of design and the foundational concepts of our field. In recent years, there has been a notable surge of interest in the materials of interaction design. Erik Stolterman writes in an early and enthusiastic blog post that "[t]he changes and challenges to come with the material turn are extraordinary and intriguing ... [T]he fun part is that the interaction design space suddenly and radically grows" [3]. Another salient example is the CHI 2012 panel on material interactions, suggesting that we are "moving into an era of post-representational design" [4]. It might even be safe to say the notion of digital materials has proven to be somewhat evocative for our community, but still there is something uncomfortable about it. We don't seem to be able to say very much about the nature of digital materials, their qualities and affordances. The difference is particularly clear when comparing our efforts with the kinds of material discourses going on in traditional craft-based design fields such as cabinet making or textiles.

I think the reason for this conceptual discomfort might be found in the heritage of the terms involved. Before design, there was craft, and in craft the categories were (relatively) clear, the general gist going something like this: The craftsperson used a tool to shape a material into a final artifact. Enter the industrial age, the separation of form-giving from production, and the hundred years of evolving sketching techniques and design representations to uphold the connections to material qualities across the design-production gap. So far, so good, in a very simplified retrospective view. But interaction design poses new conceptual problems that may complicate our use of the key concepts of traditional craft and design, however evocative we find them to be. For example, telling tools from materials in interaction design can sometimes be a highly contrived exercise (as presciently pointed out by David Wroblewski more than 20 years ago! [5]). Is a programming language a tool or a material?

And this takes me back to the significance of making in interaction design research. There is a small but growing body of research in our field that likes to frame itself as the digital cousin of traditional art and design's material exploration [6,7]. However, the difference so far seems to be that not very much conceptually crisp knowledge has been produced on the qualities of digital materials. This could be a question of time and maturity, of course, but I would like to submit that a different conceptualization might be fruitful.

The kind of research I have in mind is the one in which researchers engage with digital technologies aiming to explore which interactive experiences they may give rise to. Such engagements typically consist of tinkering and experimental making, guided in equal parts by curiosity toward the technologies' properties and a sense of what constitutes a compelling use experience. I would like to propose that what those researchers do through their making is to create responsive sketching media, that is, materialized what-if worlds that talk back to the researchers' probing design moves and indicate promising ways forward.

My example here is the quaint and speculative project Capable Books (Figure 3), where I spent some time last year combining physical computing and classical bookbinding to try and make a book that would know which page it was opened to. The project was dutifully framed as an interaction design hybrid-material exploration, but I (predictably) found myself unable to say anything coherent and enlightening about the qualities of the hybrid virtual-physical digital materials. On the other hand, I did get a useful reminder of the power of a responsive sketching medium to drive ideation.

At one point, my idea was to gild the top edge of a book with conductive paint and measure the resistance between front board and headband to be able to gauge how many pages were in the left half of the opened book (Figure 4).

After making such a book and handling it for some time, I found the idea to be a dead end—there was no way I could get reliable and replicable readouts.

Then, by coincidence, I happened to squeeze the book while watching the resistance readout, and I noticed a spike. Minutes later, I had learned to control the resistance level by applying left-thumb pressure to the open book. The sketch had shown me a different design concept: a pressure-sensitive book. I can say perfectly honestly the idea hadn't occurred to me before that day, but after seeing it I felt suitably embarrassed for overlooking such an obvious idea where the implications open a whole new field of potential augmented-book experiences.

---

I have found repeatedly that the easiest way to go from high-level letters of intent to concrete action is to define a joint task of making something together.

---

To me, this example illustrates a possibly powerful way of thinking about making in interaction design research, namely to orient your making toward creating responsive experimentation environments (or, as Wroblewski would have had it in 1991, facile tools and responsive materials) and then listening carefully to the backtalk.

  A Nexus for Co-Production Activities

Here we leave the myopic levels of tinkering, tweaking, and twiddling, and move to the higher grounds of interaction design research in collaboration with extramural actors: companies, organizations, associations, NGOs, heterogeneous groups of people, and what have you. For some researchers, external collaboration ranges from political concession to outright annoyance, and I agree the overheads are sometimes considerable. However, I would maintain that design (and design research) ultimately concerns itself with possible worlds, and in my experience, much of the experimentation with those possible worlds requires engagement and intervention in the world to be meaningful.

Hence, most of my design research since the early 1990s has been done in collaboration with extramural actors. The most fruitful approach I have found for this work is one of co-production, where you enter into a collaborative undertaking together with one or more extramural actors, guided by a common interest but very explicitly aware that you bring different agendas to the table. Typically, the researchers' agenda features the production of knowledge that is of interest to their research community, while a company might aim for new product ideas or new ways of working, an NGO could be focused on furthering their cause, and so on.

The upside of this approach is that after an initial time period of building rapport and trust, the transparency around the different agendas tends to make for a high level of mutual respect and relatively straightforward negotiations of IP rights and other conditions for collaboration.

The downside of the different agendas is ... different agendas. Specifically, when it comes down to concrete collaborative work, nothing comes for free in a situation where the different stakeholders want different takeaways. And this is where making comes in. I have found repeatedly that the easiest way to go from high-level letters of intent to concrete action is to define a joint task of making something together.

The reason seems to be that the concrete something to be made is both undefined and highly specific. It is undefined in the sense that it has not yet been made, which means that all the actors around the table can project their respective expectations onto it. At the same time, it is highly specific in terms of providing a deadline and a number of milestones and activities. And once concrete collaborative work commences, the actors tend to get drawn into the fascination of making, and most what-if issues that were difficult to resolve at the planning meetings fade into the background.

Back in 2001, I initiated the Avatopia project with Swedish public service TV broadcaster SVT and a couple of other, slightly more peripheral partners. The main incentive for SVT was strategic, revolving around the question of what public service could mean in the changing media landscape. A secondary aim for them was to improve their relevance among younger audiences. Our agenda as researchers was to gain knowledge about the design of communicative experiences combining broadcast and collaborative media, and the potential transformation strategies for established mass-media actors.

The brief for joint making focused on enabling teenage activists in sparsely populated geographical areas to join a community and engage in nonviolent societal action. Next, an overall project approach was developed to accommodate the different agendas: For SVT, a concrete result in the form of a deployed initiative would be an example of reinterpreting public service in a more contemporary way; the desire to reconnect with younger audiences was already in the brief. For the researchers, their interest in design-oriented knowledge led to an orientation toward combinations of broadcast media and online interactive communication. Both of these proposed concretizations were acceptable to the other parties of the project, and thus they were made into starting assumptions.

Due to the demanding nature of the project goals, we planned a participatory design process involving a number of young teenagers in the development of a crossmedia communication platform (Figure 5). Importantly, the young members of the design team would form the core community of the platform once deployed, embodying the norms and values of the community, and serving as mentors and trailblazers for newcomers.

The design process involved some 20 teenagers together with researchers and staff members from the project partners over the course of nine months. The collaboration led to a crossmedia spiral concept, in which a small and focused online community would be the hub for nonviolent societal activism; the activities of the online community would be professionally broadcast weekly on public-service TV, reaching a large audience and attracting potential members to join the online community. This concept was developed into a 3D avatar world and corresponding broadcast TV formats, and launched in September 2003. The community of teen activists grew according to plan until an unforeseen budget cut forced the premature termination of the project.

The point here is not the project outcome in itself, but how an early and continuous focus on joint making enabled us to execute a collaborative project where the stakeholders fulfilled their different agendas. SVT understood some of the possibilities and challenges of taking a more proactive stance on public service for young audiences, and the researchers formulated and published knowledge contributions on collaborative media design. Most important, the general feeling of the collaborative work was one of shared excitement, curiosity, and responsibility—and my sense is that this was due to the emphasis on joint making.

  Artifacts Carry Knowledge — Yeah, so?

My final point is slightly different from the previous ones. Instead of proposing and exemplifying how making can be significant in interaction design research, all I can do here is more or less point to a current dilemma and the ways in which we are grappling with it. It goes like this: Making leads to artifacts, things that were made. In the design tradition, the artifacts are typically the final outcomes. There are many examples in design communities of how the artifacts are seen to carry knowledge. For instance:

• A designer's résumé is convention-ally a portfolio, a mediation of artifacts with minimal annotation or elaboration.
• Many hours are spent in classical design education appropriating selections of canonical examples.
• Designer contributions to multiparty projects customarily consist of envisionments and prototypes.

Andrew Richardson describes evocatively how a design tradition considers artifacts, including digital ones, capable of standalone communication: "Digitally programmed objects that involve human interaction ... can engage the viewer in a kind of intuitive, sensory experience that evokes the same type of emotion and delight gained from handling a well-made, physically crafted, object ... [The] experience of encountering a well-crafted object that expresses the character of its material and the mysterious 'how did they do that' skill of its creator is echoed when viewing beautifully created digital artefacts (a J. Maeda or G. Levin piece, for example)" [6].

On the other hand, our existing interaction design research community has largely been built on scholarly traditions where propositional knowledge is conveyed in verbal language and written down in papers and theses.

When I propose that making is significant in interaction design research, the dilemma is right there: Making yields experiential knowledge embodied in artifacts, but the research tradition expects propositional knowledge embodied in academic writing. As noted by Carl DiSalvo [8], the question is whether the artifact is the ultimate focal object in design research, or merely a prop in a design experiment.

As is usually the case with such rhetorical questions, the answer seems to converge on a position of both and neither. The art and design communities have had a significant head start on the problem since the demands to academize started to become pressing; in the seminal note where Christopher Frayling articulated the now influential "into/for/through" distinction, he spoke about research for art and design—where the end product is an artifact and the knowledge contribution is embodied in the artifact—as "the thorny one" [9]. An early attempt to negotiate the thorns was the survey by Alex Seago and Anthony Dunne of three Ph.D. projects in the art and design context, illustrating how making could form a key research activity and how the artifacts could enter the scholarly discursive practices:

[R]ather than arguing that a radically new electronic product or a new method for producing metals can be constituted as providing new knowledge 'in themselves,'[the sample researchers] situate their discoveries in a research context. All three of these doctoral programs have been conducted as systematic research activities and contain explicit data. The record of the conduct of both [sic!] programs is 'transparent' in the sense that a future researcher could uncover the same information... [T]he data employed and the results obtained are validated and related to a review of previous research in appropriate fields. In short, if research in 'any' discipline can be described as a systematic inquiry whose goal is knowledge, then [the sample researchers] offer good examples of systematic and original research by projects in art and design in which the object occupies a central place [10].

The traditional role of making in interaction design research has arguably been to produce artifacts that can be used as instruments for data collection through experimental or quasi-experimental empirical evaluations. The making itself and its design choices have generally not been detailed, and the knowledge contributions have been mostly located to the outcomes of the experiments. In recent years, however, our research community is starting to show some awareness of the dilemma and some ambition toward leveraging the knowledge outcomes of making.

---

We seem to be starting to recognize that the artifacts of interaction design research are more than props in design experiments.

---

I already mentioned Gaver's ambitious attempt to characterize design research as rooted in the tradition of design practice [1]. In his own words, "an endless string of design examples is precisely at the core of how design research should operate," and his main point is to argue against the scientistic emphasis on generalized theory (at the expense of the specific artifacts). The argumentation in his case leads to the annotated portfolio approach for knowledge communication in design research; others have followed his lead to identify a scale of knowledge abstraction levels between universal theory and particular artifacts, pointing out a range of intermediate-level knowledge forms residing between the endpoints of the scale (refer to [11] for a survey). What all those knowledge forms have in common is that they are amenable to established criteria for academic quality and can be communicated within the text-oriented boundaries of our scholarly tradition.

There is also a growing interest in artifacts and artifactual knowledge among those who organize conferences, edit journals, and contribute to the structuring of our academic discourse in other ways. The exhibitions and demos at CHI conferences appear to be growing in significance and quality over the years; the Pictorials format of the DIS 2014 conference was another interesting experiment in this direction.

The most ambitious and systematic attempt in this direction is probably the Research Through Design (RTD) conference that was inaugurated in the U.K. in 2013 and then repeated in 2015 with plans for a continued biannual presence. The stated aim of the conference organizers was to create "an experimental, inclusive platform for disseminating practice-based research" [12].

This aim is most clearly demonstrated in the idea of Rooms of Interest, a format intended to replace conventional auditorium-style presentations. Submissions to the conference were carried out in a two-step selective process with an abstract, a committee review, a full "visual paper," and a peer review. The acceptance rate was around 15 percent, and all accepted contributions consisted of an exhibitable piece as well as an archival paper. Presentations were organized into thematic Rooms of Interest in a multi-track schedule: Each session would hold about 30 people around a large round table, and three or four authors would present their work in turn while their exhibits were shown and passed around the audience. In successful cases, the presentations were rather short and a significant portion of the session time was devoted to thematic discussions across the presentations.

The RTD Rooms of Interest format and its execution were far from perfect, of course, but to me it still demonstrates very interesting steps forward for design research. First, it explores how the conference can offer structured opportunities for scholarly discourse in which knowledge is produced, rather than serving as a forum for reporting existing archival contributions. Second, it recognizes that the artifacts of design research carry knowledge and demonstrate concrete ways in which the artifacts can take part in the joint deliberation.

The RTD organizers are explicit about taking hints from the design tradition on how to structure discourse around artifacts—including time-honored formats such as exhibitions, crits, and curation— and their work seems to reflect a sound emphasis on situated knowledge production where the artifacts of design research also gain a seat at the table, so to speak.

To return to the original dilemma, we seem to be starting to recognize that the artifacts of interaction design research are more than props in design experiments, that they do in fact carry knowledge, and that we can find new ways of structuring our academic discourse to leverage that knowledge, but that the artifacts are by no means the only focal objects of design research. In a semi-recent survey of three seminal research-through-design projects within interaction design, Zimmerman et al. concluded:

All of the projects employed a Research through Design approach, creating artifacts that included products, prototypes, and models that illustrated future visions, uses of new materials, and potential ideas. All of the projects generated guidelines and sensitizing frameworks to provide the design research community with information about how to design. Finally, aspects of all projects were documented, ranging from methods and design processes to work that generated scholarly publication with the goal of multidisciplinary outreach [13].

It is striking how similar this recent characterization is to the prescient words of Seago and Dunne, perhaps illustrating a converging direction (taking some hints from art and design) where making is recognized as a knowledge-producing activity, and where we better understand the roles of artifacts as knowledge elements.

  In Closing

Stepping back, it may be noted that the growing interest in making and craft within interaction design research coexists with recent developments toward design thinking and other attempts to abstract design approaches from their materials and disciplines. These two trajectories are perpendicular in the sense that one is vertical, going deeper into the techniques and materials of a specific design discipline, whereas the other is horizontal and spans traditional disciplinary boundaries.

In my opinion, there is fruitful complementarity between the two rather than competition. There is no denying that interaction design and its abstractions contribute to disruptive innovation in product development, to sociopolitical interventions, and to the ongoing infrastructuring of services and mediated communication. At the same time, it is clear to me that we are still far from understanding digital materials and exhausting their design possibilities. Our work needs to continue in both directions; what I have tried to do here is provide a few stepping stones for moving further into the practices and the open questions of making and craft in interaction design research.

P.S. Incidentally, I did eventually come up with what I think might be a scalable and aesthetically tolerable solution for a handcrafted book that knows which spread it is on. It involves small photoresistors inside the boards, and punched page holes covered with semi-translucent paper. Solutions for manufacturing in larger volumes should rather be sought in the direction of printed electronics, I believe.

  References

1. Gaver, W. What should we expect from research through design? Proc. of the 2012 ACM Annual Conference on Human Factors in Computing Systems. ACM, New York, 937–946.

2. Victor, B. Inventing on principle. 2012; http://vimeo.com/36579366

3. Stolterman, E. HCI and the material turn. 2006; http://transground.blogspot.se/2006/04/hci-and-material-turn.html

4. Wiberg, M., Ishii, H., Dourish, P., Rosner, D., Vallgårda, A., Sundström, P., Kerridge, T., and Rolston, M. 'Material interactions': From atoms & bits to entangled practices. CHI'12 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, 2012, 1147–1150.

5. Wroblewski, D. The construction of human-computer interfaces considered as a craft. In Taking Software Design Seriously: Practical Techniques for Human-Computer Interaction Design. J. Karat, ed. Academic Press, Boston, 1991, 1–19.

6. Richardson, A. New media, new craft? SIGGRAPH Electronic Art and Animation Catalog. ACM Press, New York, 2006, 164–166.

7. Vallgårda, A. and Sokoler, T. A material strategy: Exploring material properties of computers. International Journal of Design 4, 3 (2010), 1–14.

8. DiSalvo, C. Disseminating research through design: Challenges and opportunities learned. Constructivist Foundations 11, 1 (2015), 22–23.

9. Frayling, C. Research in art and design. Royal College of Art Research Papers 1, 1 (1993), 1–5.

10. Seago, A. and Dunne, A. New methodologies in art and design research: The object as discourse. Design Issues 15, 2 (1999), 11–17.

11. Löwgren, J. Annotated portfolios and other forms of intermediate-level knowledge. Interactions 20,1 (2013), 30–34.

12. Durrant, A., Vines, J., Wallace, J., and Yee, J. Developing a dialogical platform for disseminating research through design. Constructivist Foundations 11, 1 (2015), 8–41.

13. Zimmerman, J., Stolterman, E., and Forlizzi, J. An analysis and critique of research through design: Towards a formalization of a research approach. Proc. of the 8th ACM Conference on Designing Interactive Systems. ACM Press, New York, 2010, 310–319.

  Author

Jonas Löwgren (jonas.lowgren.info) is professor of interaction and information design at Linköping University, Sweden. He specializes in interactive visualization, collaborative media, and the design theory of digital materials. [email protected]

  Figures

Figure 1. The final Pinpoint concept screen.

Figure 2. One of the Processing sketches.

Figure 3. Capable Books work in progress.

Figure 4. Schematic of the conductive top edge idea.

Figure 5. Snapshots from the Avatopia project.

 

Copyright held by author. Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2016 ACM, Inc.