Imaginaries Lab 2018 Review

It’s the end of December, which means it’s time for an update. Here at the Imaginaries Lab we’re just completing our second year, currently based within Carnegie Mellon School of Design. We’re a pretty part-time lab at present, but have aims to do much more in the years ahead. We’re using creative approaches to envision alternative ways of thinking and living, now and in the future, to inform interdisciplinary research and practical applications for social and environmental benefit. Our goal is to become a world-leading center for this kind of research, collaborating internationally to support transformative innovation. We carry out research projects, publish and run workshops internationally, teach studio classes, and build collaborations externally and within Carnegie Mellon.

What does the Imaginaries Lab do?

The lab’s basic premise is that how we imagine affects how we understand the world, how we live, and what we see as possible in our collective futures, with consequences for sustainability, society, our relationships with technology, and our everyday lives.

At the Imaginaries Lab we believe that humanity needs tools to enable new ways of understanding and imagining, and new ways to live, that provide more equitable socially and environmentally sustainable futures. We create those tools through developing creative research methods, adapted from those used in design practice, and explore their use in a variety of cross-disciplinary contexts.

In 2018, the Imaginaries Lab team, including — over the course of the year — Devika Singh, Matt Prindible, Gray Crawford, Saloni Sabnis, Silvia Mata-Marin, Rachel Gray Alexander, Shengzhi Wu, Katie Herzog, Michelle Chou, Ashlesha Dhotey, Aadya Krishnaprasad, and Dan Lockton (as well as Bella), have worked on a range of projects in three main areas:

█  Imaginaries, mental models, and mental imagery: using design methods to investigate how people understand abstract or complex concepts (from mental health to energy to metaphor generation to the structure of disciplines themselves), help them understand and imagine in new ways, and imagine new ways of living. This research covers the development of creativity methods, workshop and facilitation methods, and new kinds of interface design (human-computer interaction) and qualitative data visualization.

█  Research through design, and design as inquiry: investigating the use of design practice as a form of research and creative inquiry, including how to teach design studies through critical making, speculative and critical design, and how design methods can contribute to new knowledge generation beyond the design discipline itself.

█  Design for behavior change: exploring the links between designed technology and influence on human behavior, particularly in relation to sustainability and social benefit, and how designers can practically engage with issues of ethics and effects in this area. The Design with Intent toolkit is one of the most highly-cited pieces of work in this field, both academically and through use in design practice, but how is the field evolving in the light of mass surveillance, individual behavioral profiling, and weaponized behavioral economics?

In practice these areas have been woven through projects with a range of themes — new methods for design and creativity, new types of interface, intelligences, and futures.

New methods for design and creativity
Among our projects exploring what we might loosely call ‘new methods for design and creativity’, New Metaphors has seen the most development during 2018, with workshops at Interaction 18 in Lyon, UX Lisbon, Plurality University’s Founders’ Meeting in Paris, a keynote at Interaction Latin America in Rio de Janeiro (Dan Lockton), and numerous sessions at Carnegie Mellon including a workshop for the Swartz Center for Entrepreneurship and a dSHARP Digital Humanities talk. A metaphor is just a way of expressing one idea in terms of another, often used in design to introduce people to new ways of doing things, by relating them to familiar ideas, from desktops, files and windows, to the net, web, websites and browsers, cloud storage, even blockchain. Many of these are so familiar now that we perhaps no longer even think of them of as metaphors. But they are not inherently ‘right’; they can be challenged — including creating novel metaphors, which can persuade us to think differently and accept new ideas, or help us reframe the ways we think at present. The New Metaphors workshop format is a simple juxtaposition approach using cards and a variety of structured worksheets — or Devika Singh‘s Inspiro SMS bot — but can generate ideas applicable to a wide range of domains within and beyond design and futures.

⇧ A New Metaphor generator

⇧ Interaction Latin America 2018 keynote, ‘New Metaphors’

In November, Michelle Chou, Saloni Sabnis, Devika Singh and Dan Lockton ran a DIF On Air session for the Ellen MacArthur Foundation’s Disruptive Innovation Festival on ‘New Metaphors for Design, Economies, and the Systems of Everyday Life’ (video below). The team, facilitated by Laura Franco Henao, discussed how — inspired by the metaphor inherent in the circular economy — other kinds of metaphors could help give us an expanded conceptual vocabulary around economies and our relationships with products, reframing them for more sustainable ways of thinking.

⇧ Michelle Chou, Saloni Sabnis, Devika Singh and Dan Lockton, hosted by Laura Franco Henao, for the Ellen MacArthur Foundation DIF 2018

The New Metaphors project offers a form of intentional apophenia — deliberately provoking oneself to see patterns or relationships or parallels where (perhaps) none actually exists but where proceeding as if it does offers us some new way of thinking or interesting angle for seeing the world differently. This has some overlap with the ‘event scores’ of the Fluxus movement, for example the pieces collected in Yoko Ono’s Grapefruit — which led to Dan Lockton taking part in the Disruptive Improvisation workshop at CHI 2018 in Montreal, organized by Kristina Andersen, Laura Devendorf, James Pierce, Daniela Rosner, and Ron Wakkary. The New Metaphors method, introduced via a short paper called ‘Apophenia As Method—Or, Everything Is Either A Metaphor Or An Analogue Computer’ was tried out at the workshop along with a wide range of other generative and experimental techniques, collected in a zine. We intend to develop this direction further in more work, since the potential of ‘apophenia as method’ has interesting implications for the intersection of machine learning and creativity in particular. More on this next year.

⇧ New Metaphors at the CHI 2018 Disruptive Improvisation: Making Use of Non-Deterministic Art Practices in HCI workshop in Montreal

Other work from the lab in 2018 on new methods for design and creativity included: Dan’s participation in the Sketch Model Summer Workshop at Olin College of Engineering in Needham, MA, in June, funded by the Mellon Foundation, in which a wonderful group of people from technology, humanities, and the arts, led by Sara Hendren, Benjamin Linder, Jonathan Stolk, Deb Chachra, and Jonathan Adler, explored interdisciplinarity and how to bring both critical and creative methods into engineering education; and a paper applying ideas from R D Laing and Gregory Bateson in the context of investigating people’s understanding of systems, presented by Dan at the (new) Systemic Design Association‘s RSD 7 conference at Politecnico di Torino in October, followed by a fascinating ‘de-conference’ at the Monviso Institute in Ostana.

⇧ Left: Sketch Model Summer Workshop at Olin College; Right: The systemic design community explores Ostana, Piedmont

Sarah Foley presented her new method for designers to rethink services and human-technology relations, Service Fictions through Actant Switching, at the Design Research Society’s DRS 2018 conference in Limerick in June. The paper was developed from Sarah’s MDes thesis (advised by Dan Lockton and Cameron Tonkinwise) and offers an approach combining actor-network theory, design fiction, and service design to generate speculative, provocative ideas for the future of services.

⇧ Left: Sarah Foley presents her Service Fictions project at DRS 2018; Right: Drinks in Limerick after the DRS 2018 Designing for Transitions track.

Also at DRS 2018, Dan Lockton joined Joanna Boehnert (Loughborough University) and Ingrid Mulder (TU Delft) to chair a full-day track, Designing for Transitions. Building on work around Transition Design emerging from Carnegie Mellon, but also other systemic approaches to designing with wider social and environmental change in mind, the ten papers in the track explored an expanded field for design research seeking to engage with change at scale in time and place. Our editorial provided an overview of some issues and challenges in the field as we see it developing. As part of the track, Dan Lockton and friend of the Lab Stuart Candy‘s paper ‘A Vocabulary for Visions’ brought together many of the themes of imaginaries, futures, and new metaphors that underpin the Lab’s work. We covered (briefly) a set of concepts which can be thought of as seven ‘ways of seeing’, for tackling the ‘visionary’ aspect of designing for transitions—lenses, imaginaries, backcasting, dark matter, circularity, experiential futures, and new metaphors—drawing on work by a range of people and different disciplines. Dan Lockton was also a member of the Conversations committee for DRS 2018, and a discussant at PhD by Design Limerick.

New types of interface
A big theme through our work is exploring new kinds of interface design, through various perspectives including a more qualitative approach. One such project this year is Electric Acoustic (Shengzhi Wu, Gray Crawford, Devika Singh, Dan Lockton) which explores data sonification — turning data into sound — as an alternative way to engage with patterns in energy use data. Building on Dan’s previous Powerchord project (developed with Flora Bowden at the RCA), Electric Acoustic is supported by Carnegie Mellon College of Fine Arts’ Fund for Research and Creativity, using data provided by CMU Facilities Management Services. Following a public engagement workshop at the Pittsburgh Children’s Museum in fall 2017 for Pittsburgh Maker Faire, we have built a multi-modal prototype also incorporating cymatics (vibration displays), which we installed in May 2018 during CMU Design Week. Cymatic displays seem to offer some interesting possibilities for more qualitative ways of representing the effects of phenomena and their interactions with each other, and we’re hoping to explore this further in some different contexts.

⇧ Ubiquitous Inclusion, by MacKenzie Cherban

⇧ Silent Scene, by Chang Hee Lee

In general, shifts in sensory experiences for interaction design have been an area of lots of interest for people in our community this year. MacKenzie Cherban‘s MDes thesis, Ubiquitous Inclusion (advised by Dan Lockton) examines the design process and the role of technology in relation to the d/Deaf community, through building on the affordances of sign language (ASL) and participatory futuring, arriving at an ecosystem connecting ‘future artifacts’ developed from participants’ ideas, including a ‘machine learning for personal use’ approach to the smart home (above left), trained using Wekinator. Dr Chang Hee Lee — who passed his PhD at the RCA this October! — (supervised by John Stevens and Dan Lockton) has been investigating synaesthesia and design for the last four years, and as a development from this work created Silent Scene (above right), an exploration of “zero interaction” as a playful mode of experience. In the grand tradition of Claude Shannon / Marvin Minsky’s Ultimate Machine, Silent Scene is “a stationary device that appears to do nothing. However, when there are no humans in its environment — when no sound, motion, or light is detected — it secretly starts to create beams and rays of stunning colors. The device will not function if anyone is near it.” Here’s Chang’s DIS 2018 abstract with Dan Lockton and Ji Eun Kim, and a write-up in Interactions‘ Demo Hour. Dr Lee’s fellow RCA Innovation Design Engineering PhD researcher — and medical practitioner — Dr Dave Pao (also supervised by John Stevens and Dan Lockton) is continuing with his redesign of electronic medical record interfaces for doctors, to use a more visual, qualitative style which enables not just better usability but also higher self-perceived clinician competence.

⇧ Dixon Lo’s ShapeShift demo

⇧ CHI 2018 presentation of Dixon Lo’s Experiential Augmentation project (presented by Dan Lockton)

How can we make use of the affordances of virtual and augmented reality — spatial computing more widely — to create new kinds of interface, with new possibilities for understanding, rather than just adapting existing paradigms? Dixon Lo‘s CHI 2018 paper, based on his MDes thesis Experiential Augmentation (advised by Stacie Rohrbach and Dan Lockton), examined qualitative indexical visualizations for AR building on our learned understanding of physical phenomena in the real world, from shadows to floating, arriving at recommendations for designers working in this space. Dan presented Dixon’s paper (see video). A very different approach is being taken in Gray Crawford‘s thesis project, Assimilating Hyperphysical Affordances (advised by Dan Lockton and Daragh Byrne), in which he is exploring neuroplasticity in relation to “more unorthodox physical phenomena” in VR — can we learn to interact in ways which are very different to the real world? Are there opportunities for new kinds of interfaces?

Working on raymarched hands in VR, seeing how UI might be directly incorporated into / emerge from the body

Super satisfying how grabbed objects fuse with the body —
mirrors a bit how we mentally assimilate tools into our body schemata@LeapMotion @kevinwatters @compositeredfox pic.twitter.com/XZXahGZfop

— Gray Crawford (@graycrawford) October 19, 2018

⇧ A demonstration of one of Gray Crawford’s experiments

Intelligences
Another theme running through our work this year has been around ‘intelligences’ and the questions of other minds (whether human, animal, or artificial). Within Carnegie Mellon, we’re situated in (and saturated with) an environment strongly flavored with AI development — and indeed, increasing consideration of ethics via an initiative funded by K&L Gates — but interaction design’s engagement with the changing intelligences around us has a lot of potential for critical and generative exploration and development.

⇧ Projects on show at Where Are the Humans in AI?, May 2018 — the class show for the Environments Studio ‘Intelligence(s) in Environments’

Dan Lockton’s 2018’s junior Environments Studio class Intelligence(s) in Environments — Maayan Albert, Gautam Bose, Emma Brennan, Cameron Burgess, Aisha Dev, Anna Gusman, Monica Huang, Soonho Kwon, Marisa Lu, Jessica Nip, Lucas Ochoa, and Helen Wu — examined intelligence of different kinds, from social interaction to theory of mind to everyday interaction with AI, through practical projects and guest talks focused on investigating, understanding, and materializing intelligence and other invisible and intangible qualitative phenomena and relationships (covered by Chris Togneri here). An excellent group of projects included investigating how people see different web services as analogous to rooms in their home (Maayan Albert), communal spaces as manifestations of others’ thought processes (Emma Brennan), communicating intangible emotions with computers via poetry and sculpture (Monica Huang), physicalizing moral codes and decision-making (Aisha Dev), new visual approaches to end-user programming (Cameron Burgess), Wekinator-trained voice control of 3D modeling (Anna Gusman), and a VR Museum of Taste (Soonho Kwon). See all the projects here. The highest profile project, Emoto AI, by Marisa Lu, Gautam Bose and Lucas Ochoa offers an alternative embodiment for phone-based virtual assistants such as Siri, enabling them to transform into a ‘sidekick’ (see also Michal Luria‘s work) making use of nonverbal communication cues through expressive motion. Emoto AI received an honourable mention in the Fast Company Innovation by Design Awards 2018 and a paper, ‘Emoto: From Phone to Emotive Robotic AI Sidekick’ by Gautam Bose, Lucas Ochoa, Marisa Lu and Dan Lockton has been accepted to TEI 2019‘s work-in-progress track.

⇧ Emoto AI Sidekick by Marisa Lu, Gautam Bose and Lucas Ochoa

The Environments Studio projects received a range of guest critique throughout, including a visit from Bruce Sterling and Jasmina Tesanovic, and culminated with a three-day show, Where Are The Humans in AI?, in May 2018, following which Cameron Burgess, Emma Brennan, Monica Huang, and Gautam Bose exhibited their projects at Data & Society’s Future Perfect event in New York, organized by Ingrid Burrington.

⇧ Emma Brennan and Cameron Burgess demonstrate their projects at the Data & Society Future Perfect event, New York

Going in depth on a specific dimension of our interaction with other intelligences, Meric Dagli‘s MDes thesis Designing for Trust (advised by Dan Lockton and Daragh Byrne) examined interaction design for trust in the context of multiple chatbots — developing designs guidelines for maintaining and increasing trust in scenarios where multiple virtual e-commerce agents collaborate with each other. Meric won a Kynamatrix Research Network Innovation through Collaboration Grant for his project.

The major question of ‘other minds’ is, of course, how can we ever know how each other thinks? How can we understand other people’s thought processes and emotions, when we have no way of experiencing others’ experiences? In some ways, much of the Lab’s work is about externalizing imaginaries and mental models, or developing tools for imaginaries to be externalized, to enable sharing and discussion. One field where this approach has a particularly practical application is in mental health — using design methods to help people think about and explore creative ways to describe, talk about, and share our own often invisible experiences. According to research compiled by the Wellcome Trust (UK), “one in four people will experience a mental health problem in any given year”, and “75% of people with a mental health problem develop it before the age of 24”. Sales of books on anxiety are “soaring”. Carnegie Mellon students, in common with many people in high-pressure environments, can experience a broad range of mental health issues.

⇧ Projects from New Ways to Think: Materializing Mental Health

Yet as a society, we don’t always have good ways of talking about mental health. In New Ways To Think: Materializing Mental Health, an eight-week research studio run by the lab, undergraduates, Master’s students, and PhDs from CMU’s School of Design, School of Art, Human-Computer Interaction Institute, Tepper School of Business, and Integrated Innovation Institute explored how we can adapt participatory design and facilitation methods, often used in user experience, service design, and working with communities, to a mental health context. We believe they have the potential to help people capture qualitative dimensions of their experiences, to make them palpable, to enable discussion, reflection, and peer support. Our initial focus has been working within the Carnegie Mellon community, including receiving very valuable input from the university’s Counseling and Psychological Services, but we hope that the methods developed can be of use more widely through further development. Four projects — Lexicon of Feelings (Aisha Dev, Kailin Dong, Katie Glass, Zhiye Jin, Soonho Kwon, and Jessica Nip), Emotional Modeling (Laura Rodriguez, Katie Herzog, Josh LeFevre, Nowell Kahle, and Arden Wolf), Empathy Rock Garden and Personalized Potions (both by Jen Brown, Carlie Guilfoile, Michal Luria, Uluwehi Mills, and Supawat Vitoorakaporn) — each work with different aspects of mental health, from anxiety and stress to loneliness, to enabling feelings that perhaps don’t have a name yet to be expressed and shared. We are currently working on finding ways to publish what we’ve done so far, and take some of this work further.

Futures
‘New ways to live’ is a dimension of the Lab’s work that brings together imaginaries of futures and some of the design for behavior change work on which Dan’s research was founded. The basic premise is that if we can develop better ways of helping people imagine themselves living and acting differently then this makes larger-scale behavior and practice changes for sustainability easier to achieve, ultimately, for humanity and for the planet. (We draw here on some of the experiential futures framing developed by our CMU colleague Stuart Candy.) Starting in November 2018, with a short course called New Ways To Live: Future Pittsburgh — and continuing in 2019 with a publication project, lab researchers Rachel Gray Alexander and Saloni Sabnis, with students Aisha Dev, Kailin Dong, Monica Huang, Soonho Kwon, Jessica Nip, Nicole Pinto, Tamara Amin, Jen Brown, Jeffrey Chou, Katie Herzog, Laura Kelly, Michal Luria, Ulu Mills, Laura Rodriguez, Devika Singh, and John Zoppina (undergrads, Master’s, PhDs, and staff, from Design, Environmental Engineering, Psychology, Business, Human-Computer Interaction, Professional Writing, and University Advancement) have been developing projects exploring the Pittsburgh of 2030 — speculative (but well-informed) scenes from possible future everyday life and work in the city, shot through a more realistic lens of the kinds of small businesses and cultural phenomena that are present here rather than the entirely shiny visions of automation that are sometimes proposed. This could be relevant to many rust-belt cities in the US, and former industrial towns elsewhere too. More on this project in due course.

The Imaginaries Lab, represented by Dan Lockton, is excited to be a founding member of Plurality University (U+), a Paris-based global collective “that detects, connects, and federates people or organizations who mobilize the resources of the imaginary to broaden the scope of thinkable futures: activist artists and sci-fi authors, speculative designers, reflexive utopians, creative futures thinkers, engaged researchers, etc”. We’re in some excellent company and looking forward to building on ideas developed at the founders’ meeting in Paris at the end of November.

⇧ Plurality University Founders’ Meeting, New Metaphors workshop

Looking ahead

We’re actively seeking collaborations, projects where we can contribute, and opportunities to apply for funding together. We’re also pretty experienced at running workshops, short courses and projects in both commercial and academic contexts. If you’re interested in any of the ideas or methods we’re working on, or think we might be able to work together in 2019, internationally or within the US, please do get in touch. As we look to the future, the Lab is exploring the options for new funding models, host institutions and partners, inside and outside of academia.

Other activities from the Lab in 2018

Finally, we should mention some of the other activities the Lab’s been involved with in 2018. We’ve been pleased to welcome guest speakers for the classes we run, both in-person and virtually, including Simone Rebaudengo, Bruce Sterling, Jasmina Tesanovic, Emily LaRosa, David Danks, Madeleine Elish, Deepa Butoliya, Jill Simpson, Tobias Revell, Cennydd Bowles, Viviana Ferrer-Medina, Cheryl Dahle, and Emily Blaze — thank you all for your time. Dan Lockton and Ahmed Ansari’s MDes Seminar III class have published a great set of articles about the research they’re doing. Dan Lockton has talked about the Lab’s work for TEDx University of Pittsburgh, Carnegie Mellon Human-Computer Interaction Institute (thanks to Brad Myers), IxDA Pittsburgh (thanks to Simon King), CMU’s dSHARP digital humanities group (thanks to Scott Weingart), CMU School of Architecture’s ‘Introduction to Ecological Design & Thinking’ (Dana Cupkova), and for London College of Communication’s MA Communication Design (thanks to Tobias Revell). In terms of professional service, Dan has been an Associate Chair for the CHI 2018 Design subcommittee, an invited discussant at PhD by Design in Limerick, a jury member for the IxDA Interaction Awards 2019, and a guest critic / respondent for CMU School of Architecture’s EX-CHANGE in May 2018. PhD advisees Chang Hee Lee (RCA), Michael Arnold Mages (CMU) and Deepa Butoliya (CMU) have all passed and are embarking on academic careers, at the RCA, Northeastern, and Stamps (University of Michigan) respectively.

We’ve published a bit during the year, mostly at conferences:

• Joanna Boehnert, Dan Lockton, and Ingrid Mulder (2018). ‘Editorial: Designing for Transitions’. DRS 2018: Design Research Society, 25–28 June 2018, Limerick.
• Sarah Foley and Dan Lockton (2018). ‘Service Fictions through Actant Switching‘. DRS 2018: Design Research Society, 25–28 June 2018, Limerick.
• Chang Hee Lee, Dan Lockton, and Ji Eun Kim (2018). ‘Exploring Cognitive Playfulness Through Zero Interactions’. DIS 2018: ACM Conference on Designing Interactive Systems, 9–13 June 2018, Hong Kong.
• Chang Hee Lee, Dan Lockton, David Verweij, David Kirk, Kay Rogage, Abigail Durrant, Aubree Ball, Audrey Desjardins, Adam Haar Horowitz, Ishaan Grover, Pedro Reynolds-Cuéllar, Oscar Rosello, Tomás Vega, Abhinandan Jain, Cynthia Breazeal, and Pattie Maes (2018). ‘Demo hour‘. Interactions 25, 6 (October 2018), 10-13. DOI: https://doi.org/10.1145/3279993
• Dixon Lo, Dan Lockton, and Stacie Rohrbach (2018). ‘Experiential Augmentation: Uncovering The Meaning Of Qualitative Visualizations When Applied To Augmented Objects’. CHI 2018: ACM Conference on Human Factors in Computing Systems, April 21–26 2018, Montreal.
• Dan Lockton (2018). ‘Old Rope? Laing’s Knots and Bateson’s Double Binds in Systemic Design’. RSD 7: Relating Systems Thinking and Design Symposium, 24–26 October 2018, Turin.
• Dan Lockton and Stuart Candy (2018). ‘A Vocabulary for Visions in Designing for Transitions’. DRS 2018: Design Research Society, 25–28 June 2018, Limerick.
• Dan Lockton, Some Cracks In The Paving, and Water Trapped In The Window Of A British Rail Class 450 Train Carriage (2018). ‘Apophenia As Method—Or, Everything Is Either A Metaphor Or An Analogue Computer’. Disruptive Improvisation: Making Use of Non-Deterministic Art Practices, workshop at CHI 2018: ACM Conference on Human Factors in Computing Systems, 21–26 April 2018, Montreal
• Dave Pao, John Stevens, Dan Lockton, and Netta Weinstein (2018). ‘Electronic Medical Records: Provotype visualisation maximises clinical usability’. EVA London 2018: Electronic Visualisation & the Arts, 10–12 July 2018, London.
• Dave Pao, John Stevens, Dan Lockton, and Netta Weinstein (2018). ‘Design better EPR: a mixed methods survey and ‘test drive’ comparing clinical usability across two systems and a provotype interface’. HIV Medicine 19, S99-S100.

---

 

Thanks

Thanks to everyone who’s helped this year, and all the students and participants in our projects and classes, to event organizers who’ve taken us all over the world, to Carnegie Mellon colleagues who’ve understood what we’re trying to do, and to our long-suffering families. Happy New Year to all: 2019 will be better.