Interactions

Authors: Jessica Hullman
Posted: Mon, August 05, 2019 - 3:10:09

---

Few people in visualization research have had careers as long and as impactful as Ben Shneiderman. I caught up with Ben over email in between his travels to get his take on visualization research, what’s worked in his career, and his advice for practitioners and researchers.

Jessica Hullman:  How would you answer the question: What is visualization research?

Ben Shneiderman: First let me define information visualization and its goals; then I can describe visualization research.

Information visualization is a powerful interactive strategy for exploring data, especially when combined with statistical methods. Analysts in every field can use interactive information visualization tools for:

• more effective detection of faulty data, missing data, unusual distributions, and anomalies;
• deeper and more thorough data analyses that produce profounder insights; and
• richer understandings that enable researchers to ask bolder questions.

Like a telescope or microscope that increases your perceptual abilities, information visualization amplifies your cognitive abilities to understand complex processes so as to support better decisions. In our best moments, information visualization users work on problems that address the grand challenges of our time, such as the UN Sustainable Development Goals.

Visualization research seeks new visual displays, control panels, features, and workflows that improve the capabilities of users. To accomplish these goals, visualization researchers develop perceptual and cognitive theories that guide design, in concert with developing new tools. Visualization researchers also develop quantitative and qualitative evaluation methods to validate their hypotheses and refine their theories.

My strong way of promoting information visualization is to declare that it is such a powerful amplifier of human abilities that it should be illegal, unprofessional, and unethical to do data analysis using only statistical and algorithmic processes. (However, as with all visualizations, the accompanying design has to account for users with visual disabilities by using sonification or other methods. Visualization’s potency in revealing unusual distributions and interesting clusters may productively encourage statistics and algorithm designers to extend their methods to detect these patterns as well as incorrect, anomalous, inconsistent, and missing data.)

And finally, remember that: the purpose of visualization is insight, not pictures. By addressing meaningful problems and difficult decisions, we can help leaders and managers to be more effective.

Hullman: You’ve obviously made some major contributions to human-computer interaction and visualization research. Take, for example, direct manipulation and dynamic queries, two foundational concepts. Can you give us a sense of what it was like to be “pioneering” back when visualization was still a nascent field?

Shneiderman: I see direct manipulation as providing key principles which still guide design of most new technologies:

• Continuous representations of the objects and actions of interest with meaningful visual metaphors
• Physical actions or presses of labeled interface objects (e.g., buttons, sliders, etc.) instead of complex syntax
• Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately.

It was thrilling to be working on user interfaces, just as graphical user interfaces were emerging. The 1982 Gaithersburg, MD, conference was an historic event that triggered the founding of the ACM SIGCHI group. We hoped to draw 200 to 300 people from computing, psychology, and human factors to that conference, so it was electrifying when 906 attendees from even more diverse disciplines showed up.

My recently published book Encounters with HCI Pioneers: A Personal Journal and Photo History (Morgan & Claypool Publishers, 2019) tries to tell the story of how the field of human-computer interaction was formed and the bold, creative, and generous personalities who were involved.

Hullman: How do you perceive the work that you and your collaborators have done in these fields? For instance, how do you describe the contributions of your work when talking to people who don’t know these fields very well?

Shneiderman: The easiest stories to tell at airport encounters and dinner parties are about the creation of the 1) web link: a form of direct manipulation selection by clicking on highlighted links and 2) the high-precision touchscreen strategy for small keyboards called lift-off, which is still used on the Apple iPhones and many other devices. These two small innovations have had a huge impact, enabling billions of users to have easier access to information, be in communication with family, get medical care, and much more. These innovations and their evaluations were published openly with no patent protection, allowing widespread refinement and application. I thought they were both important and valuable, but I did not anticipate the immense impact they would have, nor that they would become so common and obvious to so many users.

Treemaps used in FinViz.com, 2019.

My airport and dinner discussion partners often take a moment to realize that these innovations had to be invented, but then they want to know more, so I tell the story of Steve Jobs’s visit to our lab in October 1988 and how he went from demo to demo saying “That’s great! That’s great! That sucks! That’s great! That sucks!” He wasn’t always right, but he had strong opinions. If they want to know more, I talk about how direct manipulation ideas led to dynamic queries, information visualization, and treemaps. The most fun is to show all these innovations through commercial applications, such as the FinViz.com treemaps, on my mobile phone.

Early implementation of the treemap for hard drive overview.

While treemaps have had great success, being used on thousands of websites, added to Excel in 2014, and used in most commercial information visualization tools, I’m also proud of the ideas that went into creating the commercial success story of Spotfire. It was a great experience to help take the academic ideas of direct manipulation and dynamic queries into a widely used product that revolutionized pharmaceutical drug discovery, oil/gas exploration, manufacturing, and many other application areas. Spotfire enabled rapid exploration of large datasets with many variables to discover the distributions, anomalies, outliers, and relationships among variables. Early successes included discovery of a new compound for a life-saving drug, a hidden performance failure in a large oilfield, and a manufacturing flaw that had caused more than a billion-dollar loss. It also paved the way for the hugely successful Tableau, which also came from academic colleagues, including Pat Hanrahan and Jock Mackinlay, a few years later. Tableau is now a $10 billion company with millions of users and many competitors.

Other satisfying stories are the widespread use of our network analysis and visualization tool, NodeXL, which pioneered new layout strategies that more clearly showed meaningful clusters and communities. The six patterns of Twitter discussions were an important insight: NodeXL is the most widely used tool for education about network analysis; our book on its use will appear in second edition in 2019. The striking example below shows the polarized voting pattern in the U.S. Senate. The tightly linked red Republican Senators are at the top, while the tightly linked blue Democrats are at the bottom. The two green Independent Senators typically vote with the Democrats
 

A visualization of voting patterns in the U.S. Senate created using NodeXL.

Finally, the work on event analytics, especially electronic health records, to show aggregated patient histories opened up new possibilities for visual analytics projects that revealed successful or failed treatment patterns. The EventFlow tool has been used by 50 to 60 projects, opening new directions for information visualization research.
 

A visualization in EventFlow.

Hullman: Is there a single contribution you’ve made, either working alone or in collaboration, to visualization research are you most proud of?

Shneiderman: Yes, my students! My greatest satisfaction was to work with Catherine Plaisant for 30-plus years to train a new generation of students who did projects, earned degrees, and went on to do great work on their own. We engaged with students in ways that promoted their self-efficacy to build their capacity and confidence as researchers and practitioners. The University of Maryland Human-Computer Interaction Lab (HCIL) community encouraged collaborations to improve the research, writing, speaking, and social skills of each member. Paper clinics to review each other’s writing and practice talks to improve slide presentations became common formats in which faculty, staff, and students worked hard to improve the products and skills of every HCIL member.

Hullman: If readers were going to read one thing you’ve written, what would you recommend and why?

Shneiderman: If they are serious, they should read the 6th edition of Designing the User Interface: Strategies for Effective Human-Computer Interaction, but that is asking a lot. An easier starting point would be the 1996 paper “The Eyes Have It: A Task by DataType Taxonomy of Information Visualizations” (Proc. IEEE Symposium on Visual Languages ’96). This keynote paper was hastily written close to the publication deadline, but the ideas were fresh and my informal phrasing of what I called the Information Visualization Mantra (overview first, zoom and filter, then details-on-demand) has attracted many readers. The simple ideas in this paper, presented at a second-tier conference, has received more than 5000 citations.

Hullman: At a few times in its history the visualization community has reflected on its value to the world, sometimes using terms like the “death of visualization” to refer to the way the research community has evolved from existing to help other scientists solve problems to existing more independently. What’s your take on the state of visualization research today? Are we past any risk of being irrelevant? Are we focused on the right things?

Shneiderman: I have strong confidence in the importance of information visualization, but it will take a generation for it to gain acceptance by diverse audiences. Our challenge is to show how algorithmic and automatic approaches based on statistical methods can be improved by interactive information visualization. Just as telescopes, microscopes, cameras, x-ray machines, and sonograms extend our perceptual abilities, information visualization amplifies our cognitive abilities. We will need to shift education curricula in elementary and high schools so that they teach data visualization literacy and fluency with interactive information visualization tools.

Hullman: You have always advocated for visualization research that has a real and practical impact in the world. How do we make visualization useful and relevant for practitioners? And what is the best way to bridge gaps between people who do research in visualization and those who use visualization in their profession (designers, analysts, product developers, etc.).

Shneiderman: By working with practitioners, information visualization researchers can accelerate their research by testing their tools and methods on real problems. The evidence is clear that the payoff for working with practitioners is strong. It is also very satisfying to help others and I find that our work has immense value in dealing with real problems whose solution promotes understanding and improves lives. Several of us make the case in the paper “Apply or Die: On the Role and Assessment of Application Papers in Visualization.”

Hullman: Is there anything (else) you think visualization researchers should do more of?

Shneiderman: Have fun in your work, do your best to save the world, and tip your drivers generously.

Hullman: A few years ago there was some discussion among visualization researchers about whether visualization research had “grand challenges” in the same way that other fields have (e.g., NP-hard problems in more formally theoretical subfields of CS). Do you have any thoughts on this question? Are our problems as well defined and all encompassing as those in other branches of computer science? Or are visualization and human-computer interaction research fundamentally different, and if so, what would you say is the important difference?

Shneiderman: The information visualization community is working on some of the grandest challenges of our time: amplifying human cognition in the exploration of data. Data science and explainable artificial intelligence are clearly among the biggest opportunities of our time—information visualization is vital to successful outcomes for both topics. We’ve made important contributions and had high impact already, but our influence will grow as visualizations become more important parts of products and data visualization literacy grows. Other grand challenges for information visualization researchers are to improve storytelling capacity for the general public, to engage users to explore on their own, and to support researchers in understanding causality. A larger challenge is to encourage the shift from rationalism, which assumes that algorithms are the answer, to empiricism, which assumes that continuous exploration, persistent questioning, and vigorous dialog will promote a deeper understanding of our world.

Hullman: These days many people are drawn to work with visualization but may not know how to best contribute. What advice would you give a new Ph.D. student who wants to have a real impact on visualization use in the world?

Shneiderman: Start by working on a real problem—one that you have or that you get from someone else. Working on real problems leads to better theories and better tools.

Hullman: What about a visualization practitioner, like a designer or developer?

Shneiderman: Build something, test it, improve it, and then do it again on another problem. Keep producing, keep working to serve the needs of real users working on real problems with real data.

Hullman: What about someone who is already doing visualization research, say in their early to mid career (like my colleagues and I!) Sometimes it’s hard to decide between problems to devote one’s time to, and to know how much one should care about trying to increase one’s metrics (citations, etc.) so as to achieve greater “visibility” versus to work on the problems that seem most important, even if they are not considered “core InfoVis topics” (uncertainty visualization being one example that comes to mind). How did you choose your problems? Did you ever experience any tension between the problems you wanted to work on and those that the research community thought were important?

Shneiderman: While I did have topics that I was attracted to, I invested greater time and energy where there was a person who cared about the result. Working to serve the needs of a real person raises the value of your work and focuses your attention on real problems, not the imaginary ones you invent. My 2018 paper “The Twin-Win Model: A Human-centered Approach to Research Success” showed that papers written with a non-academic co-author produced 2 to 10 times as many citations as papers that had only academic authors.

Hullman: Do you have any personal heroes, or people you look up to?

Shneiderman: Heroes in our field: Stu Card was always a wonderful colleague whose work was deep and important. Current leaders that really impress me are Maneesh Agrawala, Katy Börner, Jeff Heer, and Tamara Munzner, who are remarkably productive, creative, and influential. Also the amazing team of Fernanda Viegas and Martin Wattenberg—constant sources of brilliant and beautiful projects.

Larger figures: Marshall McLuhan, Buckminster Fuller, Lewis Mumford, Rachel Carson, Donald Knuth, Grace Hopper, Mihaly Csikszentmihalyi, Rita Colwell, and Walter Isaacson. A few years ago Robert Kosara invited me to write a longer list of those who influenced me.

Hullman: Can you tell us a bit about what you’re thinking about now? What problems have most motivated you recently?

Shneiderman: I still see big opportunities in medical applications:

• Analyses of electronic health records to support clinical research that detects successful treatment patterns,
• Personal data from patients who are tracking their disease and health, and
• Wellness data, such as exercise, diet, stress, sleep, etc., from quantified self users.

Visualization could do much to improve the presentation of even basic information such as blood pressure, sleeping patterns, or mental health over time. Visualization is also an important component for deeper understanding of key issues in climate change, sustainability, environmental protection, cybersecurity, community safety, and many other domains.

I’ve also been addressing policy issues in talks such as "Algorithmic Accountability: Design for Safety," and writing about human-centered design philosophies to guide new technology. My 2016 book The New ABCs of Research: Achieving Breakthrough Collaborations continues to draw invitations to speak, while suggesting new ways to raise research impact by choosing the right problems and forming the right teams.

Hullman: Any parting thoughts you’d like to leave us with?

Shneiderman: Visualization is a powerful approach to understanding the world, so we enjoy our work to make it more central to research, education, journalism, and public policy. It will take time and energy, but embrace the struggles, take on the challenges, and celebrate your successes. These are worthy efforts.

Ben Shneiderman in Torres del Paine National Park in Chile, January 2019.

Ben Shneiderman is a Distinguished University Professor in the Department of Computer Science, Founding Director (1983–2000) of the Human-Computer Interaction Laboratory (HCIL), and a member of the UM Institute for Advanced Computer Studies (UMIACS) at the University of Maryland. He is a Fellow of the AAAS, ACM, IEEE, and NAI, and a Member of the National Academy of Engineering, in recognition of his pioneering contributions to human-computer interaction and information visualization. His widely used contributions include the clickable highlighted web-links, high-precision touchscreen keyboards for mobile devices, and tagging for photos. Shneiderman’s information visualization innovations include dynamic query sliders for Spotfire, development of treemaps for viewing hierarchical data, novel network visualizations for NodeXL, and event sequence analysis with EventFlow for electronic health records.

Ben is the co-author with Catherine Plaisant of Designing the User Interface: Strategies for Effective Human-Computer Interaction (6th ed., 2016). With Stu Card and Jock Mackinlay, he co-authored Readings in Information Visualization: Using Vision to Think (1999). His book Leonardo’s Laptop (MIT Press) won the IEEE book award for Distinguished Literary Contribution. He co-authored Analyzing Social Media Networks with NodeXL (Elsevier, Second Edition 2019) with Derek Hansen and Marc Smith. The New ABCs of Research: Achieving Breakthrough Collaborations (Oxford, April 2016) has an accompanying short book Twin-Win Research: Breakthrough Theories and Validated Solutions for Societal Benefit: Second Edition (2019). His reflections on the growth of human-computer interaction will appear in 2019: Encounters with HCI Pioneers: A Personal Journal and Photo History (Morgan & Claypool Publishers).

---

Posted in: on Mon, August 05, 2019 - 3:10:09

Jessica Hullman