Visualization Evaluation Criteria
This page enumerates the 9 criteria we’ll use for evaluating visualizations in this course. Not every criterion will be relevant for every assignment and you will not be responsible for priciples that have not yet been discussed in class. Evaluating visualizations can be highly subjective, so no particular weight is assigned to any property, rather these are guidelines for how we’ll think about what to look for in a good visualization.
Note that it will not always be possible to perfectly optimize all of these criteria simultaneously. For example, creating a significantly innovative or artistic visualization may mean sacrificing effectiveness to some extent. When creating visualizations your goal should be to balance these criteria as well as possible in order to have your visualization accomplish your goals.
Expressiveness
The principle of expressiveness dictates that no information present in the data should be hidden by the visualization. To paraphrase Jock MacKinlay: a visualization is expressive if in encodes all the facts in a set. (We’ll leave the second part of his definition to a different section). When evaluating visualizations according to this metric we are looking to see that any information ostensibly shown in the visualization is easily parse-able by a reader.
If individual observations are shown, each observation should be distinguishable in the visualization. Avoid overplotting, hiding observerations beyond axis limits or removing observations without explicit justification.
If aggregations or density estimates are shown, make sure they are not over-smoothed or abstracting away too much information.
Make sure that axes and labels are present, legible, and span the full range of the presented data.
Effectiveness
Effective visualizations allow readers to easily answer questions about the data using the visualization. This means that the visualization is designed such that comparisons are easy and accurate and that points of interest can be easily identified.
When evaluating effectiveness we are most interested in a readers ability to distinguish important qualitative differences.
Aesthetic mappings, geometries and other elements of the visualization should be chosen to facilitate easy and accurate comparison. This means using the hierarchies of aesthetic mappings to assign effective mappings to the most important variables.
Non-linear or non-cartesian scales should be used carefully, where appropriate, ensuring that readers can understand the relevant values. In particular, color scales should be chosen to be easy to read and distinguish.
Highlights, annotations and multiple encodings may be used to enhance effectiveness by increasing the legibility of specific, important comparisons.
Intuitiveness
Most good visualizations are intuitive, meaning that readers should be able to understand how the visualization works with little effort and take away the primary message quickly and easily.
When evaluating the intuitiveness of a visualization we are interested in minimizing the amount of time and effort a reader needs to come to accurate conclusions about the data as well as minimizing a readers frustration.
The visualization should align with established conventions and expectations for aesthetic mappings, scales and other elements (e.g. time on the x-axis, stacks represent proportions)
The complexity of the visualization should be well-balanced with its goals. Too many mappings, transformations and non-standard layouts make visualizations hard to understand.
Available interactions should be clear and straightforward, the reaction of the visualization should match the users expectations
If the visualization has a central takeaway, this should be highlighted and explained, rather than leaving it to the reader to guess.
Honesty
Honest visualizations avoid misleading readers about the properties of the data. Here we’ll consider the second part of Jock Mackinlay’s definition of expressiveness: a visualization is expressive if in encodes all the facts in a set and nothing else. That is, we not only want to tell the whole truth, but also nothing but the truth.
When evaluating visualizations for honesty we are interested in whether the takeaways of a casual reader will match those of someone who does a deeper dive into the data.
The visualization should align with established conventions and intuition, for scales (e.g. up is larger, red is hotter, time goes from left to right etc.), geometries (e.g. stacks/pies correspond to proportions, lines indicate continuity), coordinate systems (e.g. radial coordinates indicate cycles) and other elements.
Differences are shown in proportion to their importance; a visualization should not be zoomed in to make a tiny difference look large or vice-versa, ineffective aesthetic mappings should not be chosen to hide important details.
Auxiliary choices such as orderings and categorical assignments, should not imply meaning where there is none. Remember the Alabama first caution!
Accessibility
Data visualization is inherently a visual medium, but this can limit the accessibility of the information presented to many readers. When we are creating visualizations, we should always be mindful to make sure our choices don’t exclude those with visual impairments or different backgrounds.
When evaluating visualizations for accessibility we are interested seeing how much information can still be parsed from the visualization when accounting for possible limitations of our readers.
Colormaps should be chosen such that values are still distinguishable to readers with common forms of colorblindness or redundant encodings should be used.
Text and geometries should be large and distinct enough to be easily readable to readers with impaired vision. Alt-text should always be provided for readers relying screen-reader software.
It should not be assumed that readers are native English speakers, so text should avoid overly esoteric words and phrases.
Impact
A good visualization will help a reader understand something about a dataset, a great visualization will have them thinking about it long after they move on from looking at it. Since the goal of visualization is most often to educate and inform readers, we want to make sure that they actually remember the takeaways of what they are presented. To this end we want to make sure our visualizations are memorable and impactful.
Impact is an especially subjective quantity, when evaluating it we will primarily focus on how much the visualization stands out (without violating other principles).
Aesthetic choices that are thematic are a common way to make visualizations more impactful. This could mean color, geometry or other aesthetic choices that are relevant to the the context of the data being visualized.
Creative, unique and artistic choices can also help to bolster impact. Some impactful visualizations might incorporate artwork, while others might innovate on how the data is presented.
Impact can also be achieved by subverting expectations and introducing elements of shock or surprise. This can be reinforced by first setting up or reinforcing audience expectations, before presenting the surprising takeaways.
Insight
Good visualizations should be insightful, that is, they should teach the reader something new, unexpected, and useful. More often than not this is a function of what they chose to visualize rather than how they choose to visualize it. This means choosing interesting and important data to visualize and making sure that the reader can takeaway lessons that would not have been obvious.
When evaluating the insightfulness of a visualization, we are looking to see that interpreting the visualization what a good use of the reader’s time.
The data visualized should be interesting and unique to the extent possible. Avoid benchmark datasets (e.g. Titanic, iris, penguins, etc.) that have already been visualized many times and contain little new information.
Make sure the reader can take away something interesting or useful from the visualization. E.g. showing that a player’s points-per-game distribution is normal is expected and not very interesting, but showing that they perform particularly well against certain teams or in the post-season might be!
Avoid showing spurious or non-sensical relationships; showing that the popularity of the name Andrea correlates with cottage cheese consumption isn’t particularly useful.
Writing
Visualizations don’t exist in a vacuum and it’s important to provide context and explanation through writing. In particular, visualizations should generally be paired with a title and caption that explain: the purpose and context of the visualization, how to read it, and what the main takeaways are. It is important that these pieces of text are clear and well-written, a poor caption can lead to an otherwise excellent visualization being misinterpreted or dismissed.
When evaluating writing we are looking for text that is clear, concise and illuminating.
Make sure captions and other forms of writing are short and to-the-point. Avoid flowery writing, overly long sentences and over-explanation. Remember that the reader should get most of their information from the visualization itself.
Do make sure to highlight any important takeaways that a reader might miss and help guide the reader through complex visualizations. Make sure they know where to look first and what to look for if it is not obvious.
If necessary, provide context into why the data shown is important and where it came from. The visualization should not be dismissed because the reader doesn’t trust it.
Engineering
This criteria is primarily relevant for interactive visualizations. A good interactive visualization should be well-engineered. That is, it should provide a good experience for the user without bugs, lag or other issues. When evaluating visualization (particularly in a class setting) we may also want to consider the engineering challenge that the visualization presented.
When evaluating engineering we are looking for bug-free and well-optimized code.
Make sure that the visualization is well-tested, including edge cases, and that bugs are addressed, so that the visualization does not freeze, crash or enter an error state from user interaction.
Optimize the visualization so that it runs smoothly on typical hardware for readers and does not have significant lag.
Take on engineering challenges in creating your visualizations, remember that the goal of this class is to learn to build complex and interesting visualizations. Using basic, built-in interactions with Altair will not be graded as highly as custom interactions built with D3.js.