An Accidental Visual Reminder of Humility

I have an image pinned to the noticeboard by my desk that resulted from a moment of incompetence but seemed worth keeping in view. It was generated while working on data during a recent research project. With a team of colleagues (Steve McMullen, Kara Sevensma, and Marj Terpstra), I was investigating the impact of technological change on the learning cultures and self-understanding of Christian school communities. The result was the book Digital Life Together: The Challenge of Technology for Christian Schools. It was a mixed-methods project involving surveys, classroom observations, documentary analysis, case studies, and focus groups. This combining of methods was intended to increase confidence in the results as findings were compared across different forms of inquiry. As might be apparent even from the bare list of methods, the project quickly generated daunting quantities of raw data.

During one phase of the project, my task was to zoom in on the documentary evidence. We asked the schools involved in the project to give us any documents connected to the development of their technology program from its inception more than a decade earlier to the present. These included policies, meeting minutes, letters to parents, professional development presentations, reviews, handbooks, and more. The schools delivered seven DVD disks containing 28,184 files. This gave me a rather large, if not always scintillating reading list.

Over several months I was able to whittle these files down to 815 documents that contained material plausibly relevant to the questions guiding our investigation. The goal was to use these documents, in conjunction with data from the other research methods, to gain a rich understanding of how members of the school communities were thinking about technology and its relationship to Christian education. Through several iterations of coding, we developed a coding manual and set about assigning codes to relevant phrases in these documents, as well as in transcripts of the focus group conversations.

Once coding was done (if only it were as quick as writing that phrase) we were finally in a position to begin analyzing the textual data. We used NVivo, one of the standard software packages for qualitative data analysis. The process included systematically running queries against the data using combinations of codes to look for systematic correlations, seeking things that appeared together in people’s discourse often enough to suspect more than coincidence.

It was at this point that my accident happened.

At the end of a long afternoon of running repeated queries I set up the next query and told it to run without noticing that I had failed to restrict the code parameters. Essentially, instead of telling it to check how often A happened in close proximity to B, I had told it to check how often anything was in proximity to anything. Like setting out to look for evidence that ten-year-olds tend to like ice cream and instead running a query for all of the ways living beings like food.

Being a generally obedient piece of software, it complied. The diagram it gave me is below:

Around the outside are our thematic codes. Each line is a relationship between two of them.

Once I was past my initial exasperation, my reaction was a kind of wonder. Perhaps my aesthetic tastes are unusual, but I find it has a kind of beauty. The myriad threads are suggestive of weaving, the intricate whorls suggest something in motion in the murky depths. It’s not what I wanted: a finding, a testable hypothesis that specific ideas were clustering together in participants’ imaginations. But I still appreciate it as a reminder of humility.

The point of working in this way to analyze the data is to isolate significant patterns from the considerable noise. Those patterns then serve as warrant for our account of what is happening. We arrive at appropriately hedged conclusions about what we take to be the case and report them for others to consider and perhaps replicate. The trajectory is away from the flood of information toward a few things worth highlighting for better reasons than a personal hunch. Given the sheer investment of time, energy, and self in working this trajectory, it is not surprising that we become quite attached to our nuggets of knowledge and inordinately pleased when we Found Something.

What my accidental diagram reminds me of daily is the sheer complexity of the fabric from which we are teasing out strands. And of course, this is only an image of the complexity of data already selected, narrowed, and organized. It is not every possible relationship between every word spoken in our participating schools during the decade under investigation. It is not every possible connection between every possible data point in all the documents given to us, or even in the subset selected for analysis, and those documents in turn are only a fragment of the total life of the schools. It is just the connections between words that we have selected and coded. It already represents a reality radically scaled down by our questions and our choices about how to embody those questions in methodologies, codes, and data units. In other words, dense as it is, this image is a massively simplified map of the reality under investigation. I know all of this in theory; the image helps me to imagine it.

I am not pushing for skepticism here—there are patterns in the noise, there are disciplined ways to find them, and many of them are useful when found. We learned things through our study that I think are valid and important. But I find it healthy to be visually reminded that the reality exponentially exceeds the academic maps with which we seek to represent it, even the careful, meticulous, empirical maps. That’s why the only image on my wall from the many trawls through NVivo looking for patterns is the one from the day I did it wrong.