Category Archives: research

My first academic papers on their way into the world…

I now have two academic papers on the point of being published.

One I present on Saturday 29 Feb at the HUCAPP conference: “Stuck In Limbo With Magical Solutions: The Testers’ Lived Experiences of Tools and Automation”.

The other is “Test Tools: an illusion of usability?” which I present at TAICPART in March.

Once they are published I will post the links to the papers.

Thank you to everyone who contributed to the workshops and surveys.

CHI2019 – pre and lite events – notes for Isabel

CHI 2019 was in Glasgow this year, and although I could not get to the main conference, I attended, and got a lot from, a “pre” event and a “lite” event run around the main conference.

The PreCHI day took place at the University of Dundee, and was a chance to hear a precis of papers delivered by academics from Scottish Universities at the main confererence. This was a good day, mainly fo rme to understand the breadth of research in HCI, and the types of project, that are happening. All the talks were interesting in that respect, ranging from a comparison of comics and infographics for helping to convey factual information, through virtual reality studies, to health care, network analysis, haptics and advocacy. The highlight talk for me was “developing Accessible Services: Understanding Current Knowledge and Areas for Future Support (Crabbe, Heron, Jones, Armstrong, Reid, Wilson) where among other things a useful matrix of accessiblity needs by (time?) against (type/are?) gave me pause for thought. The accesibility areas were: cognitive, communication, visual, physical, emotional, and the temporal axis was permanent, temporary, situational. So someone holding a heavy object is situationally, physically impaired from say picking up another object. When you start looking at accessiblity in that way it reinforces the idea that all of us need accessibility. It could feed into some of the ideas for the test tools work. Accessibility of the tools and of the information the tools generate. Other presentations I should follow up in terms of work on haptics, embodiment, and advocacy when thinking about my next steps. Notes are in Polish notebook.

CHIlite was an evening of highlights from the CHI conference and open to the public. Very good evening, with inspiring presentations that show how the HCI community is seeking to make the world a better place. Talks included “Bringing the Internet to the Brazilian Amazon” (Leal), “Seekign social justice through story telling” (Ahmed) and How can apps support sustainable behaviour” (Nkwo) – so heartening to see younerg people enaged in bringing technology to their communities in a postive way. to do good. Two talks by older practitioners on how we trust IT perhaps too much were by KOnstan “What makes a good recommendation?” and Sundar “Do we trust the machines too much?” which were thoughtful caveats on tech usage. And a few of the presentations spoke about the importance of the user/customer feeding back to the developer(s) about what they needed, what they liked and disliked. A call for the user to have a greater voice in what is delivered. Hofman on “putting a £D printer in the doctor’s office” was a good example, on patients requesting what they needed from a 3-d printed artificial limb, while Trllemans talked about control of the use of our smart environments, and Dereshev asked “What it is like living with a companion robot?” and Miyashita demostrated how technology can fool us with amazing visual effects that disguise reality.

Actions: to take: get the papers that are most relevant, read and add to literature review.

CHIIR 2019 – papers S1/S2 – follow up reminder notes for Isabel

The whole conference was exciting, friendly, so packed with information that by the end of Wednesday I was unable ingest any further ideas!!! It was just great. I got something from each session and there were a couple I wanted to follow up on for specific reasons – so here are some highlights of session 1 and session 2 … The audience I anticipate for this blog is 1 – namely myself when I want to remember what happened… so if you are not me reading this, apologies for the quick notes nature of it…. and there is probably both more detail than you need and yet… not enough. Follow the links to the papers if you are interested…

  • Session1, Paper 1: Learning about work tasks to inform intelligent assistant design (presented by Johanne Trippas and with a huge list of co-authors – see for the paper)
  • Here are some notes I made during the talk… and at the conference after a brief chat with Johanne:
    • wanting to empower people in their work
    • need to understand how people complete tasks
    • looked at cyber, social and physical aspects
    • asked people what tasks they were doing at work, and how much time on each task…
    • what do we mean by “context” when the context is the workplace?
    • need to understand HOW people complete tasks – thinking about collaboration, how much movement/physical activity is involved, how people are using tools (and which tools), how people classify their tasks, how the tasks change over time (of day, of week?)
    • find out what people want from intelligent assistants
      • task management
      • task tracking
      • (Isabel thought – Hmmm – so a mix of a manager and a PA??? As we talk more about self-managed teams, agile methods, etc… as we remove those human interactions and support that we get from a good manager, or a good PA… are we leaving people a little lost? feeling a little abandoned…?)
    • from the findings make recommendations for improving intelligent assistants at work.
    • Information workers do multiple tasks, what is a meaningful breakdown of those tasks? Hierarchy of activity/purpose of tasks – getting people to categorise their tasks is difficult – (thought from Isabel – do people understand their tasks in terms of the reason they are employed, why their organisation needs them, their purpose… or do they see their tasks as a series of small busy things, that don’t particularly relate to a wider purpose?
  • And here are some notes I made when reading the paper post conference:
    • a note is made about several ways to understand tasks – and refs to ways to do this ***follow up*** This could be a way to look at how people relate testing tasks to tools and to automation???
      • diary studies
      • naturaliistid field studies
      • lifelog analysis
      • statistical time use surveys
      • sudies of information needs, communications, information seeking – these could be relevant for methods???
      • survey (method used in this paper)
      • (Isabel note: cyber, physical and social activities – that is an interesting split; being at work is not just about completing tasks, there is also an element of the team or department as a community, and the physical part – that’s interesting – the effect on one’s body of the way the tasks are done…)
      • (isabel note: the poitn about the lack of penetration of intelligent assisitants for more complex tasks… I need to look again at Paul Gerrard’s talk about “testing with my invisible friend” and talk with him about what progress he has made… (see and Marianne’s sketchnote is a nice summary:
      • a note in section 2.3 about KUshmerick and Lau using FSM’s to formalise e-commerce transactions… Hmmm – could that be a tool / technique to document interactions in a test team between test designers and automators…??? ***think about this***
      • I can see looking at section 2.3 that I am looking at a subset of a subset of tasks… Uness I get interested in what distracts people from their main/key task??? leave that one alone for now…
      • The categories used in this paper’s task taxonomy could be a useful starting point for a taxonomy of testing tasks – it would be interesting to see if testers divided up their time in a similar way, and what sub-categories there might be under each category in the taxonomy. I know how I would break it down for how I work – but would it be the same for other testers? It could be quite different…
        • for example “IT” is one category and “project” is another… so if you are in IT, then (I guess) IT activities you do in order to provide yourself with an infrastructure to do your own testing are in “IT” and activities you do in order to test software being delivered in a project to a customer are “project” activities, so is managing the test automation an “IT” task – because it supports the testing… and is not in itself the purpose of the project… It would interesting to see how testers categorise it…
      • I’m interested in the point in section 4.4 about how intelligent assistants could help with longer durations tasks – the idea of an assistant that keeps a note of incomplete tasks to be resumed for example. (Isabel note to self: Have a look at agile/lean/kanban task duration recomendations and see if that fits with the task times being reported in this paper – what is the longest task people can work with as a “long task”? Is the “length of meeting” rule I was brought up onstill valid? (no more than 2 hours, pref no more than an hour, break after an hour, attention into flow state after 15-20 mins, How does that fit with the “15 min standup meeting advice for Scrum?” )
      • section 4.5 lists some tools people use (digital and physical such as post it notes, paper calendar – make sure I have physical tools included in what I ask about.
      • Concluding note – there is a lot for me to follow up in this paper, and ideas to use as a model for surveys and analysis.
  • Session 2 paper 3: Take me out: space and place in library interactions George Buchanan, Dana McKay, Stephann Makri. The paper is here:
    • This presentation and paper interested me partly as a library user, partly because of some new-to-me concepts the authors discussed, and partly as some input into UX/devices&Desires/imagine-our-customers sessions that I have coming up soon.
    • I liked the idea of place and space – the physical location and layout, versus the semantic meaning. For example “a place with lots of bookshelves is not necessarily a library” so we look at what people do as well as opposed to what they ask for… or talk about
      • Isabel note: in the same way – when does a test lab become a test lab? When is it an “information place” and what else could it be? Is this s useful idea to explore?
    • They talked about “wizard of oz” methods – I had not heard of that before – need to look into it…
    • They talked about the movement between physical and digital media when looking for information in a library. Isabel note: that too could be analogius?
    • “people reconstruct the technology you give them” – interesting quote – technologists provide methods, approaches, devices, etc but how people react to that may be unexpected, and the devices might be used for different purposes, in different ways. (That came up in the Museums keynote too – that people don’t interact with technology in the way curators expect)
    • from the paper:
      • “information interactions are strongly affected by the place where they occur”
      • “There is considerable ignorance of and resistance to the use of digital resources … some of which is related to the physical realities of the library”
      • section 2.2. seems to indicate that digital resources in a library are behaving like “closed stack” systems – where you need to know what you want and order it by name – rather than open-stack systems where you browse the shelves and serendipity leads you to new books, authors, topics…
      • paper quotes Warwick “danger of technocratic arrogance if we assume everythign can be modelled digitaly and thus improved” [ref is #21 in this paper – Warwick, C., 2017 “Beauty is truth: Multisensory inputand the challenge of designign aesthetically pleasing digital resources”]
      • note from Isabel – I was reminded of my experiences when Worcester public library merged with the Worcester Uni library – so that instead fo finding say “gardening books” all together, they were split across agriculture, horticulture, design… so that the shelves were a mix of amateur / easy to read and academic / industrial / professional – my personal experience was that I know found it harder to find what I needed… or I caught myself up in looking at additional material that was not really relevant. There is tension between relevance and serendipity…
      • note from Isabel: the lesson for the TX research is maybe about making the tester’s workspace (physical and digital) work as one – and also for other stakeholders for testing – think about how the information reaches them, how the medium for that information fits with each person’s working preference? WIthout being “gimmicky” (see section 9 of the paper)
      • quote: “designers should consider space and place carefully when designing mobile experiences”

CHIIR conference report – keynote highlights

The conference opened on Monday with a keynote from Ranjitha Kumar, which I found eye-opening and inspiring. Her team are working on “Data Driven Design: beyond AB testing” She pointed out that money spent on design does not always repay in results, and that A/B testing can be usefully supplemented with oher methods. In particular her team is working on “design mining” (rather than data mining) to find out what designs are being used elsewhere – she said there is a rich seam of designs available which give inspiration and a test / review point. She talked about the need to connect design with KPI’s, and to understand the success of designs in terms of their effect on KPI’s.

The second keynote, on Tuesday was also fascinating. Daniela Petrelli showed three case studies of making visitor experiences during museum visits multisensory, more engaging and more memorable. By using IoT technology, objects can be used to engage visitors in specific stories. I particularly loved the votary lamp that allows visitors to an exhibit on Hadrian’s wall chose three items – each a different god – and receive a personalised postcard with oracle-like messages. This a study at Chesters Fort , specifically around the Visitor eXperience of the Clayton collection. The three case studies indicated that visitors are more engaged and remember more, because they slow down and take longer to examine objects, when they use a physical object to access information – rather than a digital screen/phone. The IoT technology allows small objects – facsimiles that can be held in one’s hand – to be used to interact with video, audio, etc related to exhibits, and allow visitors to choose the viewpoint they experience in their journey through the museum.

I loved these two keynotes, interesting in so many ways – for me as a comsumer of information on the web and in museums, but also as a test consultant. Possible analogies – these gave me some thoughts about the experience of testers in their projects.

  • For example, if it true that people are more engaged and remember more when interacting with physical objects, could we use this idea to change how people examine and interact with information generated by testing? This is NOT age related… What does it tell us about how we generate, use and display information?
  • for example, if design mining is a useful supplement to A/B testing, how could it be used to supplement how we test designs – could it be a source for heuristics to use when testing interface designs?
  • for example, what we as digital experts provide and are proud of, is not always what the consumers of our work want or expect, For example, the questions that a search engine or chat bot responds to are not always the questions consumers want to ask. How can testers find out and understand what consumers actually want? That includes the consumers of the information from testing.
  • From those questions, I wonder about our testing dashboards – not for the first time in my decades in industry – and why we don’t talk with our stakeholders, in their language. I’ve been talking about this for years, presenting on it, teaching about it… I’ll continue with that. Quote from K1 about fashion websites – customers ask for “hot pink” websites talk about “Fuchsia” or “magenta”
  • K2 provided a mini lifecycle for co-design and co-development where a technical person, a designer and a curator get together and split apart repeatedly to generate and test the ideas and design for artefacts. Is there an analogy to the developer, UXer and product Owner, and if so, where is the testing, and is there a need for a specific tester role?

CHIIR Conference Glasgow March 2019: Tutorial report

This was my first time at CHIIR, and it was a really enjoyable experience; lovely people, great community spirit and the sessions were full of information and discussion. I started with the Tutorial on Sunday 10th March “Coding qualitative data: you asked them, now what to do with what they said” led by Dr Rebekah Willson (University of Strathclyde). There is a pleasure in being taught by a good teacher who enjoys their subject, even if the subject is not one of direct interest. As it happens, the subject for this tutorial was right on topic for me, right now, so a double pleasure. A really good session, which Dr Willson described as a “whirlwind tour”, but in fact gave space for us to work in pairs on an exercise, discuss and feedback. I’ve come away from that tutorial feeling more confident that I can code up the qualitative data I have collected so far in my studies.

We covered a step by step approach to coding qualitative data, bearing in mind the “paradigm shift in thinking” as one moves from quantitative to qualitative methods: we’re dealing with the human and that is messy, challenging, based on experiences and beliefs, and allows a broader, holistic understanding, albeit one that is constructionist, with the researcher involved in the research, giving multiple meanings, multiple interpretations. We are there, we are part of the process, so we have to think about the role we have and what we are doing. The result of qualitative data collection is richer data that is more difficult to interpret. We are asking “Why did they do/say that?” There are several approaches to coding, and so it is important to choose one and stick with it. There are challenges of qualitative research being in itself a learning process – it is messy, it is fun, and doing it shows you how to do it. It is normal to be confused and overwhelmed. That’s a helpful thought. Dr Willson chose to show us one route through, with a series of iterating steps, providing a robust and rigorous approach to analysing qualitative data. She reminded us that a negative/opposing result can often be the most useful and interesting thing to explore – why is that case different? It is about following where the data leads, and moving from the concrete to the abstract. Looking for similarities, grouping and classifying. She talked about the process feeling uncomfortable, which I find to be true – like wandering in a fog and occasionally glimpsing the light!

When we gather data for a qualitative study, we usually have a vast volume of material – for a example, transcribing an interview can give you 1000’s of words of material. Furthermore, when you ask open questions, the answers are unpredictable and often richer than you’d anticipated. This fits with what’s happening for me. Instead of asking “what is your job title?” and “what is your education?” in a recent survey, because of a limit on the number of questions – I combined the two into “Tell me a bit about yourself” and received back long essays that told me such a variety of things, and sparked so many questions that I had not thought to ask, around ideas that I now see are interesting to explore… Dr Willson said we must pay attention to anything that is potentially interesting, code it up and then refine our ideas – grouping, splitting up, asking new questions of the data, all the time moving from a broad view of the data to a deeper focus. Also, be rigorous and trustworthy – sharing how we code the data, what steps we took, taking an iterative approach, triangulating across data sources, including negative examples, making our codebook available, making our inclusion/exclusions available. The researcher must be trustworthy, and if more than one persons is coding – this is a good thing to check for consistency of interpretation, provided that there is inter-coder reliability; we need clear codes, clear reasons for using the codes, clear inclusion and exclusion criteria. This means we’ve moved from the initial coding exercise to a focused coding stage, using a code book. The coders code separately and then compare results.

Dr Willson described several methodologies for qualitative analysis, and explained that the choice of methology is affected by the research questions. The methodology she showed us in detail, and which we practised in the exercises is Thematic Analysis. She talked about two levels of engaging with the data: the SEMANTIC level where we look for and code things that are expliciit in the data, and the LATENT level where we look ideas and assumptions implicit in the data. We need to decide ahead of time which we do. In thinking about these levels, we start to realsie that what people say and what they do can be different – so field notes about behaviour become part of the data. As well as text, we might collect and analyse video, audio, images and so on. The steps in thematic analysis are:

  1. familiarise – read the text several times and take notes. Do it line by line!
  2. generate initial codes, get to know the data – again line by line.
  3. start to look for patterns in the codes, perhaps ways they group
  4. make themes of one or more codes – overarching ideas that cut across the codes.
  5. review the themes against the data… do they make sense?
  6. and do it again…

Defining and naming the themes provides the analytic power – think about what the thme can contribute. Themes can have subthemes, so there can be a hierarchy of themes, subthemes, categories, and codes. The code book has the full description of these, and each code and theme has a single word or short phrase descriptive name. Relate the codes and themes back to the research questions. As this process is worked through, the research questions might change – because we realise the data is pointing us in a new direction. We need durign research to constantly revist our questions, out data, our themes and codes t ensure we are following the data, asking the right questions, revisiting, enlarging and clarifying, all the time. Whether we start from a deductive approach (where we predefine the codes to support our idea and the questions we want to explore) or an inductive approach (where we explore the data, come up with codes and build to themes and questions) or move between the two – always we need to keep revisiting the data. Follow up, change the questions, revisit ideas, identify what is different, look for variations…

Later in the week, the conference dinner was at the Science Museum, and while there I noticed a mural/display that said “We are all scientists; we all observe, find reasons, look for relationships, categorise and make models” Unfortunately my photo of it is too blurry to share… but it summarised the tutorial and the week for me. Thank you, Dr Willson for a brilliant tutorial!