Research results so far, and a request for your help

I’m now half way through my research on testers’ experiences with tools and automation, with an end date of October 2025. Thank you everyone who contributed their stories and comments to the data so far. I have published three papers (below)

My findings so far provide an insight into the usability and human issues that impede success with tools and automation, and also show the large range of people’s backgrounds before they come into testing. These findings, along with the complexity of testers’ multi-tasking roles means that designing tools for testers, automation for testers, and approaches for testers to use, is not easy.

I’m embarking on the next stages of my research. I’m asking some deceptively simple questions: Who are we testers? Where do we come from? What are we doing? And how are we doing it? The answers are not as straightforward as we might think. But they should feed into development of personas and other models to better tool design, with a better UX for the testers. They also feed into a better understanding of each other, of the diversity in our community of testers, and help us engage with each other – because not everyone doing testing is a tester, and not everyone working in IT is an engineer.

Please contribute to this research – if you have software testing as part of your role: please complete this survey.

The academic papers have more details and argumentation. If you would like copies of these papers, links are below. Note that the published copies are generally behind pay walls from the publishers. Pre-publication drafts can be made freely available, please contact me at isabel.evans.17@um.edu.mt

[a] I. Evans, C. Porter, M. Micallef, and J. Harty, “Stuck in limbo with magical solutions: The testers’ lived experiences of tools and automation,” in Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. SCITEPRESS-Science and Technology Publications, 2020, pp. 195–202. Link: Stuck in Limbo with Magical Solutions

[b] I. Evans, C. Porter, and M. Micallef, “Scared, frustrated and quietly proud: the testers’ experiences of tools and automation,” in 2021 European Conference on Cognitive Ergonomics. ECCE, in press, accepted by 2021 conference. Link: Scared, Frustrated and Quietly Proud

[c] I. Evans, C. Porter, M. Micallef, and J. Harty, “Test tools: an illusion of usability?” in 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2020, pp. 392–397. Link: Test Tools: An Illusion of Usability?

My first academic papers on their way into the world…

I now have two academic papers on the point of being published.

One I present on Saturday 29 Feb at the HUCAPP conference: “Stuck In Limbo With Magical Solutions: The Testers’ Lived Experiences of Tools and Automation”.

The other is “Test Tools: an illusion of usability?” which I present at TAICPART in March.

Once they are published I will post the links to the papers.

Thank you to everyone who contributed to the workshops and surveys.

CHI2019 – pre and lite events – notes for Isabel

CHI 2019 was in Glasgow this year, and although I could not get to the main conference, I attended, and got a lot from, a “pre” event and a “lite” event run around the main conference.

The PreCHI day took place at the University of Dundee, and was a chance to hear a precis of papers delivered by academics from Scottish Universities at the main confererence. This was a good day, mainly fo rme to understand the breadth of research in HCI, and the types of project, that are happening. All the talks were interesting in that respect, ranging from a comparison of comics and infographics for helping to convey factual information, through virtual reality studies, to health care, network analysis, haptics and advocacy. The highlight talk for me was “developing Accessible Services: Understanding Current Knowledge and Areas for Future Support (Crabbe, Heron, Jones, Armstrong, Reid, Wilson) where among other things a useful matrix of accessiblity needs by (time?) against (type/are?) gave me pause for thought. The accesibility areas were: cognitive, communication, visual, physical, emotional, and the temporal axis was permanent, temporary, situational. So someone holding a heavy object is situationally, physically impaired from say picking up another object. When you start looking at accessiblity in that way it reinforces the idea that all of us need accessibility. It could feed into some of the ideas for the test tools work. Accessibility of the tools and of the information the tools generate. Other presentations I should follow up in terms of work on haptics, embodiment, and advocacy when thinking about my next steps. Notes are in Polish notebook.

CHIlite was an evening of highlights from the CHI conference and open to the public. Very good evening, with inspiring presentations that show how the HCI community is seeking to make the world a better place. Talks included “Bringing the Internet to the Brazilian Amazon” (Leal), “Seekign social justice through story telling” (Ahmed) and How can apps support sustainable behaviour” (Nkwo) – so heartening to see younerg people enaged in bringing technology to their communities in a postive way. to do good. Two talks by older practitioners on how we trust IT perhaps too much were by KOnstan “What makes a good recommendation?” and Sundar “Do we trust the machines too much?” which were thoughtful caveats on tech usage. And a few of the presentations spoke about the importance of the user/customer feeding back to the developer(s) about what they needed, what they liked and disliked. A call for the user to have a greater voice in what is delivered. Hofman on “putting a £D printer in the doctor’s office” was a good example, on patients requesting what they needed from a 3-d printed artificial limb, while Trllemans talked about control of the use of our smart environments, and Dereshev asked “What it is like living with a companion robot?” and Miyashita demostrated how technology can fool us with amazing visual effects that disguise reality.

Actions: to take: get the papers that are most relevant, read and add to literature review.

CHIIR 2019 – S3 follow up reminder notes for Isabel

More notes from CHIIR 2019 –
so here are some highlights of session 3 … The audience I anticipate for this blog is 1 – namely myself when I want to remember what happened… so if you are not me reading this, apologies for the quick notes nature of it…. and there is probably both more detail than you need and yet… not enough. Follow the links to the papers if you are interested…

Session 3 paper 1: Knowledge context in search systems: towards information-literate actions By Catherine L Smith and Soo Young Rieh, see https://dl.acm.org/citation.cfm?id=3298940 for the paper. This really interested me – a perspectives paper about how we learn, and whether we learn, when using search engines. Main points:

• “the knowledge content in SERPs has great potential for facilitating human learning, critical thinking and creativity by expanding searchers’ information-literate activities such as comparing, evaluating, and differentiating between information sources”
• “we discuss design goals for search systems that support metacognitive skills required for long-term learning, creativity and critical thinking”
• I made a note during the presentation – we don’t remember information stored on teh computer but we have a feeling that we do know it, and we do remember where we stored it (?) – it makes it harder to learn somethign new. Quoted Sparrow, Liv & Wegner 2011 – we remember where but we don’t remember what e.g. phone numbers. It strikes me that this is perhaps OK for phone numbers – we’ll find them on the phone or in an address phone (virtual or physical) – but for information generally on the web, it must be harder – the “where” is much more diffuse. Comment in the presentation that the feeling of knowing increases with searching on the web even if the search returns irrelevant information. Comment in the presentation that the accuracy of our judgement about whether we know something is reduced by using websearch.
• the paper and presentation calls for the support of information literate searching. The design of search engines to support greater information literacy by conextualising search results, and actually slowing people down so they are supported in long term learning.
• I compare this paper to the paper “Chooosing the right test automation tool: a grey literature review of practitioner sources” (2017) Raulamo-Jurvanen, Mantyla, Garousi
  • in the grey literature review, one of the findings was that when people look for information on the web about test tools, they pick off the most popular, most mentioned tools and resources. Therefore if those tools are popular / fashionable but not necessarily right for the searcher’s context, they may end up with the wrong tool for their purpose.
  • quotes from that grey literature review: once people had chosen a tool based on their web-search for information “trial use would often lead to wrong decisions” Question: the popular tools – are they popular because they are good, or popular because they are popular and therefore user groups, support, etc? Also note their point at the end of the paper on cognitive overload – so people choose what is obvious. “tendency for cognitive overload is likely to increase the prevalence of shortcut decision making proportionately” “social proof as a weapon of influence is claimed to be most influential under 2 conditions: uncertainty and similarity” the authors referring to Cialdini.
  • Taking the two papers together, does this indicate that testers (and other people invlved in test tool selection) need support for better decision making – better information literacy when looking for information about tools and automation?
    • do I know it?
    • can I find it?
    • having found it do I know how to judge it and whether to trust it?
• The knowledge context for a tester is testing as a discipline, within IT the industry, to serve a particular domain. A tester requires knowledge and infomation literacy across all those knowledge contexts. Testers need to be critical thinkers – the points made in Smith & Rieh about the use of ILA “may be seen as an indicator that the system is not sufficiently optimised” – does that indicate that search engines as a source for information about tools reduces critical thinking? Key quote “In order to learn, understand, and gain confidence in their knowledge, information literate people ask and answer questions about the information they encounter” Critical thinking and making indeppendent judgements are key characteristics of good testers.
• Also explore the points on transactive memory – where teams / pairs “split responsibility for remembering parts of the information required to complete a task” – how does that sit with the dev/test relationship? different track to purpue – not for research, just interesting
• Summary findings are that when people believe information will be stored on a computer they are less likely to remember it, and more likely to remember where the informaiton is. … the use of web search leads people to overestimate how much they know.”
  • in testing we use the concept of the oracle for test results
  • which I have always found funny given that oracles (eg Delphi) tended to be ambiguous and easy to misinterpret
  • information literacy includes the use of multiple oracles, and comparing them – and indeed not treating them as oracles, but as information sources to be critically assessed and questioned.
  • The ways we understand whether to trust information includes the “bibliographic knowledge-context” (publisher, author, form, reading level scores) and the “inferential knowledge-context” (other works, comparisons, citations, history, versions, valence / biases) – can this be mapped to how we understand tools?
  • for testers, there is a tension between a need to get information quickly and the need to critically assess that information – especially when we are in a hurry. What can we trust?
    • testers use web sources to learn – need to critically assess those sources
    • testers provide information obtained from tools – need to critically assess that information
• this reminds me of the point in the conversation with Dot Graham on the “illusion of usability”

CHIIR 2019 – papers S1/S2 – follow up reminder notes for Isabel

The whole conference was exciting, friendly, so packed with information that by the end of Wednesday I was unable ingest any further ideas!!! It was just great. I got something from each session and there were a couple I wanted to follow up on for specific reasons – so here are some highlights of session 1 and session 2 … The audience I anticipate for this blog is 1 – namely myself when I want to remember what happened… so if you are not me reading this, apologies for the quick notes nature of it…. and there is probably both more detail than you need and yet… not enough. Follow the links to the papers if you are interested…

• Session1, Paper 1: Learning about work tasks to inform intelligent assistant design (presented by Johanne Trippas and with a huge list of co-authors – see https://dl.acm.org/citation.cfm?id=3298934 for the paper)
• Here are some notes I made during the talk… and at the conference after a brief chat with Johanne:
  • wanting to empower people in their work
  • need to understand how people complete tasks
  • looked at cyber, social and physical aspects
  • asked people what tasks they were doing at work, and how much time on each task…
  • what do we mean by “context” when the context is the workplace?
  • need to understand HOW people complete tasks – thinking about collaboration, how much movement/physical activity is involved, how people are using tools (and which tools), how people classify their tasks, how the tasks change over time (of day, of week?)
  • find out what people want from intelligent assistants
    • task management
    • task tracking
    • (Isabel thought – Hmmm – so a mix of a manager and a PA??? As we talk more about self-managed teams, agile methods, etc… as we remove those human interactions and support that we get from a good manager, or a good PA… are we leaving people a little lost? feeling a little abandoned…?)
  • from the findings make recommendations for improving intelligent assistants at work.
  • Information workers do multiple tasks, what is a meaningful breakdown of those tasks? Hierarchy of activity/purpose of tasks – getting people to categorise their tasks is difficult – (thought from Isabel – do people understand their tasks in terms of the reason they are employed, why their organisation needs them, their purpose… or do they see their tasks as a series of small busy things, that don’t particularly relate to a wider purpose?
• And here are some notes I made when reading the paper post conference:
  • a note is made about several ways to understand tasks – and refs to ways to do this ***follow up*** This could be a way to look at how people relate testing tasks to tools and to automation???
    • diary studies
    • naturaliistid field studies
    • lifelog analysis
    • statistical time use surveys
    • sudies of information needs, communications, information seeking – these could be relevant for methods???
    • survey (method used in this paper)
    • (Isabel note: cyber, physical and social activities – that is an interesting split; being at work is not just about completing tasks, there is also an element of the team or department as a community, and the physical part – that’s interesting – the effect on one’s body of the way the tasks are done…)
    • (isabel note: the poitn about the lack of penetration of intelligent assisitants for more complex tasks… I need to look again at Paul Gerrard’s talk about “testing with my invisible friend” and talk with him about what progress he has made… (see https://conference.eurostarsoftwaretesting.com/event/2017/testing-with-an-invisible-friend/ and Marianne’s sketchnote is a nice summary: https://twitter.com/marianneduijst/status/928189626929614848)
    • a note in section 2.3 about KUshmerick and Lau using FSM’s to formalise e-commerce transactions… Hmmm – could that be a tool / technique to document interactions in a test team between test designers and automators…??? ***think about this***
    • I can see looking at section 2.3 that I am looking at a subset of a subset of tasks… Uness I get interested in what distracts people from their main/key task??? leave that one alone for now…
    • The categories used in this paper’s task taxonomy could be a useful starting point for a taxonomy of testing tasks – it would be interesting to see if testers divided up their time in a similar way, and what sub-categories there might be under each category in the taxonomy. I know how I would break it down for how I work – but would it be the same for other testers? It could be quite different…
      • for example “IT” is one category and “project” is another… so if you are in IT, then (I guess) IT activities you do in order to provide yourself with an infrastructure to do your own testing are in “IT” and activities you do in order to test software being delivered in a project to a customer are “project” activities, so is managing the test automation an “IT” task – because it supports the testing… and is not in itself the purpose of the project… It would interesting to see how testers categorise it…
    • I’m interested in the point in section 4.4 about how intelligent assistants could help with longer durations tasks – the idea of an assistant that keeps a note of incomplete tasks to be resumed for example. (Isabel note to self: Have a look at agile/lean/kanban task duration recomendations and see if that fits with the task times being reported in this paper – what is the longest task people can work with as a “long task”? Is the “length of meeting” rule I was brought up onstill valid? (no more than 2 hours, pref no more than an hour, break after an hour, attention into flow state after 15-20 mins, How does that fit with the “15 min standup meeting advice for Scrum?” )
    • section 4.5 lists some tools people use (digital and physical such as post it notes, paper calendar – make sure I have physical tools included in what I ask about.
    • Concluding note – there is a lot for me to follow up in this paper, and ideas to use as a model for surveys and analysis.

• Session 2 paper 3: Take me out: space and place in library interactions George Buchanan, Dana McKay, Stephann Makri. The paper is here: https://dl.acm.org/citation.cfm?id=3298935
  • This presentation and paper interested me partly as a library user, partly because of some new-to-me concepts the authors discussed, and partly as some input into UX/devices&Desires/imagine-our-customers sessions that I have coming up soon.
  • I liked the idea of place and space – the physical location and layout, versus the semantic meaning. For example “a place with lots of bookshelves is not necessarily a library” so we look at what people do as well as opposed to what they ask for… or talk about
    • Isabel note: in the same way – when does a test lab become a test lab? When is it an “information place” and what else could it be? Is this s useful idea to explore?
  • They talked about “wizard of oz” methods – I had not heard of that before – need to look into it…
  • They talked about the movement between physical and digital media when looking for information in a library. Isabel note: that too could be analogius?
  • “people reconstruct the technology you give them” – interesting quote – technologists provide methods, approaches, devices, etc but how people react to that may be unexpected, and the devices might be used for different purposes, in different ways. (That came up in the Museums keynote too – that people don’t interact with technology in the way curators expect)
  • from the paper:
    • “information interactions are strongly affected by the place where they occur”
    • “There is considerable ignorance of and resistance to the use of digital resources … some of which is related to the physical realities of the library”
    • section 2.2. seems to indicate that digital resources in a library are behaving like “closed stack” systems – where you need to know what you want and order it by name – rather than open-stack systems where you browse the shelves and serendipity leads you to new books, authors, topics…
    • paper quotes Warwick “danger of technocratic arrogance if we assume everythign can be modelled digitaly and thus improved” [ref is #21 in this paper – Warwick, C., 2017 “Beauty is truth: Multisensory inputand the challenge of designign aesthetically pleasing digital resources”]
    • note from Isabel – I was reminded of my experiences when Worcester public library merged with the Worcester Uni library – so that instead fo finding say “gardening books” all together, they were split across agriculture, horticulture, design… so that the shelves were a mix of amateur / easy to read and academic / industrial / professional – my personal experience was that I know found it harder to find what I needed… or I caught myself up in looking at additional material that was not really relevant. There is tension between relevance and serendipity…
    • note from Isabel: the lesson for the TX research is maybe about making the tester’s workspace (physical and digital) work as one – and also for other stakeholders for testing – think about how the information reaches them, how the medium for that information fits with each person’s working preference? WIthout being “gimmicky” (see section 9 of the paper)
    • quote: “designers should consider space and place carefully when designing mobile experiences”

CHIIR conference report – keynote highlights

The conference opened on Monday with a keynote from Ranjitha Kumar, which I found eye-opening and inspiring. Her team are working on “Data Driven Design: beyond AB testing” She pointed out that money spent on design does not always repay in results, and that A/B testing can be usefully supplemented with oher methods. In particular her team is working on “design mining” (rather than data mining) to find out what designs are being used elsewhere – she said there is a rich seam of designs available which give inspiration and a test / review point. She talked about the need to connect design with KPI’s, and to understand the success of designs in terms of their effect on KPI’s.

The second keynote, on Tuesday was also fascinating. Daniela Petrelli showed three case studies of making visitor experiences during museum visits multisensory, more engaging and more memorable. By using IoT technology, objects can be used to engage visitors in specific stories. I particularly loved the votary lamp that allows visitors to an exhibit on Hadrian’s wall chose three items – each a different god – and receive a personalised postcard with oracle-like messages. This a study at Chesters Fort , specifically around the Visitor eXperience of the Clayton collection. The three case studies indicated that visitors are more engaged and remember more, because they slow down and take longer to examine objects, when they use a physical object to access information – rather than a digital screen/phone. The IoT technology allows small objects – facsimiles that can be held in one’s hand – to be used to interact with video, audio, etc related to exhibits, and allow visitors to choose the viewpoint they experience in their journey through the museum.

I loved these two keynotes, interesting in so many ways – for me as a comsumer of information on the web and in museums, but also as a test consultant. Possible analogies – these gave me some thoughts about the experience of testers in their projects.

• For example, if it true that people are more engaged and remember more when interacting with physical objects, could we use this idea to change how people examine and interact with information generated by testing? This is NOT age related… What does it tell us about how we generate, use and display information?
• for example, if design mining is a useful supplement to A/B testing, how could it be used to supplement how we test designs – could it be a source for heuristics to use when testing interface designs?
• for example, what we as digital experts provide and are proud of, is not always what the consumers of our work want or expect, For example, the questions that a search engine or chat bot responds to are not always the questions consumers want to ask. How can testers find out and understand what consumers actually want? That includes the consumers of the information from testing.
• From those questions, I wonder about our testing dashboards – not for the first time in my decades in industry – and why we don’t talk with our stakeholders, in their language. I’ve been talking about this for years, presenting on it, teaching about it… I’ll continue with that. Quote from K1 about fashion websites – customers ask for “hot pink” websites talk about “Fuchsia” or “magenta”
• K2 provided a mini lifecycle for co-design and co-development where a technical person, a designer and a curator get together and split apart repeatedly to generate and test the ideas and design for artefacts. Is there an analogy to the developer, UXer and product Owner, and if so, where is the testing, and is there a need for a specific tester role?

Conferences 2018 – a reflection

It’s been a busy year for conferences and teaching – so here is a quick summary…

UKSTAR 2018

This was in London, UK, in March.

I enjoyed day 1, with excellent keynotes from Christina Ohanian, and Gustav Kuhn. Christina’s messages on building teams were full of insights. As well as Gustav’s keynote, I’d also enjoyed Gustav’s workshop with Alan Richardson. The application of magic to psychology and then to how we observe (or don’t observe) the phenomena about us was very revealing, and certainly applicable to testers. Do we see the things other people don’t see? Can we be tricked? Powerful stuff.

Day 2 of UKSTAR 2018 was a very – extreme – experience for me. I was booked to present the opening keynote and then a workshop, and just before I was due to go on stage I had a completely unexpected phone call to say my mother had died. I made the decision that I wanted to go ahead – she would have wanted and expected that – discussed it with the programme committee and the UKSTAR team, and with their full and immense support went ahead to give my keynote – on Leadership, Followership and Fellowship – which went well, on a surge of adrenlin that also carried me through my workshop where we shared information around the UX of testing tools for testers – contributing data to my research and allowing delegates to share experiences and learn some UX techniques to apply to their own test tool acquisitions.

I used the data I collected during the workshop as one of the inputs to the Webinar I did for EuroSTAR, which also allowed me to open an online survey, and collect more data. The webinar and survey are still available here: https://huddle.eurostarsoftwaretesting.com/webinar-questionnaire-no-shelfware-lets-drive/

Thank you so much EuroSTAR and UKSTAR teams for your support during UKSTAR and beyond!

Romanian Code Camp

This was in Iasi in March. I was so pleased (given my personal circumstances) to be there with my friend and colleague Sue Atkins. Again the conference organisers were so supportive! As well as a masterclass on quality in use and UX, and a masterclass on Leadership, Followership and Fellowship, I presented on Human Factors for Test Automation, Sue and I presented a joint session on State Transition Testing, and also we both gave lightning talks. I was delighted to meet Vijay Kiran, who gave an inspirational lightning key on the importance of ethics in development – that excellent software is not just well engineered, not just exhibiting excellent UX, but also is ethically sound – doing good. I’ve been quoting him all year since! It was enjoyable to be at a conference with a range of tracks as well as a testing track: architecture, design, frontend, web, IoT, engineering, leadership, agile, entrepreneurship, and – my favourote track title – “funa dn fearless”. This conference was also beautifully hosted, ending with a cocktail party for speakers, hand made, natural ingredients – and the most delicious non-alcoholic one featuring apples and peanuts! I wish I could remember the name of the people who produced them!

STAREAST

STAREAST is always good fun, and this year was no exception…  I presented two tutorials: “Influence Diagrams – a new way to understand testing” and “Transforming Testing: building your road map”. These were both half day workshops, although closely linked together. I also presented a track on “Devices and Desires” about our attitude to technology and how that matches against the needs and desires of people outside IT. An interesting couch discussion with a group of delegates about leadership completed my contributions to the conference. Photo:  https://stareast.techwell.com/conference-photo/se18-couch-session-isabel-evans

While there, I enjoyed the Women Who Test day; each time I attend I learn something new about myself, and enjoy the perspectives of the other attendees (men and women are welcome, by the way).

HTB Workshop and HUSTEF preparation

During 2018, I was honoured to be the HUSTEF Programme Chair. More on the latter later! In June, while in Budapest for the programme planning meeting, I also presented a workshop for the HTB on Quality in Use, and at the Tezst and Tea meet up presented “No more shelfware!”

The programme planning went well, it was a real joy to work with the programme committee and the HUSTEF organising team!

BCS SIGiST

This meeting took place in London in June. Stuart Reid and the committee had organised an “all keynotes” day, which was really good fun. I met up with old friends, and enjoyed their talks, as well as presenting “Devices and Desires” to an appreciative audience.

ODIN

Off to Oslo in September for a delightful conference, I gave tutorials on UX, and on Human Factors for test automatio, as well as a keynote on Leadership. At ODIN I also met up with Lorraine and Siobhan of the EuroSTAR team, who took me aside for a coffee, a catch up… and an interesting request…

TESTJAM

Straight from Oslo to Utrecht, where I presented at Capgemini’s Testjam, on Devices and Desires, and met up with old friends (Nathalie and Kimberley and Rik) from Capgemini… A lovely evening, ending with a dash to the airport as storms disrupted air travel… 

STARWEST and STARCANADA

Before STARWEST, the various personal events of the year meant that I could get to conferences and deliver, but was unable to take in any information! By STARWEST and STARCANADA, I felt more settled, and enjoyed listening to the keynotes and tracks. STARWEST was busy busy busy and great fun!

Across the two conferences, I taught 5 tutorials (test design, requirements testing, UX, influence diagrams, automation: a human-centred approach) which all went well plus a lightning key, 2 track sessions, 2 couch sessions, and a talk on failure at Women Who Test.

I’d also been on the programme committee with Rob Sabourin (the chair) and Julie Gardiner. That had been a really interesting experience, and I was delighted to have the chance to contribute, and to help choose the speakers. My feel is that these were two great programmes – but that of course is author bias!

Great keynotes! Jennifer Bonine and Janna Loeffler on story telling – great production values, wonderful illustrations by a Disney illustrator, and a great message well delivered. Jon Bach’s courageous use of a live survey with the audience via an app was really enjoyable, as well as his thoughts on how one’s behaviour changes with one’s role. Dona Sarkar on “be the lord of your own rings” was a fireball of energy. Max Saperstone showed a brilliant use of mutation syntax testing. Fiona Charles discussed leadership and how it differs from Management. Alexandre Baudin showed us how to test flight simulators, and Sophie Benjamin eloquently told how to transform testing.

Among the tracks I enjoyed Jane Jeffers from Riot Games on asking Why, and Julene Johnson from Lucid Software on Anxiety. ALso Stefan Marceau and Keith Turpin on User Stories, Fiona Charles on Gaining Consciousness, 

Women Who Test maintained its celebratory nature, in particular from a rich day, I’ll pick out getting to see the first printing of Tania Katan’s book on creative trespassing…

HUSTEF

My first big conference as programme chair! It’s been hard work, but what an experience! I am so pleased to have done this… In fact, I think it deserves a separate post…

EuroSTAR

The final conference of the year… and as always like coming home. Old friends abound, so many greetings, embraces, and conversations! Friends from all over the world! 

I think this one deserves its own post too – lots to say, and this post is already…. too long!

Book review: A practical guide to Testing in DevOps (Katrina Clokie) – get it, read it!

Headline: get this book! read this book! Link: https://leanpub.com/testingindevops

Katrina Clokie’s book “A practical guide to Testing in DevOps” is a “must read”, and not just if you are a tester encountering DevOps. As Katrina says in the preface: “This book is for testers who want to understand DevOps and what it means for their role. It’s also for people in other roles who want to know more about how testing fits into a DevOps model.” It is engaging, informative, full of useful examples, case studies and evidence, and gives pragmatic, thoughtful guidance that encourages us – the readers – to use the information provided to craft our own approach based on our own circumstances.

I’m particularly pleased with three of Katrina’s themes:

• this is about people as well as technology – how people engage, communicate, cooperate, solve problems, achieve goals – together;
• separating the activity of testing from the role “tester” – the activity is needed, the role may change;
• the purpose of testing is to contributing to our shared understanding of, and delivery of, what our stakeholders, customers, users, colleagues require from IT: a subtle and changing mix of speed, predictability, quality and a great user experience.

I kept finding points in the book that are applicable for any project, any organisation. There are useful ideas in this book if you are working in a DevOps environment, of course, but I believe you will also find it a source of useful ideas and techniques what ever your project methodology. Katrina provides chances for all of us to review our current position, and just shake it up a little, to see if we can improve. For each of us, for each organisation, team, what we need to do is different. Katrina is rightly emphatic that there is not one right answer, instead she provides aides to thinking, communicating, problem solving.

I really like the way Katrina references and quotes so many colleagues across the industry – this really helps build the feeling of a community working together, sharing ideas, not always agreeing, but debating openly and constructively.

She encourages us to try ideas and propose changes on a small, experimental scale if we or our colleagues are nervous “…spend an hour to give something new a try…”

After an introduction to the concept of DevOps, the book is divided into six main sections:

• Testing in a DevOps Culture;
• Testing in Development;
• Testing in Production;
• Testing in DevOps Environments;
• Industry Examples;
• Test Strategy in DevOps.

I was pulled into the book straight away. Highlights for me in the first section are:

• visualising the test strategy – what we think we are doing, what we are actually doing… and what we should be doing. Newsflash: get the people without “tester” in their job title to do the discussions, and as the “tester” listen, facilitate… and allow the others to challenge your assumptions!
• blazing a trail – really good ideas, hints and tips for building meaningful multi-way conversations where we listen as well as speaking… Forging the pathways between groups, teams and individuals, and reinforcing those pathways by rich rather than fast communication.

I have made so many notes and comments, just on that first section – it sparked many useful ideas, reinforced some thoughts, and provided me with new tricks, tips and tools that I shall try out.

In the second, third and fourth sections, Katrina provides examples and evidence for how testing – the activity – takes place successfully outside Testing – the team or Testing – the life cycle phase. This is such a refreshing read; my own experience since I started in IT is that good, productive, thoughtful testing can be done by people other than testers, and during activities not called testing. Developers, Ops folk, Support teams, Sales, Marketing – many people in an organisation can do good testing, often with a focus that the “Tester” may miss.  She discusses the advantages and risks of testing in development, production and DevOps environments, showing options for how risks can be mitigated, whether by use of tools, experimentation, exploration, group activities, process changes, shifting left and/or shifting right.. I’m not going to share too much here, just encourage you to buy and read the book!

The fifth section, the industry examples, is useful because it provides a wide range of approaches. Not all of us will find the same approach appropriate; something essential to success in one organisation may be damaging in another. There is a good mix of methods, tools, approaches, and types of testing covered, in this section and in the others, so you should find something to use as a starting point for making your own model.

By the time I reached the last section of the book, I had a mass of ideas, notes, and things to follow up, and I would be surprised if you did not feel the same way.

In the final section, Katrina brings everything together to help you define your own strategy for testing DevOps. Your’s will be different. It will be the strategy that is right for your team, your organisation. In this section she helps us to think about our organisation’s appetite for risk: I also have found this to be extremely useful point to consider and discuss. Are my stakeholders risk averse, or risk takers? Am I aligned with them? At this point, should I challenge or support their stance? Katrina provides useful examples of the type of question to ask in order to understand appetite for risk, identify which risks are of importance, and from that decide what activities will mitigate the risk. Those activities might include testing – the activity – but there will very likely be other activities that will mitigate the risks just as well.  From this we can draw up a strategy for testing, acknowledging that it is part of but not all of the activity required to provide our stakeholders with the quality, speed, predictability, risk mitigation and so on that is right for their circumstances.

Whether you are working waterfall, V-model, W-model, Iterative, agile or DevOps – you’ll find some useful ideas in this book.

A good ending to an excellent book, which I have no hesitation in recommending. Thank you Katrina!

 

Nokia Test Dive 2017 – audience questions

I had a great time at the Nokia Test Dive in Krakow – a lovely conference, responsive delegates, and hosted beautifully by Nokia.

After both my sessions, there were audience questions on Sligo that we did not get time to discuss, so I have taken those questions, grouped them, and made some responses. They were great questions!

Some quick answers to start…

What is your UX of Kraków / Poland

• Lovely question!
• Great UX, I loved the city, so beautiful. I’d like to visit again. The historic centre is really interesting, the different architecture, especially the art nouveau.
• I lost my hat – but that was my fault!

Is there a universal solution to prevent “shelfwareness”?

• Such a wonderful question! If I knew the answer I probably would not need to be researching! Maybe there is… I suspect if the answer was easy someone would have found it already. I’m going to borrow this question and use it when I interview experts and tools suppliers… Thank you!

The billions of annual loss due to wrong automation – is it for one company? Did they do something with that?

• This is from a report about the cost of poor test infrastructure in the USA – across the country. The report reference is in the slides, in the reference pages at the back of the slide set, but here it is:

Tassey, G. (2003) The economic impacts of inadequate infrastructure for software testing: Final report. Diane Pub Co.

Available at http://bit.ly/2kT52OP

Is there an App, Website or Program that you think has excellent UX?

• It’s hard to think of one – this was one of the questions asked at the conference – and I still cannot think of a piece of software that does not irritate me… in some way.

Questions about the role of testers

There were a couple of questions about the role of testers and the nature of testing:

How would you describe testing? Art? Engineering? Craft?

How to engage testers to report more idea to improve automation tools base on their knowledge/experience other than bugs only.

• These are questions that I have pondered over the years. Testing for me engages many skills, personal attribute, styles, approaches – and we need that. In an individual, In a team. So, testing is an art, a craft, it can be a science, it can be engineering, it can be detective work, it can be sociology, it can be ethnographic, it can be research. It can be technical or domain focused. Qualitative and quantitative approaches – both are needed.
• This is interesting: I do believe that testers have a wider role than just reporting bugs – that my role is not just about bugs. Bugs are just one aspect of quality, and just one. I have always taken a wide view – that it is not just the software that is under test, it is also the whole product, the whole service, all the activities of the organisation, everything that contributes to the success of what we are trying to do. So, for me, to engage in critiquing and improving methodology, tools, skills – including automation – anything that needs to improve, to facilitate overall success of the project, the team. Do it imaginatively, think widely.
• To engage testers to do this – to engage anyone to take this view – we need to be open to constructive criticism and improvement ideas.

Questions about introducing UX practices into projects

From your experience what % of a project budget should be spend on making UX right?

• I am not sure there is a right answer to this! It depends on the project, the risks. A more useful approach than having a set percentage could be to have a “start of project” workshop where the purpose of the project / product is discussed, along with the user personas, quality attributes, UX, project goals and so on. And, constraints like budget and timescale. From that you’d get an idea of the UX work needed / possible for that project. One thing is certain – earlier effort is more cost effective than testing post coding.
• If you are part way through the project, sometimes it can be worth halting for a few hours to ask “are we on track? Do we agree on direction and UX goals?” so if you are part way through – you can still address this.

Mostly customer is happy when software helps him even if the UX is not perfect.

How to convince management to invest in better UX?

Is UX something that user really expects from software, isn’t it enough that software Just does its job?

• Three in one! I like your style…!
• Let’s start with your first statement – the customer is happy even if the UX is not right. Now this is sometimes true: if this is brand new software, and if the customer is someone who enjoys innovation and is therefore forgiving of problems in new technology. And if they don’t mind the struggle. But often, customers can be just resigned to the experience rather than enjoying it… they’ll be grumbling. So even if people are using our products, they might not continue to use them if a better experience is offered elsewhere. People’s expectations of what they can reasonably expect change over time. In the past in IT we’ve delivered functionality – that’s not enough now, because software is ubiquitous, and people’s expectations have changed.
• Follow the money. Senior stakeholders want the organisation to be successful – and often have KPIs (Key Process Indicators) to help them measure what they find important. If poor UX is going to lose the organisation money, reputation, or market share, for example, the senior stakeholders will look for ways to halt those losses. If you can show evidence of poor UX, not just in the software but in the whole user experience, and you can show that better UX will lead to an improved reputation, market share, etc., then you’ll be listened to. You won’t always win – and if you think you are not going to win, sometimes, you have to have the conversation, again, more uncomfortably, after a failure. Also, get senior stakeholders, developers, etc to look at recording of people using the software, or to observe usability tests. Sometime the marketing department or support team can be allies.
• See answer 1… I remember my grandfather disapproved of car heaters. He thought they were an unnecessary luxury. So as children we travelled with car blankets… uncomfortable, functional, and not what I would tolerate nowadays.

How to convince customer that their vision of UX is a disaster?

• Look for points where they will lose money, status, market share, reputation. Use examples from other products, talk with them about what they like and dislike about software. Video record and play back to them real users engaging with the product, or better still ask them to be observers during usability testing and UX assessments. Engage with the marketing department – who are often very interested in the UX.

Is It our responsibility as SW testers to test software to serve users as much as possibility even if software is written user-unfriendly but written according to requirements provided by customers / specifications / project managers?

• Yes – I think we need to keep a bigger picture. Requirements can be – are often – wrong, incomplete, out of date, misunderstood. We won’t always get the result we want, it won’t be our decision, but we should put forward ideas for improvement.

Two questions that are related:

Is it plausible for testers to perform usability tests? Aren’t they biased and “cursed with knowledge” about the product? Shouldn’t we rather try to gather feedback from real users?

Is it possible to perform good quality UX tests by QA team only? Or should those always involve real users?

• Let’s look at usability tests. Testers can perform usability tests – they are not necessarily the test subject. If you are executing a usability test, you are observing someone else using the software, so as a tester you observe, record, analyse what is happening, but you are not driving the software – you are watching someone else. You are looking at the face, eye movements, hand movements, how the mouse and keyboard are being used, how confident the person seems to be. Hands off. Observe and think.

Sometimes you don’t have access to real users. So your options are:

• You know who the real users are, you can talk with them, they can come and test the system, give comments on functional attributes and other quality attributes, and also you can observe them when you perform usability tests with your real users as the subjects.
• You may only know the users by persona descriptions – for example for a website, the general public. You could ask people to come in and run usability tests, or you could take a usability or UX “road show” out to the public. Or you could ask a UX or Usability specialist test lab to do the assessment for you.
• You could ask the marketing team for their market research about the product and you could ask Marketing, Sales and Support team people to proxy for the customers – you could ask them to review the interfaces, prototypes and the software specifically considering the UX. They should have contact with and understand the users.
• You could ask people from your user group – if you have one – to come in to take part in UX assessments.
• You could ask users to beta test the software and specifically ask for UX and usability issues.
• You could monitor the use of the software in production and see how it is being used, when people drop out, and you can do online user surveys.
• If you cannot do any of those – if you have no contact with end users – as testers use empathy and thought experiments to put yourself in the shoes of the users. For example, there is technique called cognitive walkthrough. In a group, you work through the product at each point asking “what would a user be thinking now?” Not what you are thinking – what the user is thinking.

• The more new people you get looking at the software, the more you’ll keep hold of first reactions.

What is, in your opinion, the best way to combine these two different points of view – users who are feeling connected to the old version of product and users who really want new features and waiting for the change?

• You have a number of options that you can design into the product. For example:
• Design in “backwards compatibility” – make sure that older versions of companion software still work with the new version of your product;
• Give a choice about whether users upgrade or not;
• Give a choice about whether new features are switched on or not
• Make the new features learnable;
• Make the upgrade and first use help/encourage users to the new version;
• Do focused marketing to the customers who want the change.

Two more questions that belong together:

At which state of the project should we start to test UX solutions?

Do you think that UX should be done before a line of code is written or should it be done alongside it?

• Quality is built in throughout the project, and that’s true of UX as well. At every point, every stage, testing should be happening – by questioning, reviewing, thought experiments, what if questions, monitoring what people are actually doing with a product, as well as by running executable tests.
• Do it at the start. Build it in. But also, once the product is in use, monitor it and monitor responses to it, to get ideas for where else to improve. In fact, many of the activities that sustain, facilitate and support quality for the customer and end user are best started early. Get involved at concept stage, when your job as a tester is to question, suggest improvements, analyse risks, work out what will best serve the customer, user, your organisation, society… in this project.

What should we do if we make product designed by marketing team, before we know or understand who our customers are going to be?

• The marketing team should have done some market research, and may have drawn up personas and run focus groups as part of that research. So, your first step is to find out what the marketing team have done so far to understand the market and who the product is for. They should know the customers. Have a meeting or workshop with them to find out what is in their vision.

How can you tell when you should stop improving the UX not to go too far to avoid making the interface too unfamiliar for the user?

• “Too much” indicates that the UX is poor. Doing things to make the interface look prettier are not necessarily good UX -sometimes the pretty interface prevents the user from having a good experience… Good UX is about understanding the user, not about what the engineers want.

What’s your opinion on the role of documentation in UX? Should documentation be tested as part of product testing to ensure excellent UX?

• Yes absolutely – it is part of the product, and part of the experience that the user has. It needs to be tested to make sure it is functional, suitable, provides a good experience, is easy to navigate by someone who does not know the product. It could have errors, omissions, ambiguities – in other words bugs. And – don’t forget the buying process, the install process, the set up… they all add to or detract from UX.

Questions about the UX of test tools

These questions asked do we need skills or do we need UX? They are exciting questions…Ones I am asking myself too…

Imagine you have a labour with a hammer, he pins nails in wooden surface. If he never uses the second side of hammer to pull out wrongly pinned nails and instead pins another correctly – should we redesign hammers or teach him how to use it?

Increased demand on plane pilots didn’t resulted with decreasing requirements for wannabes.

Don’t you think that for most of tools we should demand more knowledge from new users than spend a lot of time on changing whole tool for better UX?

• These questions are really good – it is central to what I am looking at – do we make sure the testers have the skills to use the tools, or do we change the tools so the testers can use them? I don’t know the answer – I know what I am inclined to think, but the purpose of the research is to challenge that thinking… am I right or wrong? Is there a problem? If there is a problem how can it be solved? It depends…Do the tools need to change or the testers need to change? I hear lots of opinion, but I don’t know…
  • If we want someone to be a carpenter, and to learn those skills, and they are working on small, slow projects, or an occasional project where a small number of nails are needed, then teaching them to use the hammer well is good.
  • If someone comes up with a better design for a hammer, we’ll welcome that. And – for small home projects – maybe we’d encourage them to use a different approach – like “no more nails” [http://www.unibond.co.uk/en/diy-adhesives/no-more-nails.html]
  • If we are working on a big construction project where 1000’s of nails are needed, a hammer won’t be the right tool; we need to get them a nail gun, or some other machine, and teach them to use that. And, we’ll want the nail gun to be easy to use…
• Thanks! Your question is really good one!
• If we turn to pilots, UX and usability for pilots is really important: plane crashes were happening because of human factors, usability and UX reasons [just search “usability plane crashes” to see examples]. So much work has been done in the avionics industry to make piloting a plane easier. Still, pilots need knowledge so that they can take over from the automatic pilot if something goes wrong. There has been some interesting discussion about this – how much the use of automation de-skills pilots? [see Nicholas Carr’s writing for example] That’s for a whole other blog.
• Piloting a plane – like driving a car – has changed with automation, and the usability of the automation has been key.
• This pair of questions – I love them – this is what we need to debate, research and find out the answer to.

Do you have any suggestions how to make regression tests more reliable e.g. writing unit tests for automated test cases, keep part of regression manual, etc.?

• It depends! Automating unit testing, and getting important regression tests into the unit test automation is useful. Find and fix regressions quickly. I think also understanding the content of the regression tests is useful – which parts of the system do you always want to exercise after change? Which parts of system need exercising after a specific change? Which regression tests address those questions? Which ones don’t we need to run this time? You may well find that some tests are not usefully automated at all. Understand what you are trying to achieve with the regression testing and then decide what and how to automate. Work to have smaller, focused regression testing.

What would be your recommendation with regard to configuration/maintenance costs: Compose workflow consisting of atomic tools suitable for particular activities or use ready prepared ecosystems of tools and plug-ins.

• It depends on your project and circumstances – there is not a set answer to this question. It’s an area to explore in the research – thanks!

In your opinion what is the future perspective of ‘automating the automation’ with AI, ML, deep learning, etc – how big will be the share of tasks overtaken by those mechanisms (like test design, fault reporting, test results analysis)…

• Next big thing – we’re on the brink. There will still be a place for testers, but it will change – we’ll need to be adaptable. Tariq King gave an excellent keynote about this at STARWest 2017 – it is worth listening to him…

What do you expect to achieve at the end of your research? How do you see a day when it is over?

• I hope to have answers (at least provisional answers) to some of these questions. I hope to have engaged people who provide tools into thinking about UX. And I hope to have learnt a lot and rejuvenated my thinking -challenged myself, learnt new things, fed back ideas into the industry. It’s going to take 6-8 years, and I hope to feedback results during that time.

How can each of us help with your research?

• Send me your stories – what works for you, what does not work, why you think that might be, what you would like your tools to be like. DO that via the contact page on my website isabelevans.uk – that link will take you to the contact page.

If “Should I stay or should I go” is already mentioned – what is your favourite song describing software testing?

• At the conference, I was tongue tied, here are some thoughts – you can decide what the message is for you…
  • Paranoid
  • Follow the Yellow Brick Road
  • These foolish things
  • I can’t turn you loose
  • Sweet dreams are made of this
  • Right by your side
  • Don’t stop thinking about tomorrow
  • Wow
  • The gambler
  • All of the day and all of the night
  • Tired of waiting
  • It’s nobody’s fault but mine…
  • Funny how time slips by
  • Five o’clock somewhere

Thanks for all the questions!

 

4/11/2017

Why we need a better tool set for building software

At the end of my last blog, I asked, “So – why is technology flawed? Why not build it well in the first place?” There could be several reasons, today I want to discuss just two of many:

• Limited types of people in the industry: The software engineers do not always understand what is needed because they may not understand the people who will use the technology. Because of this they may deliver software that is the product they themselves want to use. This means technology can be biased towards the type of people who are already software engineers.
• Automation bias and poor decision making: The software engineers and the non-engineers on the project team may have a toolset that does not enable them to make good decisions. Yes, all the flaws in technology that I talked about in the previous blog are also true for the tool set that software engineers use. The tools are intended to help decision making in the project. Is the software ready? Has it been tested? Does it work? Are there any risks in delivering it now? But if the tools are flawed, and hard to use, people are more likely to make poor decisions with poor outcomes. That will lead to flawed software and potentially an ugly UX for everyone else. Note that technologists and software engineers also suffer from software with flaws in it, software that is hard to understand, and tools that prompt them into automation bias.

Two sayings that come to mind here “a fool with a tool is still a fool” and “a poor workman blames his tools” and it would be easy to stop at that. But the industry has spent too long building up some traditions around old mind sets and old tool sets that favour old ways of doing things. If technology and software is to serve and reflect the widest population, to everyone’s benefit, then the teams need to reflect society. If the teams reflect society, then the tool set needs to be easily used by a wider range of people, not just the obvious technologists.

As Seth Godin remarks in [Demand Guardrails] “If peer pressure and short-term urgencies set us up to do things we regret, we come out ahead when we support cultural changes that remove that peer pressure and lessen those short-term urgencies.” This blog is a call for a change to the tool sets used by software engineers, to make them easier to use, easier to understand, and supportive of good decision making that tends to a beautiful UX for everyone. We need to support cultural changes in the world of software engineering that remove the peer pressure of being the same as we’ve always been. We need to support cultural changes that remove short term urgencies that stop us mending out tool set. We are the cobblers, and we are the cobblers’ children with no shoes.

Old tools sets are hard to use. Early sewing machines required skills with the machines that are not needed by modern machines; a sewer today can concentrate much more on the project, and less on the tools. Early cars were hard to drive, with starting handles and double de-clutching to master. Modern cars are easy to start, easy to drive, automatic gears and power steering. The driver can concentrate on the journey and the destination. More people can use a sewing machine now, and make their own clothes. More people can drive, and have the independence to make journeys. [Rise of the female petrolhead, Car electric starter, Treadle sewing machine difficulty]

I would like to move software engineering tool sets to beyond the treadle machine and the car with a crank handle starter to a modern, smooth system where automation and tools serve the project team rather than being a hindrance. This will address both the problems above:

• Bias: The software engineering world was, when I joined it, more inclusive than it has been in the last couple of decades. [Grace Hopper; Steve Shirley; Queer history of computing; Saving-bletchley-park] Even when I first started however, a bias towards a young, masculine culture was starting. This has become self-perpetuating with terminology, behaviour, tool interfaces combining into a culture that can be off putting to women [Frig – as insult and software activity, Git – as an insult and software tool]. Gender and other bias in IT tools, for example in tool default behaviour, use of language, voice recognition, gendering of support tools, is increasingly being studied and remarked on for example in these articles and paper on gender and culture boas: What Gender Stereotypes and Sexism Have to Do With Algorithms and Robots, Gender Dimensions of Product Design , Cultural Bias in Information Systems. Here are quotes from attendees at the Women in Innovation briefing meeting I attended in June:

“it is like working with something designed to be used by a 12-year-old boy in his bedroom in the 1980’s”.

“Why would I want to use a tool called Github?”

But this is not just a problem about gender or culture; the tools are themselves flawed and old-fashioned.

• Tool flaws: IT teams are themselves beset with the same problems and frustrations with the tool set they have to use, and this leads to the release of highly flawed software. Many IT tool sets are almost deliberately arcane and are gender biased. Many become “shelfware” – purchased but hardly ever used. Others are used with automation bias – the results are wrong but are treated as correct. Here are some more quotes from developers, testers, and researchers about flaws in tools

Developer: “I spend 50% of my time wrestling with the technology instead of solving the problem I am working on” Quote from WII meeting

Software tester finding decision making not supported by IT toolset: “The test tool marked all the tests as passed except 1, but in fact none of the tests marked “passed” had actually run” Quote from Fewster and Graham “Experiences of Test Automation”

Problems with customer support tools: “Ethnographic research paints a sad picture of the current state of the ITSM market.   …vision is to build a solution designed for humans, not processes” [http://blogs.ca.com/2016/01/27/moving-itservice-management-to-the-21st-century/]

Evidence that developers do not find tools easy to use: “…so now I wanna know why raising a string exception is bad. Like what should I be doing instead? Since it thinks it’s a problem. And so none of these really help me…” (Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? By Johnson, Song, and Murphy-Hill).

Evidence that tools do not work for IT people “…a lack of consideration for how people work and think … basically it’s still the mindset that the human adapts to the computer, not vice-versa.” (A Taxonomy of Tool-Related Issues Affecting the Adoption of Model-Driven Engineering by Whittle, Hutchinson, Rouncefield, Burden and Heldal)

In the previous blog I said: The reasons that technology and software may prompt poor decision making include defects in the software […or that it…] may function correctly, but in a way that does not aid people in making good decisions. … dealing with frustrations, defects and consequences of poor engineering is a bad experience for the people using the technology; a flawed, ugly UX. That is true for tools used by the teams delivering software, and is a key root cause for the problems we all experience with delivered software.

What I will propose in my next blog is a solution to these problems.