Category Archives: user experience

CHIIR conference report – keynote highlights

The conference opened on Monday with a keynote from Ranjitha Kumar, which I found eye-opening and inspiring. Her team are working on “Data Driven Design: beyond AB testing” She pointed out that money spent on design does not always repay in results, and that A/B testing can be usefully supplemented with oher methods. In particular her team is working on “design mining” (rather than data mining) to find out what designs are being used elsewhere – she said there is a rich seam of designs available which give inspiration and a test / review point. She talked about the need to connect design with KPI’s, and to understand the success of designs in terms of their effect on KPI’s.

The second keynote, on Tuesday was also fascinating. Daniela Petrelli showed three case studies of making visitor experiences during museum visits multisensory, more engaging and more memorable. By using IoT technology, objects can be used to engage visitors in specific stories. I particularly loved the votary lamp that allows visitors to an exhibit on Hadrian’s wall chose three items – each a different god – and receive a personalised postcard with oracle-like messages. This a study at Chesters Fort , specifically around the Visitor eXperience of the Clayton collection. The three case studies indicated that visitors are more engaged and remember more, because they slow down and take longer to examine objects, when they use a physical object to access information – rather than a digital screen/phone. The IoT technology allows small objects – facsimiles that can be held in one’s hand – to be used to interact with video, audio, etc related to exhibits, and allow visitors to choose the viewpoint they experience in their journey through the museum.

I loved these two keynotes, interesting in so many ways – for me as a comsumer of information on the web and in museums, but also as a test consultant. Possible analogies – these gave me some thoughts about the experience of testers in their projects.

  • For example, if it true that people are more engaged and remember more when interacting with physical objects, could we use this idea to change how people examine and interact with information generated by testing? This is NOT age related… What does it tell us about how we generate, use and display information?
  • for example, if design mining is a useful supplement to A/B testing, how could it be used to supplement how we test designs – could it be a source for heuristics to use when testing interface designs?
  • for example, what we as digital experts provide and are proud of, is not always what the consumers of our work want or expect, For example, the questions that a search engine or chat bot responds to are not always the questions consumers want to ask. How can testers find out and understand what consumers actually want? That includes the consumers of the information from testing.
  • From those questions, I wonder about our testing dashboards – not for the first time in my decades in industry – and why we don’t talk with our stakeholders, in their language. I’ve been talking about this for years, presenting on it, teaching about it… I’ll continue with that. Quote from K1 about fashion websites – customers ask for “hot pink” websites talk about “Fuchsia” or “magenta”
  • K2 provided a mini lifecycle for co-design and co-development where a technical person, a designer and a curator get together and split apart repeatedly to generate and test the ideas and design for artefacts. Is there an analogy to the developer, UXer and product Owner, and if so, where is the testing, and is there a need for a specific tester role?

Advertisements

Conferences 2018 – a reflection

It’s been a busy year for conferences and teaching – so here is a quick summary…

UKSTAR 2018

This was in London, UK, in March.

I enjoyed day 1, with excellent keynotes from Christina Ohanian, and Gustav Kuhn. Christina’s messages on building teams were full of insights. As well as Gustav’s keynote, I’d also enjoyed Gustav’s workshop with Alan Richardson. The application of magic to psychology and then to how we observe (or don’t observe) the phenomena about us was very revealing, and certainly applicable to testers. Do we see the things other people don’t see? Can we be tricked? Powerful stuff.

Day 2 of UKSTAR 2018 was a very – extreme – experience for me. I was booked to present the opening keynote and then a workshop, and just before I was due to go on stage I had a completely unexpected phone call to say my mother had died. I made the decision that I wanted to go ahead – she would have wanted and expected that – discussed it with the programme committee and the UKSTAR team, and with their full and immense support went ahead to give my keynote – on Leadership, Followership and Fellowship – which went well, on a surge of adrenlin that also carried me through my workshop where we shared information around the UX of testing tools for testers – contributing data to my research and allowing delegates to share experiences and learn some UX techniques to apply to their own test tool acquisitions.

I used the data I collected during the workshop as one of the inputs to the Webinar I did for EuroSTAR, which also allowed me to open an online survey, and collect more data. The webinar and survey are still available here: https://huddle.eurostarsoftwaretesting.com/webinar-questionnaire-no-shelfware-lets-drive/

Thank you so much EuroSTAR and UKSTAR teams for your support during UKSTAR and beyond!

Romanian Code Camp

This was in Iasi in March. I was so pleased (given my personal circumstances) to be there with my friend and colleague Sue Atkins. Again the conference organisers were so supportive! As well as a masterclass on quality in use and UX, and a masterclass on Leadership, Followership and Fellowship, I presented on Human Factors for Test Automation, Sue and I presented a joint session on State Transition Testing, and also we both gave lightning talks. I was delighted to meet Vijay Kiran, who gave an inspirational lightning key on the importance of ethics in development – that excellent software is not just well engineered, not just exhibiting excellent UX, but also is ethically sound – doing good. I’ve been quoting him all year since! It was enjoyable to be at a conference with a range of tracks as well as a testing track: architecture, design, frontend, web, IoT, engineering, leadership, agile, entrepreneurship, and – my favourote track title – “funa dn fearless”. This conference was also beautifully hosted, ending with a cocktail party for speakers, hand made, natural ingredients – and the most delicious non-alcoholic one featuring apples and peanuts! I wish I could remember the name of the people who produced them!

STAREAST

STAREAST is always good fun, and this year was no exception…  I presented two tutorials: “Influence Diagrams – a new way to understand testing” and “Transforming Testing: building your road map”. These were both half day workshops, although closely linked together. I also presented a track on “Devices and Desires” about our attitude to technology and how that matches against the needs and desires of people outside IT. An interesting couch discussion with a group of delegates about leadership completed my contributions to the conference. Photo:  https://stareast.techwell.com/conference-photo/se18-couch-session-isabel-evans

While there, I enjoyed the Women Who Test day; each time I attend I learn something new about myself, and enjoy the perspectives of the other attendees (men and women are welcome, by the way).

HTB Workshop and HUSTEF preparation

During 2018, I was honoured to be the HUSTEF Programme Chair. More on the latter later! In June, while in Budapest for the programme planning meeting, I also presented a workshop for the HTB on Quality in Use, and at the Tezst and Tea meet up presented “No more shelfware!”

The programme planning went well, it was a real joy to work with the programme committee and the HUSTEF organising team!

BCS SIGiST

This meeting took place in London in June. Stuart Reid and the committee had organised an “all keynotes” day, which was really good fun. I met up with old friends, and enjoyed their talks, as well as presenting “Devices and Desires” to an appreciative audience.

ODIN

Off to Oslo in September for a delightful conference, I gave tutorials on UX, and on Human Factors for test automatio, as well as a keynote on Leadership. At ODIN I also met up with Lorraine and Siobhan of the EuroSTAR team, who took me aside for a coffee, a catch up… and an interesting request…

TESTJAM

Straight from Oslo to Utrecht, where I presented at Capgemini’s Testjam, on Devices and Desires, and met up with old friends (Nathalie and Kimberley and Rik) from Capgemini… A lovely evening, ending with a dash to the airport as storms disrupted air travel… 

STARWEST and STARCANADA

Before STARWEST, the various personal events of the year meant that I could get to conferences and deliver, but was unable to take in any information! By STARWEST and STARCANADA, I felt more settled, and enjoyed listening to the keynotes and tracks. STARWEST was busy busy busy and great fun!

Across the two conferences, I taught 5 tutorials (test design, requirements testing, UX, influence diagrams, automation: a human-centred approach) which all went well plus a lightning key, 2 track sessions, 2 couch sessions, and a talk on failure at Women Who Test.

I’d also been on the programme committee with Rob Sabourin (the chair) and Julie Gardiner. That had been a really interesting experience, and I was delighted to have the chance to contribute, and to help choose the speakers. My feel is that these were two great programmes – but that of course is author bias!

Great keynotes! Jennifer Bonine and Janna Loeffler on story telling – great production values, wonderful illustrations by a Disney illustrator, and a great message well delivered. Jon Bach’s courageous use of a live survey with the audience via an app was really enjoyable, as well as his thoughts on how one’s behaviour changes with one’s role. Dona Sarkar on “be the lord of your own rings” was a fireball of energy. Max Saperstone showed a brilliant use of mutation syntax testing. Fiona Charles discussed leadership and how it differs from Management. Alexandre Baudin showed us how to test flight simulators, and Sophie Benjamin eloquently told how to transform testing.

Among the tracks I enjoyed Jane Jeffers from Riot Games on asking Why, and Julene Johnson from Lucid Software on Anxiety. ALso Stefan Marceau and Keith Turpin on User Stories, Fiona Charles on Gaining Consciousness, 

Women Who Test maintained its celebratory nature, in particular from a rich day, I’ll pick out getting to see the first printing of Tania Katan’s book on creative trespassing…

HUSTEF

My first big conference as programme chair! It’s been hard work, but what an experience! I am so pleased to have done this… In fact, I think it deserves a separate post…

EuroSTAR

The final conference of the year… and as always like coming home. Old friends abound, so many greetings, embraces, and conversations! Friends from all over the world! 

I think this one deserves its own post too – lots to say, and this post is already…. too long!

Nokia Test Dive 2017 – audience questions

I had a great time at the Nokia Test Dive in Krakow – a lovely conference, responsive delegates, and hosted beautifully by Nokia.

After both my sessions, there were audience questions on Sligo that we did not get time to discuss, so I have taken those questions, grouped them, and made some responses. They were great questions!

Some quick answers to start…

What is your UX of Kraków / Poland

  • Lovely question!
  • Great UX, I loved the city, so beautiful. I’d like to visit again. The historic centre is really interesting, the different architecture, especially the art nouveau.
  • I lost my hat – but that was my fault!

Is there a universal solution to prevent “shelfwareness”?

  • Such a wonderful question! If I knew the answer I probably would not need to be researching! Maybe there is… I suspect if the answer was easy someone would have found it already. I’m going to borrow this question and use it when I interview experts and tools suppliers… Thank you!

The billions of annual loss due to wrong automation – is it for one company? Did they do something with that?

  • This is from a report about the cost of poor test infrastructure in the USA – across the country. The report reference is in the slides, in the reference pages at the back of the slide set, but here it is:

Tassey, G. (2003) The economic impacts of inadequate infrastructure for software testing: Final report. Diane Pub Co.

Available at http://bit.ly/2kT52OP

Is there an App, Website or Program that you think has excellent UX?

  • It’s hard to think of one – this was one of the questions asked at the conference – and I still cannot think of a piece of software that does not irritate me… in some way.

Questions about the role of testers

There were a couple of questions about the role of testers and the nature of testing:

How would you describe testing? Art? Engineering? Craft?

How to engage testers to report more idea to improve automation tools base on their knowledge/experience other than bugs only.

  • These are questions that I have pondered over the years. Testing for me engages many skills, personal attribute, styles, approaches – and we need that. In an individual, In a team. So, testing is an art, a craft, it can be a science, it can be engineering, it can be detective work, it can be sociology, it can be ethnographic, it can be research. It can be technical or domain focused. Qualitative and quantitative approaches – both are needed.
  • This is interesting: I do believe that testers have a wider role than just reporting bugs – that my role is not just about bugs. Bugs are just one aspect of quality, and just one. I have always taken a wide view – that it is not just the software that is under test, it is also the whole product, the whole service, all the activities of the organisation, everything that contributes to the success of what we are trying to do. So, for me, to engage in critiquing and improving methodology, tools, skills – including automation – anything that needs to improve, to facilitate overall success of the project, the team. Do it imaginatively, think widely.
  • To engage testers to do this – to engage anyone to take this view – we need to be open to constructive criticism and improvement ideas.

Questions about introducing UX practices into projects

From your experience what % of a project budget should be spend on making UX right?

  • I am not sure there is a right answer to this! It depends on the project, the risks. A more useful approach than having a set percentage could be to have a “start of project” workshop where the purpose of the project / product is discussed, along with the user personas, quality attributes, UX, project goals and so on. And, constraints like budget and timescale. From that you’d get an idea of the UX work needed / possible for that project. One thing is certain – earlier effort is more cost effective than testing post coding.
  • If you are part way through the project, sometimes it can be worth halting for a few hours to ask “are we on track? Do we agree on direction and UX goals?” so if you are part way through – you can still address this.

Mostly customer is happy when software helps him even if the UX is not perfect.

How to convince management to invest in better UX?

Is UX something that user really expects from software, isn’t it enough that software Just does its job?

  • Three in one! I like your style…!
  • Let’s start with your first statement – the customer is happy even if the UX is not right. Now this is sometimes true: if this is brand new software, and if the customer is someone who enjoys innovation and is therefore forgiving of problems in new technology. And if they don’t mind the struggle. But often, customers can be just resigned to the experience rather than enjoying it… they’ll be grumbling. So even if people are using our products, they might not continue to use them if a better experience is offered elsewhere. People’s expectations of what they can reasonably expect change over time. In the past in IT we’ve delivered functionality – that’s not enough now, because software is ubiquitous, and people’s expectations have changed.
  • Follow the money. Senior stakeholders want the organisation to be successful – and often have KPIs (Key Process Indicators) to help them measure what they find important. If poor UX is going to lose the organisation money, reputation, or market share, for example, the senior stakeholders will look for ways to halt those losses. If you can show evidence of poor UX, not just in the software but in the whole user experience, and you can show that better UX will lead to an improved reputation, market share, etc., then you’ll be listened to. You won’t always win – and if you think you are not going to win, sometimes, you have to have the conversation, again, more uncomfortably, after a failure. Also, get senior stakeholders, developers, etc to look at recording of people using the software, or to observe usability tests. Sometime the marketing department or support team can be allies.
  • See answer 1… I remember my grandfather disapproved of car heaters. He thought they were an unnecessary luxury. So as children we travelled with car blankets… uncomfortable, functional, and not what I would tolerate nowadays.

How to convince customer that their vision of UX is a disaster?

  • Look for points where they will lose money, status, market share, reputation. Use examples from other products, talk with them about what they like and dislike about software. Video record and play back to them real users engaging with the product, or better still ask them to be observers during usability testing and UX assessments. Engage with the marketing department – who are often very interested in the UX.

Is It our responsibility as SW testers to test software to serve users as much as possibility even if software is written user-unfriendly but written according to requirements provided by customers / specifications / project managers?

  • Yes – I think we need to keep a bigger picture. Requirements can be – are often – wrong, incomplete, out of date, misunderstood. We won’t always get the result we want, it won’t be our decision, but we should put forward ideas for improvement.

Two questions that are related:

Is it plausible for testers to perform usability tests? Aren’t they biased and “cursed with knowledge” about the product? Shouldn’t we rather try to gather feedback from real users?

Is it possible to perform good quality UX tests by QA team only? Or should those always involve real users?

  • Let’s look at usability tests. Testers can perform usability tests – they are not necessarily the test subject. If you are executing a usability test, you are observing someone else using the software, so as a tester you observe, record, analyse what is happening, but you are not driving the software – you are watching someone else. You are looking at the face, eye movements, hand movements, how the mouse and keyboard are being used, how confident the person seems to be. Hands off. Observe and think.

Sometimes you don’t have access to real users. So your options are:

  • You know who the real users are, you can talk with them, they can come and test the system, give comments on functional attributes and other quality attributes, and also you can observe them when you perform usability tests with your real users as the subjects.
  • You may only know the users by persona descriptions – for example for a website, the general public. You could ask people to come in and run usability tests, or you could take a usability or UX “road show” out to the public. Or you could ask a UX or Usability specialist test lab to do the assessment for you.
  • You could ask the marketing team for their market research about the product and you could ask Marketing, Sales and Support team people to proxy for the customers – you could ask them to review the interfaces, prototypes and the software specifically considering the UX. They should have contact with and understand the users.
  • You could ask people from your user group – if you have one – to come in to take part in UX assessments.
  • You could ask users to beta test the software and specifically ask for UX and usability issues.
  • You could monitor the use of the software in production and see how it is being used, when people drop out, and you can do online user surveys.
  • If you cannot do any of those – if you have no contact with end users – as testers use empathy and thought experiments to put yourself in the shoes of the users. For example, there is technique called cognitive walkthrough. In a group, you work through the product at each point asking “what would a user be thinking now?” Not what you are thinking – what the user is thinking.
  • The more new people you get looking at the software, the more you’ll keep hold of first reactions.

What is, in your opinion, the best way to combine these two different points of view – users who are feeling connected to the old version of product and users who really want new features and waiting for the change?

  • You have a number of options that you can design into the product. For example:
  • Design in “backwards compatibility” – make sure that older versions of companion software still work with the new version of your product;
  • Give a choice about whether users upgrade or not;
  • Give a choice about whether new features are switched on or not
  • Make the new features learnable;
  • Make the upgrade and first use help/encourage users to the new version;
  • Do focused marketing to the customers who want the change.

Two more questions that belong together:

At which state of the project should we start to test UX solutions?

Do you think that UX should be done before a line of code is written or should it be done alongside it?

  • Quality is built in throughout the project, and that’s true of UX as well. At every point, every stage, testing should be happening – by questioning, reviewing, thought experiments, what if questions, monitoring what people are actually doing with a product, as well as by running executable tests.
  • Do it at the start. Build it in. But also, once the product is in use, monitor it and monitor responses to it, to get ideas for where else to improve. In fact, many of the activities that sustain, facilitate and support quality for the customer and end user are best started early. Get involved at concept stage, when your job as a tester is to question, suggest improvements, analyse risks, work out what will best serve the customer, user, your organisation, society… in this project.

What should we do if we make product designed by marketing team, before we know or understand who our customers are going to be?

  • The marketing team should have done some market research, and may have drawn up personas and run focus groups as part of that research. So, your first step is to find out what the marketing team have done so far to understand the market and who the product is for. They should know the customers. Have a meeting or workshop with them to find out what is in their vision.

How can you tell when you should stop improving the UX not to go too far to avoid making the interface too unfamiliar for the user?

  • “Too much” indicates that the UX is poor. Doing things to make the interface look prettier are not necessarily good UX -sometimes the pretty interface prevents the user from having a good experience… Good UX is about understanding the user, not about what the engineers want.

What’s your opinion on the role of documentation in UX? Should documentation be tested as part of product testing to ensure excellent UX?

  • Yes absolutely – it is part of the product, and part of the experience that the user has. It needs to be tested to make sure it is functional, suitable, provides a good experience, is easy to navigate by someone who does not know the product. It could have errors, omissions, ambiguities – in other words bugs. And – don’t forget the buying process, the install process, the set up… they all add to or detract from UX.

Questions about the UX of test tools

These questions asked do we need skills or do we need UX? They are exciting questions…Ones I am asking myself too…

Imagine you have a labour with a hammer, he pins nails in wooden surface. If he never uses the second side of hammer to pull out wrongly pinned nails and instead pins another correctly – should we redesign hammers or teach him how to use it?

Increased demand on plane pilots didn’t resulted with decreasing requirements for wannabes.

Don’t you think that for most of tools we should demand more knowledge from new users than spend a lot of time on changing whole tool for better UX?

  • These questions are really good – it is central to what I am looking at – do we make sure the testers have the skills to use the tools, or do we change the tools so the testers can use them? I don’t know the answer – I know what I am inclined to think, but the purpose of the research is to challenge that thinking… am I right or wrong? Is there a problem? If there is a problem how can it be solved? It depends…Do the tools need to change or the testers need to change? I hear lots of opinion, but I don’t know…
    • If we want someone to be a carpenter, and to learn those skills, and they are working on small, slow projects, or an occasional project where a small number of nails are needed, then teaching them to use the hammer well is good.
    • If someone comes up with a better design for a hammer, we’ll welcome that. And – for small home projects – maybe we’d encourage them to use a different approach – like “no more nails” [http://www.unibond.co.uk/en/diy-adhesives/no-more-nails.html]
    • If we are working on a big construction project where 1000’s of nails are needed, a hammer won’t be the right tool; we need to get them a nail gun, or some other machine, and teach them to use that. And, we’ll want the nail gun to be easy to use…
  • Thanks! Your question is really good one!
  • If we turn to pilots, UX and usability for pilots is really important: plane crashes were happening because of human factors, usability and UX reasons [just search “usability plane crashes” to see examples]. So much work has been done in the avionics industry to make piloting a plane easier. Still, pilots need knowledge so that they can take over from the automatic pilot if something goes wrong. There has been some interesting discussion about this – how much the use of automation de-skills pilots? [see Nicholas Carr’s writing for example] That’s for a whole other blog.
  • Piloting a plane – like driving a car – has changed with automation, and the usability of the automation has been key.
  • This pair of questions – I love them – this is what we need to debate, research and find out the answer to.

Do you have any suggestions how to make regression tests more reliable e.g. writing unit tests for automated test cases, keep part of regression manual, etc.?

  • It depends! Automating unit testing, and getting important regression tests into the unit test automation is useful. Find and fix regressions quickly. I think also understanding the content of the regression tests is useful – which parts of the system do you always want to exercise after change? Which parts of system need exercising after a specific change? Which regression tests address those questions? Which ones don’t we need to run this time? You may well find that some tests are not usefully automated at all. Understand what you are trying to achieve with the regression testing and then decide what and how to automate. Work to have smaller, focused regression testing.

What would be your recommendation with regard to configuration/maintenance costs: Compose workflow consisting of atomic tools suitable for particular activities or use ready prepared ecosystems of tools and plug-ins.

  • It depends on your project and circumstances – there is not a set answer to this question. It’s an area to explore in the research – thanks!

In your opinion what is the future perspective of ‘automating the automation’ with AI, ML, deep learning, etc – how big will be the share of tasks overtaken by those mechanisms (like test design, fault reporting, test results analysis)…

  • Next big thing – we’re on the brink. There will still be a place for testers, but it will change – we’ll need to be adaptable. Tariq King gave an excellent keynote about this at STARWest 2017 – it is worth listening to him…

What do you expect to achieve at the end of your research? How do you see a day when it is over?

  • I hope to have answers (at least provisional answers) to some of these questions. I hope to have engaged people who provide tools into thinking about UX. And I hope to have learnt a lot and rejuvenated my thinking -challenged myself, learnt new things, fed back ideas into the industry. It’s going to take 6-8 years, and I hope to feedback results during that time.

How can each of us help with your research?

  • Send me your stories – what works for you, what does not work, why you think that might be, what you would like your tools to be like. DO that via the contact page on my website isabelevans.uk – that link will take you to the contact page.

If “Should I stay or should I go” is already mentioned – what is your favourite song describing software testing?

  • At the conference, I was tongue tied, here are some thoughts – you can decide what the message is for you…
    • Paranoid
    • Follow the Yellow Brick Road
    • These foolish things
    • I can’t turn you loose
    • Sweet dreams are made of this
    • Right by your side
    • Don’t stop thinking about tomorrow
    • Wow
    • The gambler
    • All of the day and all of the night
    • Tired of waiting
    • It’s nobody’s fault but mine…
    • Funny how time slips by
    • Five o’clock somewhere

Thanks for all the questions!

 

4/11/2017