I had a great time at the Nokia Test Dive in Krakow – a lovely conference, responsive delegates, and hosted beautifully by Nokia.
After both my sessions, there were audience questions on Sligo that we did not get time to discuss, so I have taken those questions, grouped them, and made some responses. They were great questions!
Some quick answers to start…
What is your UX of Kraków / Poland
- Lovely question!
- Great UX, I loved the city, so beautiful. I’d like to visit again. The historic centre is really interesting, the different architecture, especially the art nouveau.
- I lost my hat – but that was my fault!
Is there a universal solution to prevent “shelfwareness”?
- Such a wonderful question! If I knew the answer I probably would not need to be researching! Maybe there is… I suspect if the answer was easy someone would have found it already. I’m going to borrow this question and use it when I interview experts and tools suppliers… Thank you!
The billions of annual loss due to wrong automation – is it for one company? Did they do something with that?
- This is from a report about the cost of poor test infrastructure in the USA – across the country. The report reference is in the slides, in the reference pages at the back of the slide set, but here it is:
Tassey, G. (2003) The economic impacts of inadequate infrastructure for software testing: Final report. Diane Pub Co.
Available at http://bit.ly/2kT52OP
Is there an App, Website or Program that you think has excellent UX?
- It’s hard to think of one – this was one of the questions asked at the conference – and I still cannot think of a piece of software that does not irritate me… in some way.
Questions about the role of testers
There were a couple of questions about the role of testers and the nature of testing:
How would you describe testing? Art? Engineering? Craft?
How to engage testers to report more idea to improve automation tools base on their knowledge/experience other than bugs only.
- These are questions that I have pondered over the years. Testing for me engages many skills, personal attribute, styles, approaches – and we need that. In an individual, In a team. So, testing is an art, a craft, it can be a science, it can be engineering, it can be detective work, it can be sociology, it can be ethnographic, it can be research. It can be technical or domain focused. Qualitative and quantitative approaches – both are needed.
- This is interesting: I do believe that testers have a wider role than just reporting bugs – that my role is not just about bugs. Bugs are just one aspect of quality, and just one. I have always taken a wide view – that it is not just the software that is under test, it is also the whole product, the whole service, all the activities of the organisation, everything that contributes to the success of what we are trying to do. So, for me, to engage in critiquing and improving methodology, tools, skills – including automation – anything that needs to improve, to facilitate overall success of the project, the team. Do it imaginatively, think widely.
- To engage testers to do this – to engage anyone to take this view – we need to be open to constructive criticism and improvement ideas.
Questions about introducing UX practices into projects
From your experience what % of a project budget should be spend on making UX right?
- I am not sure there is a right answer to this! It depends on the project, the risks. A more useful approach than having a set percentage could be to have a “start of project” workshop where the purpose of the project / product is discussed, along with the user personas, quality attributes, UX, project goals and so on. And, constraints like budget and timescale. From that you’d get an idea of the UX work needed / possible for that project. One thing is certain – earlier effort is more cost effective than testing post coding.
- If you are part way through the project, sometimes it can be worth halting for a few hours to ask “are we on track? Do we agree on direction and UX goals?” so if you are part way through – you can still address this.
Mostly customer is happy when software helps him even if the UX is not perfect.
How to convince management to invest in better UX?
Is UX something that user really expects from software, isn’t it enough that software Just does its job?
- Three in one! I like your style…!
- Let’s start with your first statement – the customer is happy even if the UX is not right. Now this is sometimes true: if this is brand new software, and if the customer is someone who enjoys innovation and is therefore forgiving of problems in new technology. And if they don’t mind the struggle. But often, customers can be just resigned to the experience rather than enjoying it… they’ll be grumbling. So even if people are using our products, they might not continue to use them if a better experience is offered elsewhere. People’s expectations of what they can reasonably expect change over time. In the past in IT we’ve delivered functionality – that’s not enough now, because software is ubiquitous, and people’s expectations have changed.
- Follow the money. Senior stakeholders want the organisation to be successful – and often have KPIs (Key Process Indicators) to help them measure what they find important. If poor UX is going to lose the organisation money, reputation, or market share, for example, the senior stakeholders will look for ways to halt those losses. If you can show evidence of poor UX, not just in the software but in the whole user experience, and you can show that better UX will lead to an improved reputation, market share, etc., then you’ll be listened to. You won’t always win – and if you think you are not going to win, sometimes, you have to have the conversation, again, more uncomfortably, after a failure. Also, get senior stakeholders, developers, etc to look at recording of people using the software, or to observe usability tests. Sometime the marketing department or support team can be allies.
- See answer 1… I remember my grandfather disapproved of car heaters. He thought they were an unnecessary luxury. So as children we travelled with car blankets… uncomfortable, functional, and not what I would tolerate nowadays.
How to convince customer that their vision of UX is a disaster?
- Look for points where they will lose money, status, market share, reputation. Use examples from other products, talk with them about what they like and dislike about software. Video record and play back to them real users engaging with the product, or better still ask them to be observers during usability testing and UX assessments. Engage with the marketing department – who are often very interested in the UX.
Is It our responsibility as SW testers to test software to serve users as much as possibility even if software is written user-unfriendly but written according to requirements provided by customers / specifications / project managers?
- Yes – I think we need to keep a bigger picture. Requirements can be – are often – wrong, incomplete, out of date, misunderstood. We won’t always get the result we want, it won’t be our decision, but we should put forward ideas for improvement.
Two questions that are related:
Is it plausible for testers to perform usability tests? Aren’t they biased and “cursed with knowledge” about the product? Shouldn’t we rather try to gather feedback from real users?
Is it possible to perform good quality UX tests by QA team only? Or should those always involve real users?
- Let’s look at usability tests. Testers can perform usability tests – they are not necessarily the test subject. If you are executing a usability test, you are observing someone else using the software, so as a tester you observe, record, analyse what is happening, but you are not driving the software – you are watching someone else. You are looking at the face, eye movements, hand movements, how the mouse and keyboard are being used, how confident the person seems to be. Hands off. Observe and think.
Sometimes you don’t have access to real users. So your options are:
- You know who the real users are, you can talk with them, they can come and test the system, give comments on functional attributes and other quality attributes, and also you can observe them when you perform usability tests with your real users as the subjects.
- You may only know the users by persona descriptions – for example for a website, the general public. You could ask people to come in and run usability tests, or you could take a usability or UX “road show” out to the public. Or you could ask a UX or Usability specialist test lab to do the assessment for you.
- You could ask the marketing team for their market research about the product and you could ask Marketing, Sales and Support team people to proxy for the customers – you could ask them to review the interfaces, prototypes and the software specifically considering the UX. They should have contact with and understand the users.
- You could ask people from your user group – if you have one – to come in to take part in UX assessments.
- You could ask users to beta test the software and specifically ask for UX and usability issues.
- You could monitor the use of the software in production and see how it is being used, when people drop out, and you can do online user surveys.
- If you cannot do any of those – if you have no contact with end users – as testers use empathy and thought experiments to put yourself in the shoes of the users. For example, there is technique called cognitive walkthrough. In a group, you work through the product at each point asking “what would a user be thinking now?” Not what you are thinking – what the user is thinking.
- The more new people you get looking at the software, the more you’ll keep hold of first reactions.
What is, in your opinion, the best way to combine these two different points of view – users who are feeling connected to the old version of product and users who really want new features and waiting for the change?
- You have a number of options that you can design into the product. For example:
- Design in “backwards compatibility” – make sure that older versions of companion software still work with the new version of your product;
- Give a choice about whether users upgrade or not;
- Give a choice about whether new features are switched on or not
- Make the new features learnable;
- Make the upgrade and first use help/encourage users to the new version;
- Do focused marketing to the customers who want the change.
Two more questions that belong together:
At which state of the project should we start to test UX solutions?
Do you think that UX should be done before a line of code is written or should it be done alongside it?
- Quality is built in throughout the project, and that’s true of UX as well. At every point, every stage, testing should be happening – by questioning, reviewing, thought experiments, what if questions, monitoring what people are actually doing with a product, as well as by running executable tests.
- Do it at the start. Build it in. But also, once the product is in use, monitor it and monitor responses to it, to get ideas for where else to improve. In fact, many of the activities that sustain, facilitate and support quality for the customer and end user are best started early. Get involved at concept stage, when your job as a tester is to question, suggest improvements, analyse risks, work out what will best serve the customer, user, your organisation, society… in this project.
What should we do if we make product designed by marketing team, before we know or understand who our customers are going to be?
- The marketing team should have done some market research, and may have drawn up personas and run focus groups as part of that research. So, your first step is to find out what the marketing team have done so far to understand the market and who the product is for. They should know the customers. Have a meeting or workshop with them to find out what is in their vision.
How can you tell when you should stop improving the UX not to go too far to avoid making the interface too unfamiliar for the user?
- “Too much” indicates that the UX is poor. Doing things to make the interface look prettier are not necessarily good UX -sometimes the pretty interface prevents the user from having a good experience… Good UX is about understanding the user, not about what the engineers want.
What’s your opinion on the role of documentation in UX? Should documentation be tested as part of product testing to ensure excellent UX?
- Yes absolutely – it is part of the product, and part of the experience that the user has. It needs to be tested to make sure it is functional, suitable, provides a good experience, is easy to navigate by someone who does not know the product. It could have errors, omissions, ambiguities – in other words bugs. And – don’t forget the buying process, the install process, the set up… they all add to or detract from UX.
Questions about the UX of test tools
These questions asked do we need skills or do we need UX? They are exciting questions…Ones I am asking myself too…
Imagine you have a labour with a hammer, he pins nails in wooden surface. If he never uses the second side of hammer to pull out wrongly pinned nails and instead pins another correctly – should we redesign hammers or teach him how to use it?
Increased demand on plane pilots didn’t resulted with decreasing requirements for wannabes.
Don’t you think that for most of tools we should demand more knowledge from new users than spend a lot of time on changing whole tool for better UX?
- These questions are really good – it is central to what I am looking at – do we make sure the testers have the skills to use the tools, or do we change the tools so the testers can use them? I don’t know the answer – I know what I am inclined to think, but the purpose of the research is to challenge that thinking… am I right or wrong? Is there a problem? If there is a problem how can it be solved? It depends…Do the tools need to change or the testers need to change? I hear lots of opinion, but I don’t know…
- If we want someone to be a carpenter, and to learn those skills, and they are working on small, slow projects, or an occasional project where a small number of nails are needed, then teaching them to use the hammer well is good.
- If someone comes up with a better design for a hammer, we’ll welcome that. And – for small home projects – maybe we’d encourage them to use a different approach – like “no more nails” [http://www.unibond.co.uk/en/diy-adhesives/no-more-nails.html]
- If we are working on a big construction project where 1000’s of nails are needed, a hammer won’t be the right tool; we need to get them a nail gun, or some other machine, and teach them to use that. And, we’ll want the nail gun to be easy to use…
- Thanks! Your question is really good one!
- If we turn to pilots, UX and usability for pilots is really important: plane crashes were happening because of human factors, usability and UX reasons [just search “usability plane crashes” to see examples]. So much work has been done in the avionics industry to make piloting a plane easier. Still, pilots need knowledge so that they can take over from the automatic pilot if something goes wrong. There has been some interesting discussion about this – how much the use of automation de-skills pilots? [see Nicholas Carr’s writing for example] That’s for a whole other blog.
- Piloting a plane – like driving a car – has changed with automation, and the usability of the automation has been key.
- This pair of questions – I love them – this is what we need to debate, research and find out the answer to.
Do you have any suggestions how to make regression tests more reliable e.g. writing unit tests for automated test cases, keep part of regression manual, etc.?
- It depends! Automating unit testing, and getting important regression tests into the unit test automation is useful. Find and fix regressions quickly. I think also understanding the content of the regression tests is useful – which parts of the system do you always want to exercise after change? Which parts of system need exercising after a specific change? Which regression tests address those questions? Which ones don’t we need to run this time? You may well find that some tests are not usefully automated at all. Understand what you are trying to achieve with the regression testing and then decide what and how to automate. Work to have smaller, focused regression testing.
What would be your recommendation with regard to configuration/maintenance costs: Compose workflow consisting of atomic tools suitable for particular activities or use ready prepared ecosystems of tools and plug-ins.
- It depends on your project and circumstances – there is not a set answer to this question. It’s an area to explore in the research – thanks!
In your opinion what is the future perspective of ‘automating the automation’ with AI, ML, deep learning, etc – how big will be the share of tasks overtaken by those mechanisms (like test design, fault reporting, test results analysis)…
- Next big thing – we’re on the brink. There will still be a place for testers, but it will change – we’ll need to be adaptable. Tariq King gave an excellent keynote about this at STARWest 2017 – it is worth listening to him…
What do you expect to achieve at the end of your research? How do you see a day when it is over?
- I hope to have answers (at least provisional answers) to some of these questions. I hope to have engaged people who provide tools into thinking about UX. And I hope to have learnt a lot and rejuvenated my thinking -challenged myself, learnt new things, fed back ideas into the industry. It’s going to take 6-8 years, and I hope to feedback results during that time.
How can each of us help with your research?
- Send me your stories – what works for you, what does not work, why you think that might be, what you would like your tools to be like. DO that via the contact page on my website isabelevans.uk – that link will take you to the contact page.
If “Should I stay or should I go” is already mentioned – what is your favourite song describing software testing?
- At the conference, I was tongue tied, here are some thoughts – you can decide what the message is for you…
- Follow the Yellow Brick Road
- These foolish things
- I can’t turn you loose
- Sweet dreams are made of this
- Right by your side
- Don’t stop thinking about tomorrow
- The gambler
- All of the day and all of the night
- Tired of waiting
- It’s nobody’s fault but mine…
- Funny how time slips by
- Five o’clock somewhere
Thanks for all the questions!