Category Archives: Uncategorized

Book review: A practical guide to Testing in DevOps (Katrina Clokie) – get it, read it!

Headline: get this book! read this book! Link:

Katrina Clokie’s book “A practical guide to Testing in DevOps” is a “must read”, and not just if you are a tester encountering DevOps. As Katrina says in the preface: “This book is for testers who want to understand DevOps and what it means for their role. It’s also for people in other roles who want to know more about how testing fits into a DevOps model.” It is engaging, informative, full of useful examples, case studies and evidence, and gives pragmatic, thoughtful guidance that encourages us – the readers – to use the information provided to craft our own approach based on our own circumstances.

I’m particularly pleased with three of Katrina’s themes:

  • this is about people as well as technology – how people engage, communicate, cooperate, solve problems, achieve goals – together;
  • separating the activity of testing from the role “tester” – the activity is needed, the role may change;
  • the purpose of testing is to contributing to our shared understanding of, and delivery of, what our stakeholders, customers, users, colleagues require from IT: a subtle and changing mix of speed, predictability, quality and a great user experience.

I kept finding points in the book that are applicable for any project, any organisation. There are useful ideas in this book if you are working in a DevOps environment, of course, but I believe you will also find it a source of useful ideas and techniques what ever your project methodology. Katrina provides chances for all of us to review our current position, and just shake it up a little, to see if we can improve. For each of us, for each organisation, team, what we need to do is different. Katrina is rightly emphatic that there is not one right answer, instead she provides aides to thinking, communicating, problem solving.

I really like the way Katrina references and quotes so many colleagues across the industry – this really helps build the feeling of a community working together, sharing ideas, not always agreeing, but debating openly and constructively.

She encourages us to try ideas and propose changes on a small, experimental scale if we or our colleagues are nervous “…spend an hour to give something new a try…”

After an introduction to the concept of DevOps, the book is divided into six main sections:

  • Testing in a DevOps Culture;
  • Testing in Development;
  • Testing in Production;
  • Testing in DevOps Environments;
  • Industry Examples;
  • Test Strategy in DevOps.

I was pulled into the book straight away. Highlights for me in the first section are:

  • visualising the test strategy – what we think we are doing, what we are actually doing… and what we should be doing. Newsflash: get the people without “tester” in their job title to do the discussions, and as the “tester” listen, facilitate… and allow the others to challenge your assumptions!
  • blazing a trail – really good ideas, hints and tips for building meaningful multi-way conversations where we listen as well as speaking… Forging the pathways between groups, teams and individuals, and reinforcing those pathways by rich rather than fast communication.

I have made so many notes and comments, just on that first section – it sparked many useful ideas, reinforced some thoughts, and provided me with new tricks, tips and tools that I shall try out.

In the second, third and fourth sections, Katrina provides examples and evidence for how testing – the activity – takes place successfully outside Testing – the team or Testing – the life cycle phase. This is such a refreshing read; my own experience since I started in IT is that good, productive, thoughtful testing can be done by people other than testers, and during activities not called testing. Developers, Ops folk, Support teams, Sales, Marketing – many people in an organisation can do good testing, often with a focus that the “Tester” may miss.  She discusses the advantages and risks of testing in development, production and DevOps environments, showing options for how risks can be mitigated, whether by use of tools, experimentation, exploration, group activities, process changes, shifting left and/or shifting right.. I’m not going to share too much here, just encourage you to buy and read the book!

The fifth section, the industry examples, is useful because it provides a wide range of approaches. Not all of us will find the same approach appropriate; something essential to success in one organisation may be damaging in another. There is a good mix of methods, tools, approaches, and types of testing covered, in this section and in the others, so you should find something to use as a starting point for making your own model.

By the time I reached the last section of the book, I had a mass of ideas, notes, and things to follow up, and I would be surprised if you did not feel the same way.

In the final section, Katrina brings everything together to help you define your own strategy for testing DevOps. Your’s will be different. It will be the strategy that is right for your team, your organisation. In this section she helps us to think about our organisation’s appetite for risk: I also have found this to be extremely useful point to consider and discuss. Are my stakeholders risk averse, or risk takers? Am I aligned with them? At this point, should I challenge or support their stance? Katrina provides useful examples of the type of question to ask in order to understand appetite for risk, identify which risks are of importance, and from that decide what activities will mitigate the risk. Those activities might include testing – the activity – but there will very likely be other activities that will mitigate the risks just as well.  From this we can draw up a strategy for testing, acknowledging that it is part of but not all of the activity required to provide our stakeholders with the quality, speed, predictability, risk mitigation and so on that is right for their circumstances.

Whether you are working waterfall, V-model, W-model, Iterative, agile or DevOps – you’ll find some useful ideas in this book.

A good ending to an excellent book, which I have no hesitation in recommending. Thank you Katrina!



Building and Testing the Tool Builders’ Workbox

In my last blog, I presented a vision of a better UX for those who use tools – a Workbox for the tool builders. How can I design, build and implement the tool builders’ Workbox? Not by myself; I need a team of people around me with different skills and knowledge, that complement my own knowledge and skills.

So far, I have identified the tasks that are needed as:

  • Research and Development (R&D) to isolate and understand out the specific problems with the current tools for the people who use them. Methods might include:
    • Interviews with people on software development teams, but also …
    • Interviews / questionnaires with people entering the industry;
    • Interviews / questionnaires with people leaving the industry, or choosing not to enter the industry when they have the capacity to do that;
    • Analysis of other research on whether the choices people make are affected by gender, culture, or other factors, based on existing research from others;
    • Data analytics;
    • Usability testing of tools;
    • Heuristic evaluations of tools;
    • AI/semantic analysis.
  • Prototyping of a new Tools User eXperience
  • Composing and trialling guidelines and templates for tools builders
  • Building and delivering education and training courses, via for example webinars
  • Awareness building through the industry via conferences (I am going to be speaking on this topic at UCAAT in Budapest in October and also will be at the Google Test Automation Conference (GTAC) this year, with a chance to discuss the ideas);
  • Trialling the prototype Workbox with tools builders.

In the first instance, it makes sense to talk with the tools builders in the open source world but I also want to converse with commercial vendors about this. It makes sense to me to start with test tools, in particular test automation, because I work in testing & quality and so know people with expertise in that area of the industry. I have called this part of the Workbox: TUXT (Tester User eXperience of Tools).

It could be wonderful. It will be wonderful.



A vision for the future: a workbox for tool builders

Software is ubiquitous, and yet so often fails to delight us.

I work in the IT industry, principally in software quality improvement, user experience and software testing. I have over 30 years of experience. I observe a problem with software: it is ubiquitous, yet fails to delight us. I observe a problem with software engineering: the tools used by engineers is hard to use. This leads to poor decision making and hence to delivery of inadequate software that delivers an inadequate user eXperience (UX).

The problem

The last few decades of experience highlight for me a conundrum: technology changes rapidly, yet the mistakes made when delivering software are repeated as each new generation enters the industry. In my last two blogs, I discussed Why everyone needs a better experience of software and Why we need a better tool set for building software. I said that technology and software are flawed. I said that the industry and its tools reflect an unconscious bias, towards a notion of a “typical software engineer” which then becomes self fulfilling.

Supposing we could change that?

The vision

My vision is to radically improve the quality of software, specifically the user experience (UX), by addressing one root cause of the problem: the engineers’ tool set.

  • The tools used by the teams who design, build and support software are essential, and could be a delight to use – if designed with UX in mind.
  • The tools could provide a beautiful UX to a wider group of people so that a diversity of people are encouraged into the industry
  • The tools are then made by “diverse software engineers” and used by “diverse software engineers”
  • The tools reinforce the use of interface designs that appeal to “everyone” because they are naturally and usefully beautiful.

If the tools are improved and modernised, then it will easier for engineers to deliver software that delivers a beautiful UX, because they themselves will experience a beautiful UX. Also, a wider range of people will want to be engineers and use the tool set.

By engineers I mean anyone working on the delivery; some may not see themselves as engineers. They may be product owners, business representatives, acceptance testers, even sales and marketing teams may contribute. In terms of job roles within the team, they may be designers, developers, testers, release teams, support teams.

All of those people contribute to the users’ experience of the software. All of those people rely on tools to provide information about the software and how it behaves. All of those people will contribute information – often into a tool – which feeds decision making about the software. Few of those people build their own tools. Many feel undervalued, unsung, and, as we saw in my earlier blog, blocked/dehumanised by the tools they use.

The workbox

My plan is to implement a project that will to help tools builders improve their tools. This will enable the engineering teams to concentrate on making software delightful, improving the UX for everyone. I want to provide a Workbox for the tool builders, that contains everything they need to build beautiful tools for the software engineers. A workbox is more than just tools:

  • a sewing workbox is used to hold everything a sewer needs: tools, materials, equipment, notions, patterns, machinery, supplies.
  • a professional’s mental workbox has tools, materials, equipment, notions, patterns, machinery, supplies, plus an attitude of mind and approaches that are versatile, flexible, strong, multipurpose, and cost effective.

As we saw in my last blog, the current tools do not help the teams provide delightful software. They are old and require the engineers to think about the tools rather than what they are making. Usually, engineers do not build their own tools, so providing tool builders with a Workbox of tools, patterns, and methods that improves UX for the engineers will enable them to deliver better solutions for their customers.

The Workbox

The Workbox for tool builders will include:

  • Tools for checking the UX of the software engineers’ tool set
  • Methods for designing, reviewing and testing the UX of the software engineers tool set
  • Notions, training materials and guidelines that explain the why and how of UX.

The UX will not just be focused on what is right for a “typical software engineer” but will also give guidance for the UX required for non-technical people using the tools. These can include managers, business representations, acceptance testers, people who are contributing to the IT project but in a non-development engineer capacity. They too need to use and understand the tools. In my next blog, I will start to form a plan for building the Workbox.

It could be wonderful. It will be wonderful.