Monday, September 28, 2009

Exclusive interview with Mark Crowther

I spoke to Mark Crowther a few months ago about his book An Introduction to Web Test Automation with Selenium & Ruby and agreed to do a review of the first few chapters. When I followed up with comments and feedback we chatted about the project and what follows is some of the main questions and answers that came up during that conversation. I really appreciate Mark's kind behavior and quite encouraging response he responded to all my questions quite patiently.

How long have you been in software testing and what’s your current role?
Just over ten years in software testing and about five in manufacturing QA and testing. My first Test Manager job was with AOL (UK) in London and I was there about 5 years or so. Since then I’ve worked at Electronic Arts and as their Head of QA. My current role is with NMQA a test solutions provider based in London where I’m Head of Testing Practice. That means I’m responsible for anyone in the business who does, thinks, talks or breaths testing.

What’s your testing philosophy?
A take a very context driven, holistic approach to the way I do testing. I rarely deliver a testing engagement or testing to a project in quite the same way. My perspective is that Waterfall, V-Model, Agile, Scrum, etc, and even artefacts such as Test Strategy’s or Bug Reports along with practices such as bug advocacy and reporting are all useful and have a place in the testing world. The way I see it a test professional has to understand all of these so they can participate fully in the test community and respond to project needs in a way that is appropriate and correct. Just saying you’re agile or only manual tester isn’t professional. You face a testing problem you need to pick the right tools.

Who are the testing professionals you admire and why?
Well, James Bach is obviously one because he was the earliest influence on me in terms of challenging me to think about what I was thinking about. Michael Bolton partly because he’s been around forever and partly because I learn something off this guy every single time I read something he’s written. James Lyndsay who teaches Rapid Testing with Michael Bolton, the material he teaches can transform a tester if they’re willing to take the material on board. Elizabeth Hendrickson as she’s a deeply technical, talks constant sense and really seems to contribute a lot to the testing community. Matt Heusser, Bret Pettichord,

What are you currently reading?

I have time for reading? Just kidding! Book wise I’m reading ‘The RSpec Book’ to learn more about that and related items such as BDD and Cucumber. Most of the time I do my reading on the forums to be honest.

In a few sentences how would you describe the book and who is the book written for?
The book is very much a hands-on user’s guide to getting set-up and working with the Selenium tools and using Ruby as the language of choice. The idea is that in a week of sitting with the book at your computer you’ve got a fully open source Selenium-Ruby based automation framework, with a set of template test examples, some neat ideas on how to manage your automation effort and you’re off using it.

The book is a kind of idiots guide or a Selenium-Ruby for Dummies?
Right, think of the book as a manual you’d expect to get if Selenium-Ruby was some kind of software package you bought off a vendor and are now sitting down to install and configure. It’s meant to be step by step with some insights and techniques that’ll help you learn to use the framework. The book assumes you know what testing is but doesn’t assume knowledge of Selenium tools or Ruby.

Can you give me some detail about the book structure and specific examples of what’s covered?
Sure, the book is currently ten chapters and just two chapters are about setting up the Selenium tools of IDE, Remote Control and Grid. We’ve got a chapter dedicated to installing the Ruby language and Ruby Gems we need, getting SciTE and EasyEclipse set-up and making sure Ant and Java are on the system correctly.

There’s a walkthrough of how to use the IDE for rapid prototyping of tests and over two chapters a detailed look at using Grid in various ways. In another chapter we do a code walkthrough of a Selenium-Ruby template script looking at each piece of the script and why it’s there. The book also has a chapter dedicated to creating ten template scripts that cover common testing challenges such as doing cross browser and data driven tests, testing Flash, dynamic content, handling pop-ups and Ajax and also capturing screen shots.

OK, so what would you say the book isn’t? What will a reader not find in there?
It’s not a file by file technical dissection of the tools themselves, I don’t go through scripts and files parsing out the lines of code and looking at how and why it’s programmed in a certain way. There are small amounts of this where relevant to setting up the framework, for example where we look at configurations appearing in the Grid Hub page, but I’m not cracking open jar files just for the technical fun of it.

What new things are in the book that I can’t find elsewhere?
There are three sections within various chapters that are about practices I’ve either developed or employed. They’re related to Test Case Trees, Environment Management and using Pivotal Tracker. I’ve added these to give readers some extra support around their new automation framework. Given there’s no book focused on Selenium-Ruby the whole book is kind of unique and add in it contains ‘fixes’ for common configuration issues there’s a lot there for the tester trying to quickly get the framework in place and move on to doing testing.

Why did you choose to combine Selenium with Ruby? Why not choose Java for example?
I wanted to create a framework based entirely on open source technology and with components that are easy to learn and use. As a tester I don’t consider Java that easy to use, Ruby is easy to read, learn and use and it’s far prettier and efficient, at least in my view!

How much Ruby is there in the book? Does a reader need to be a Ruby guru?
In terms of complexity of coding and use of the language there’s not that much. There’s just enough to complete the templates plus a few neat tricks to show how Ruby can help us in our testing. This isn’t a Ruby programming book, remember the book is a intended to get all the components in place and the reader delivering testing it’s not a Selenium architecture or Ruby course.

Isn’t there already a book or website about Selenium automation?
You’d think so wouldn’t you? In fact there’s not the expected raft of Selenium books out there that you’d expect. If you look over Barnes & Noble, Amazon, etc. there appears to be exactly one about Selenium with Twill. There’s nothing like the book I’ve written as far as I can tell.

What’s been the most difficult for you when writing and researching the book material?
Writing’s been fairly easy, sure it takes forever, but it’s not hard. I’ve also implemented everything I’ve written on a number of client sites and in my test team at NMQA which while sometimes tricky was the real fun part. It helped prove out everything I’ve written too. I’d say the hard or tiring bit has been the research and the disappointment of not getting answers to questions I’ve asked on forums, etc. It just disappointing to put the effort in to work bits out yourself, search for answers online or with colleagues and post questions thinking it’ll be fun to get talking to folks and end up with no replies.

Final question, when can we get a copy and from where?
I’m hunting for a publisher now, the manuscript is ready subject to review and no doubt some rewrites that will be required. I’ll have to ask folks to wait a few months for the publisher to give the book the nod and start making it available. If I get enough knock-backs, as in no interest, I’ll just self-publish via Lulu ready for November or December latest.

Final thoughts before we finish?

You know, this book is really about helping other testers like myself avoid the issues I encountered that might stop them adopting Selenium tools, that might mean they never get to learn about them and use them. I hope folks will enjoy the book and get excited about Selenium and Ruby, I also hope that the Java, C#, Python, etc. Selenium community out there will ‘translate’ the book into examples that apply to their chosen languages.

This interview is reproduced with permission of the author and is copyright Kashif Ali (C) 2009. It is prohibited to reproduce this interview in any form without written permission from the copyright holder. All rights reserved.

Tuesday, September 15, 2009

Is it possible to find all error in the application?

Software testing is an integral phase of software testing, testers are the people who always question the reliability and stability of the software application, considering the role of software testing, it is also claimed that it is not possible to find all errors in the application. Let’s use basic testing types in order to start discussion about this statement.

Black Box Testing
Lets explore Black testing type, this testing type is also called Data Driven testing, Input/Output testing. As it quite known in this type of testing, testers are only concerned with the functionality of testing, and treat system as a black box, they derive their test cases from the functional behavior and least bothered about the internal structure of the application. So if we want to find all errors in the application using black box testing , then we have to use to technique of exhaustive input. That is we have to cater each possible input to the application.
Testing software with each possible input, sounds impossible because it is impossible to measure all possible valid or invalid input condition for example consider a program, which takes any two numbers for addition, now if we analyze the input to that program, it can be from 0 to infinity, in the same way negatives infinity to zero. All of us is agreed that it is the most simple computer program, but still in order to test program with each possible input condition is not possible. Consider a program like Compilers, operation system, or flight control system in which you have to derive test case for each possible input, what do you think it is possible?
So testing software with all exhaustive input is not possible, so we can test the program with every input to the program, thus claiming that software is error free for all possible input can not be feasible and practically impossible task to do,
But yes we can use different techniques to build our confidence about the stability of the program by using test case design technique that is (Equivalence partition, boundary value analysis etc and the objective should be to maximize the yield on software testing investment by finding maximum number of errors in definite number of test cases.

White Box testing
Another testing technique, white box testing also know as logic driven testing, in this type of testing test cases are derived from the internal code logic of the program, that by considering all logical paths, testers design test cases, in this type of testing all logical path are tested to find maximum errors, so as we discussed in black box testing we test program with exhaustive input, in this scenario we will say it as exhaustive logic path testing.
Testing software by designing test cases that cover all logic paths to ensure that software is error free, is quite impossible because we have to go through all possible paths of control flow in order to test software 100 % accurately, when we talk about testing each logical control path, consider a program which runs almost 50 line of code within loop, and for every iteration of loop, it passes through different branching conditions, so writing test case for every nested statement, seems impractical, if not impossible.
Considering if we are somehow able to test each control flow path of the program, but still it is not sure that software will be error free, due to following reasons:
1-If we test all logical code paths, it will not ensure that code has been written according to specification provided. For example if programmer writes ascending order routine instead of descending order, by testing all logical paths, we can not guarantee that there will be no error in the application.
2-There can be error due to missing input paths, because exhaustive input can not determine important logical paths.
3-Exhaustive input testing does not cover data sensitivity errors.
Looking for your feedback :)

Tuesday, September 8, 2009

Interview with Mark Crowther

I spoke to Mark Crowther a few months ago about his book An Introduction to Web Test Automation with Selenium & Ruby and agreed to do a review of the first few chapters. we had a dicussion on different topics of the book, i compiled the conversation in the form of an interview, it contains interesting information about mark's book. and mark's passion about software testing so be ready for "Mark Crowther exclusive interview :)

Friday, September 4, 2009

Simplify your bug report

Communication between developers and testers plays quite important rule, often we see a debate going on about the bugs between developer and tester, there are few Things that makes programmers to resist spending their time on the bug that are:

• The programmer can’t replicate the defect.
• Strange and complex set of steps required to induce the failure.
• Not enough information to know what steps are required and it will take a lot of work to figure them out.
• The programmer doesn’t understand the bug report.

The point of testing is to find bugs. Bug reports are your primary work product. This is what people outside of the testing group will most notice and most remember of your work. Programmers operate under time constraints and competing priorities. A bug report is a tool that you use to sell the programmer on the idea of spending her time and energy to fix a bug.

How to simplify your bug report
1-Write each step in detail.
2-If you see two failures, write two reports combining failures on one report creates problems: 3-You’ll often see one bug get fixed but not the other.
The summary description should not be vagued. The words like “fails” or “doesn’t work” should not be used instead of describing the failure more vividly. This weakens the Impact of the summary.
4-Put Variations after the Main Report Suppose that the failure looks different under slightly different circumstances. For example, the timing changes if you do a additional two sub-tasks before hitting the final reproduction step. This is all useful information for the programmer and you should include it. But to make the report clear.
5-Write suggested solution if possible.
6-Write expected Benefits to the customers and vendors

you can read the detailed article on bug advocacy from here