Context-driven test thoughts

Writing your own test framework with a DSTL in java?

2011-03-24T13:13:00.000-07:00

In my current assignment I´m working on writing a non-functional test framework.

I´ve been pushing very hard for implementing and supporting a DSTL (Domain Specific Test Language) since it´s my belief that in doing so we can have more testers focus on writing actual test cases without having to worry about the implementation.

Also I´ve been itching to get an excuse to implement my own DSTL for a while ;-)

A test case could look something like the example below and can be written in either a text file or a spreadsheet (or anything else for that matter, wiki page etc etc as long as you convert it to a format the server understands).

Configure simulator | on sim1 | using traffic_type1

Start simulator | on sim1 | with 800 calls/second

Start CPUProbe | on $[sut1]

wait | 20min

The first column/cell is the actual test command and the following columns/cells are parameters to the command.

In the text file we have currently chosen to use a pipe-sign for clarity.

Notice how the test commands reads out as almost normal human sentences.

A clear benefit of this approach is that the test cases have a higher chance of being:

- Clear and obvious, even to other stakeholders

- Robust and requiring less maintenance, since we separate test purpose from implementation

- Lends itself naturally to becoming data-driven

As you can see in the "Start CPUProbe" I have actually taken one of the parameters and indicated (with the dollarsign) that this should be replaced with data from some external source, like another data file, database or perhaps some wiki-page.

Now when you send a test case like that into the test server you have to map the test command to some function/code that is going to be executed.

This weekend I had some time to think on how I could solve this in an elegant way, and I really didn´t wan´t beeing stuck in an endless if/else or switch case story.

I wanted seomthing that would later be easy to use for dynamically adding new test commands/plugins in runtime.

The server is written in java and so what I came up with was defining a hash map with the test command as the key and then the fully qualified classname as the data.

Then I could use that hash map when a test command was sent to the server to use a dynamic class loader and instantiate a test command object.

The requirement for the test command class is that it is a child class of "TestCommand" which through an interface specifies that the Run()-method has to be implemented.

Here is an example of the code I used for the solution.

public class TestClass

{

private Hashmap m_cmdClass;

public Testclass()

{

m_cmdClass = new Hashmap();

m_cmdClass.put("Start simulator", "my.server.testcommands.StartSimulatorCommand");

}

public TestCommand getTestCommand(String aTestCmd, String aTestParamsBuffer)

{

TestCommand testcommand = null;

ClassLoader classLoader = Thisclass.class.getClassLoader();

String fullClassName = m_commandAndClasses.get(aTestCmd);

Class aClass = classLoader.loadClass(fullClassName);

testcommand = (TestCommand)aClass.newInstance();

testCommand.loadParameters(aTestParamsBuffer);

}

return testcommand;

}

So this code is just an example of how you can use a dynamic class loader and a hash map to connect a test command to an actual class.

This production code looks a bit different and there is error handling as well, but this should give you some ideas what you could do.

Recomended reading: "The Effort Effect"

2010-12-18T04:55:00.000-08:00

Just read an article that made me remember reading James Bach:s (@jamesmarcusbach) book "Secrets of a Buccaneer-Scholar", and it´s really interesting.

The article is called "The effort effect" and goes into explaining the different kinds of people that either think that you are born with a natural talent and you don´t need to put in more effort (or it´s just not worth it) as opposed to those that realize that we can learn and be better at stuff if we apply effort into that area (grossly simplified).

Well worth the read.

http://www.stanfordalumni.org/news/magazine/2007/marapr/features/dweck.html

Positive feedback in mail, negativ face-to-face

2010-12-18T03:22:00.000-08:00

Yesterday I made a mistake, and yes it happens for me as well sometimes...

I got to emotionally invested in a retrospective that I had just conducted with my new team.

Since I was "the scrum guy" I was asked to explain how I usually do my retrospectives.

So I layed down some basic ground rules.

- I always divide my retrospectives into a couple of phases.

1. What worked less well?

2. What worked well?

3. What can we improve in the next sprint?

- Also every phase is started with ~5 minutes of solitary thinking and writing on post-its

This is because some people are more inclined to taking "the back-seat" and let others do the talking when the others start.

Also I might have some stray thoughts on some matter but when others start to talk about something else it´s easy to forget that line of thought and start thinking about what the others are currently talking about.

Finally it´s a very nice indicator of the importance of something if several people thought about it independently of each other.

- The retrospective is a way for us to improve in the comming sprints

This is not something new, we continously inspect, adapt and improve, and thats one of the basis for having a retrospective.

- It also serves as a "venting point", a place were we can "cleanse ourselfs" or just let of some steam

Sometimes people just needs to talk a bit about the problems/people they´ve encountered during this sprint, but we don´t always need to act upon it.

For me its a very cleansing feeling, a catarsis (greek word meening "cleansing" or "purging"), where I can say what I want in that room and then leave that behind as I walk out the doors

of that meeting room.

- Anything can be said, and if we want some comment or note to stay in the room thats ok, we don´t always have to communicate everything we say.

So the reason why I divide my retrospective into the phases is because it´s easier to focus on the negative stuff at one time and the positive at another time.

Kind of like going into a certain mind-set, the most popular theory about this is the "DeBono:s Six thinking Hats".

As this was a new project and we made some great progress, and since the mission and requirements were really unclear we had a lot of stuff to vent and write down.

Some we chose not to write down, most we wrote down and then we decided upon 2 things to improve for the next sprint.

However since we were "in that mode" and didn´t quite think things through I missed upon one thing.

This project is a central project that covers multiple nodes/departments and we have designated representatives for all of departments, they are supposed to talk for the departments and take

stuff home to discuss with other people.

Since this project is new and has been a bit unclear from the start the representatives haven´t always had time to invest in this.

One of the representative is a colluege of mine that I´ve been working with for some time.

It´s one of the persons that I have the highest respect for.

The reason I have this respect for him is that he is enthusiastic, driven, experienced, lots and lots of technical and testing knowledge, and also thinks a lot like me (the best indicator for a great person ;-) ).

As you might have guessed a person like this has a lot of other commitments at the same time and therefore the attention for this project has slipped a bit.

He is not the only representative where this has happened, and since we felt that this project is important and that we really cn make a difference we wanted to point out this out.

We mentioned that the representatvies from certain nodes were really well suited becuse of the background and experience for this task and that we´d prefer to continue working with them,

but if it was hard to find time and focus for this project perhaps some other person could take the job instead.

We sent our all our feedback out in mail summarising all our findings and went on our business feeling very relaxed after a succesfull retrospective.

However the person I have mentioned above became very upset when he received this feedback in an email and felt very afronted by this public critisism.

He confronted me afterwards and had a talk with some very heated emotions eminating from him and I quickly realised exactly what we had done.

We had confused the role and the individual behind the role.

We felt unconsiouslty that we only critisised the role and not the individual. However anyone can see that that´s not how it is going to be received.

The even stranger thing is that I have on previous scrum teams where I have been a scrum master said that we give positive feedback in emails (preferably CC the persons manager as well).

But negative feedback we should take face-to-face ot avoid confusion and also avoid exposing the person to other people.

I have apoligised to this person and explained the situation to him and he has said that in the future he wants this kind of feedback feedback but delivered in person instead.

Also we have agreed to put this behind us and forget it.

Still ... I feel a bit ashamed since I should have known better, but I got to emotionaly invested in the project and the current team, and didn´t quite think through what and how we

communicated our retorspective findings.

So a lesson to be remembered is: Positive feedback in email (and in person), negative feedback face-to-face.

Recomended reading: "Time To Look Seriously At Software Developer Apprenticeships"

2010-12-16T07:04:00.000-08:00

I just saw a tip at twitter from Markus Gärtner (@mgaertne) about an absolutly fantastic blog-entry.

Well worth your time to read and should be equally applicable towards software testing.

"Time To Look Seriously At Software Developer Apprenticeships"

http://parlezuml.com/blog/?postid=977

Trinity Testing - The power of 3

2010-12-12T11:04:00.000-08:00

Last year at Eurostar (2009) I saw a really fascinating presentation done by Julian Harty from Google (since then he has moved on to ebay) about something called "Trinity testing".

These are the kind of ideas that I really like, where the focus is to put the right people together and make them start talking while the information is fresh and relevant.

Some of the material in this post is drawn from material that Julian was kind enough to send me after last years Eurostar.

Let me explain about trinity testing...

The motivation behind this idea is that even great requirements don´t communicate all that we need.

- There are few great requirements... and usually there are few requirements at that.

- So people have to guess the desired behaviour of features and fixes.

- This leads to software that is flawed in ways that we don´t even know!

Also there is the problem of latency.

- Features are bundled into releases

- Developers finish code in between releases

- Testers are expected to test releases

In the meantime the developer has moved on to the next feature.

When someone is working an idea, in this case developer working on a new feature, the knowledge or focus on that particular area is high.

Therefor if you were to ask her something about that code it would be easy for her to remember the correct details and explain them for you.

However ... as time progress and that developer starts to work on other features/bug fixes etc. and that knowledge is put further back in the mind and the attention of the developer is

on the new things that she is working on, she also has an incentive and interrest to work on the new thing currently occupying her mind (this being planned for the next release) and it takes time to remember the details.

Also during this time the needs might have changed and more changes might be required.

To often the normal scenario is just this.

A developer finishes with the code and submits it, and after the feature has been packaged into a release a tester normally picks up the release for testing.

Hopefully the tester has had time to do some preparations beforehand at their desk, separate from developers and the people writing the requirements of couse, and some time elapses, bugs

are most often found.

Now during this time the developer in question has started the work on a new feature/bug and has her mind, focus and attention on this new work.

When a new defect arrives this forces her to do a context switch and also affects her current work.

The longer the time that has elapsed since the developer left her work behind, the bigger the impact when she does this switch.

Also a not uncommon scenario arises here and the defect that has been found is not clearly breaking any requirements, but it has enough characteristics for the tester to be classified as a bug.

Now a discussion starts between the two parties regarding the intended behaviour of the software and sometimes a third party has to be consulted, usually a system specialist, requirement analyst or system architect.

This increases the overhead required for all parties to start communicating.

So the general idea of trinity testing is that when the work is being started, a developer, a tester and a "domain expert/analyst" sits down on a brief session (30-90 minutes) and talks the feature through, and this is when the concept is being concrete.

The participants can vary from session to session depending on the information needed.

All the parties have different roles, responsibilities and things to prepare up-front.

- The developer explains the features implementation details and will learn during this session the intention of the app, how it is going to be tested, how it might be broken and test automation ideas

- The tester explains about fault models, concerns that he might have, how he intend to test and if there are test automation ideas and learns about the intentions of the app (what it should do) and the implementation details (what it is actually doing)

- The domain expert understands the application in depth and what the application should do and learns about the implementation details

The idea here is that the session should not be a long painful meeting with lots of stakeholders, just a representative of the people that are going to do the job.

Sometimes the developer isn´t the sole person that is going to do the work and likewise for the tester, in this case that person acts as a representative for the group in order to decrease the number of people.

Also the domain expert can be different people depending on the area that needs to be covered.

The three persons exchange information, asks questions, communicate and then go home and do the work.

People can even do pair work afterwards (pair exploratory testing with the developer for instance) if they feel brave and adventurous.

Everything is fresh in their mind and this is the current task at hand.

- The analyst/domain expert works at home with the customer, defines features etc.

- The developer fixes issues found at the session, implements the feature etc.

- The tester tests in more depth and decides what should be automated if appropriate, finds defects etc.

After everyone is finished with their work they get together again and exchange information in a new session.

- How did the developer implement the code?

- Did the analyst uncover new information from the customer?

- What did the tester find during his test sessions?

Sometimes if the work that is being done is of the larger magnitude it can be necessary to have sessions in the middle as well.

So what are your thoughts on this concept, anything you already tried at your work or feel like trying out, or just don´t agree?

New inspiration

2010-10-12T12:36:00.000-07:00

Today I saw a webinar (‘Do we need Labels? Are we all not ‘Testers’?) at work presented by Henrik Andersson.

He talked about the tendency for testers to label themselfs and put themselfs into boxes (agile testers, exploratory testers, telecom testers, senior tester etc etc.)

In his view, the tendency to label yourself is amongst other things often based on the need to cover up your lack of testing skills by showing of a fancy title.

His wishes are that we instead of labeling ourselfs let our work and reputation become what recruiters look at when hiring instead of what title you have.

The means to that end is to take control over your own test career and do things that let you grow as a tester (blog, be active in forums, read articles, weekend testing etc.)

As that was my intention when I started my blog here it got me inspired to continue with it.

Since the last time I wrote in this blog I have been on paternity leave and wonderful as that has been it kind of distracts you and let you think of other stuff than testing.

But since I´m back at work and currently at a very stimulating assignement at a systems test department getting to do all kinds of fun stuff like non-functional test automation, version handling of our test data etc. I felt it was time to get back to the blog.

So welcome back and enjoy your reading.

Just fininshed reading "Weinberg: Perfect Software..."

2010-01-02T04:53:00.000-08:00

As the title says I just finished reading Weinbergs book "Perfect Software and other illusions about testing".

The book was easy to read and had lots of examples and anecdotes from his and other persons life making it interesting and rich and not dry as these kind of books easily can be .

He stated in the introduction that this book was written to enlighten people about testing (managers,developers, testers and more) and that a good tester would recognize and know about a lot of the things described in the chapters.

The first chapters did indeed contain lots of material that was familiar and that I have experienced over the years, but this just got me more inspired to recommend this book to all developers/managers that I know of (test enlightenment crusade).

It really addresses a lot of the questions that typically arises in a development organisation (why can´t we test everything?, Why bother testing?) and answers them in a very clear manner (answers you wished you had when sitting in that meeting with a developer or manager that just did not understand).

Another interesting aspect of the book is that Weinberg goes into the psyche of people and how we react to information (Information immunity, defensive reactions etc).

The last chapters really got interesting when discussing making meaning of information and determining the significance of it.

I can really recommend this book to anyone who´s interested in software testing or just wants to get a better insight into the subject.

Now which book to read next, J.Bach:s "Secrets of a buccaneer scholar" or another Weinberg book "Becoming a technical leader"?

New assignement - going native?

2009-12-28T14:01:00.000-08:00

Come January I´m taking a break from my normal assignment for parental leave.

I´ll be gone for half a year together with my son and it will be most interesting to put it mildly.

But for two weeks in January I got a quick assignment doing a job for a large mobile operator.

The work will consist of me doing an analysis of their current test situation in an business area and possible recomendations.

I will mostly be looking on the testing done by the developers and test engineers situated in the development teams.

In other words the unit, functional and regression testing.

I still wonder how my company managed to get me this assignment but I am very enthusiastic going into this, but also very nervous.

I´ve never done this kind of assignment before so I will have to find how I should approach the problem at hand.

One tip I´ve got is from my wife is: "Living with the natives" or "Going native"

She´s an academic in the social sciences and the approach mentioned above is used in ethnography when gathering material for research.

Basically one goes into the organisation and does the same work as the employees (a.k.a "The natives").

The idea being that it will be giving me a better insight into their work, daily lives and problems.

This approach is very suitable for me since I have worked as a tester, developer, test lead etc.

The question is if there will be enough time to do this over the course of only two weeks... ?

This assignment has really gotten me excited as it will give me the opportunity to use all my experience from testing to development and further on seeing the "whole picture" and how the organisation works, but most importantly to work with testing.

Just hope I don't bite of more than I can chew.

My very own book account

2009-12-18T12:28:00.001-08:00

At my company I have some good benfits, but one I praise especially high is my very own personal book account.

Yes you read it right ... my own book acount.

I can order books for 2000 swedish crowns (around 275$) of my liking every year.

The only attachement is that it has to be somehow related to mywork/career (software development, testing, personal development etc.).

The other day I got the latest batch of books I ordered.

After all a new year is approaching and I had money left to spend on my account ;-)

I ordered J. Bach's new book "Secrets of a buccaneer-scholar", G M. Weinberg's "Becoming a technical leader" and "Perfect Software, and other illusions about testing".

I´ve especially been wanting to read the last one (Perfect Software...) for quite a while now and started reading it tonight.

I have high expectations for it since I´ve heard lots of good feedback on it.

Anyone that have read it and care to share their thoughts on it or any of the other books?

Also as a note i think that the concept of your very own book account is something that every company should adopt if they want to encourage their employees to evolve and feel appreciated at work.

Just think about it, my company has ~50 employees and the total cost per year will be ~13.750$ (plus I´m sure lots of tax rules etc that makes it deductible and what not).

This is practically nothing when we are talking about company budgets of that size.

On the other hand it is a very luxurious privilege and something that makes me feel very appreciated and important.

The company on the other hand gets motivated people who hopefully will become better more well informed engineers.

Oh and by the way, I´m hoping that I will get back on my thoughts on "Perfect Software" and possible ideas and inspiration I might get from it.

Mind the gap - and how we bridge it

2009-12-12T02:59:00.001-08:00

This post is growing into a large one and I will divide it in two parts.

One thing that has struck me through the years are how divided we are.

Testers Vs. Developers ... and all the prejudices that exists between the two groups.

This was an issue that was raised several times at Eurostars by several speakers and also on the panel session in the last day of the conference.

This has been especially clear to me since I routinenly move between the two camps being a so called "test developer".

On one hand I love coding and the craftmanship involved and on the other hand I love testing and the whole intellectual challenge in that process.

This text will be a bit stereotypical and exagerated, but all in order to show my point.

But on the whole I´ll bet you will have seen the things I mention here more or less in your daily work regardless if you´re a tester, developer or something in between.

But is it really as black and white as testers vs. developers?

No in between we actually have a few other groups that can bridge the gap and act as ambassadors between the two groups.

I´ve already mentioned test developers and theres also testers who have coded in the past.

Of course there also exists the group of developers who have tested in the beginning of their career, although these tend to have seen testing only as a necessary evil in order for them to get a foothold into the company to do "real coding" instead.

The other day I thought of another group that exists, the so called Design&Maintenance developers, in other word when you're on permanent defect fixing duty.

The work involed in defect fixing actually has lots of elements that resembles exploratory testing.

The developer looks for the reported fault, gathers data/information, learns about the product/feature, tests theories, gathers more information and so on, all in a continuous feedback cycle with the aim of learning enough of the problem to be able to fix it properly.

Also when we fix a defect we have to be on the lookout for other sideeffects or symptoms of this defect that we might not have considered and that also have to be taken care of.

Kaner decribes testing like CSI in that we have tools and procedures, but they don't define the investigation, that there is to much information to process everything and the investigator must select what to investigate and what not to.

This is also very applicable to the process of fixing a defect for a developer.

It would be very interesting to see if D&M developers would benefit from a class in exploratory testing (or a class aimed towards developers: "Exploratory Defect Fixing") and what their thoughts would be on it.

I might be tempted in the future to give a presentation at my company (which is heavily developer-centric, I´m the only one with a testing background) and see what the developers think of exploratory testing.

In the next part of this post I will try to discuss different ways that can be used to make the gap smaller between testers and developers.

Upcoming webinar with James Whittaker

2009-12-08T23:49:00.000-08:00

Tomorrow 10/12 there will be a open webinar held by James Whittaker.
The topic will cover Exploratory testing done at Google.

"I'll be giving a talk about how we do testing at Google, specifically related to an update on our use of exploratory testing that I introduced at STAR East."

http://www.testingreflections.com/node/view/8386

Could be interresting and I'll try to catch tomorrow.

Regroup - Rethink

2009-12-08T04:45:00.001-08:00

When you work on a problem, sometimes you become stuck in a rut.
Sounds familiar?
You sit with the same problem trying the same approach over and over again, hopefully with small variations (read: banging your forehead at the monitor from slightly different angles).

This is when you can be greatly helped by taking a step back and rethink your approach.

You can for instance change the scope of your focus for the current problem.

Perhaps you have been visualising the problem on a too low-level to see the full picture properly.

Or perhaps you have had a too wide approach and not quite gotten started at all because of the sheer size of it all. If so it is perhaps time to categorize and take a closer look at the different parts of the problem.

You can also change the technique you are using altogether:

"Ok, so I can´t break the product using boundary testing, perhaps if I use special characters instead that will reveal some new information for me". Or:

"I´ve been debugging this code from the end of the system and what comes out of it, lets take a look at the beginning and what input is sent into the system".

This is something James Bach calls "Focusing and De-focusing", in other words changing the way we are looking at the problem at hand.

He teaches this technique in his class for "Rapid software testing" and I can full heartedly recommend you take the class if you get the opportunity to.

This technique might sound easy enough, but the real trick lies in practising this often enough so that it becomes a habit and we don´t even have to think about it when we get stuck with a problem.

In the beginning I found myself (and still do at quite a lot of times) working on a problem like debugging code hours on end using the same technique (read: banging my head on the monitor) and suddenly realised I got nowhere and had to stop to change my approach, very oftenly this quickly lead to pleasant results.

Sometimes it isn´t even very important what change you do, just that you take a new approach and through that process get new ideas and information for the problem at hand.

This might not sound like much new under the sun, but you´d be suprised how often you find yourself stuck with a problem using the same approach over and over again, leading to frustration and draining of energy and attention.

The importance of communication skills for a tester

2009-12-06T13:19:00.001-08:00

Why is it of any importance how I communicate in my role as a tester?

I just publish the latest defect I found and its off for fixing or rejection ... ???

Wrong!

It is my belief that the first and foremost important mission for us testers is providing information to the project.

This information will contain details such as, what defects we found, what potential threats and risks we see, how much of the product have we covered with our testing and a number other factors.

If we do this with confidence and integrity we may even over time gain the status of "trusted advisor" to the project leader, but in order for us to achieve that it is a requirement (among others) that we have the skills necessary to communicate our message.

Coming into the office of the project leader screaming and shouting about the latest defect that got rejected at the review board will most often not get us much and it will definitely not work in our favor in the long run.

Communcation skills (or even Social skills) is not limited to verbal communication face to face but also communication through different mediums such as email, telephone etc.

All different forms of communication has its own strengths and weaknesses and it is important to choose the one suitable for the current context.

Sometimes it will be necessary walk over to the developer and sweet talk him into looking at your defect and giving it a second thought.

Other times you might have to email the developer and also cc her team leader for that extra punch (don't misuse this though ...)

Other times yet speed will be of the essence and you just have to pick up the phone and talk to the project leader for escalation of your defect (it can be a show-stopper after all).

All these examples call for different kinds of communication and it is up to you to be able to master them all.

When posting a defect it represents different things for different stakeholders.

The project leader might see a risk in delaying the project, the developer might see critique of his or her code (aka. their "baby").

Just simply posting the defect will not do.

We must first document and post the defect in a clear manner that is easily understood by different parties.

It must be general enough for non-developer to estimate its severeness.

It must also be technical enough for the developer to easily dig into it and reproduce it.

After this the defect might come into question in the review board and we might have to either defend it ourselfs in front of the board or we might have had to prepare our test lead with the necessary details so they can do that on our behalf.

Finally the developer might question our defect (when has that ever happened?) and we will have to walk over there and discuss it with them.

All this above is part of something called "bug advocacy" ("Lessons learned in Software Testing", Kaner/Bach/Petichord) and its a vital skill for all testers in my opinion and is helped by good communication skills.

To summarize I beleive it is our duty for the project and the stakeholders to learn how to communicate our message or we will not be doing our job to our fullest potential.

Home from conference - fresh start

2009-12-06T12:21:00.000-08:00

Earlier this week I got back from Eurostar 2009, tired but happy and so so inspired.

Let me introduce myself first.

I´m a so called test developer that got introduced to testing in 2005 something, and I love testing, coding and above all combining the two.

During the years I've been working with development, agile coaching, testing, test automation and so on.

I discovered pretty soon the joy of testing and all things around it and first and foremost the challenge for the mind that it introduces.

During the last year or so I have worked as a developer consultant and been in kind of a rut, this due to the fact that the former company where I was employed went into bankrupcy.

Don´t get me wrong I like coding as much as the next guy, but what really reminded me of how much I miss the testing community was getting the opportunity to go to Eurostar again.

There I had the great pleasure of seeing such big personalties as Michael Bolton, Lee Copeland, Susan Windsor, Thomas Graham and many many more.

Afterwards when I was on my way home from the conference I started thinking about what I should do with all this newfound energy and inspiration?

One of the things I decided upon was to start a test-blog to start getting my thoughts and ideas down into printing.

The concept of writing down my thoughts and speculation is a bit scary for me as I have always seen myself as a person who is more at home with a perl-script than putting down stuff into writing for my peers to scrutinize and potentially criticize.

So with that I declare the blog "Context-driven thoughts" open.

Oh and if you are wondering I'm a big believer in the Context-Driven School of Software Testing