Focus groups and survey questions

[Sonya Norris]

I just finished a user study of the extranet I maintain for seven
libraries. I used two types of testing: standard, over-the-shoulder task
monitoring of 5 users in my building, and an on-line survey from a
company called Zoomerang for staff at all seven libraries.

I built out the questions for both types of testing and tried them out
on staff members before they went to the whole group. I also got
feedback from our resident survey expert to make sure the on-line
questions were formatted properly and would not be confusing. I found
the testing extremely useful and I was quite surprised by the results. 

For the on-line survey, I asked card-sorting type questions (where you
give users a list of links or pages, usually on 3 X 5 cards, and ask
them to sort them into the groups that make the most sense to them) to
help me rearrange the navigation structure, questions about how they
used the site, how often, etc. I asked questions that I thought I knew
the answers to only to be really surprised by the responses. For
instance, I was so sure that no one was paying attention to the "What's
New" section that I didn't even advertise the user testing survey there!
I also asked about how often bibliographies were being printed out for
patrons -  in my mind one of the key reasons for the site - only to find
that this was rarely being done (I was able to add a "printer friendly"
button that has increased the frequency with which this is happening
now).

For the over-the-shoulder testing, I used candy bar incentives and
tried to make people as comfortable as possible. I encouraged them to
talk throughout the testing, to voice what was on their minds as they
tested so I could hear them thinking about the navigation and reasoning
out what worked and what did not for them. I wrote down step by step
what they did to fulfill the tasks.

Here is a link to the results of the on-line survey, 
http://students.lisp.wayne.edu/~ar1314/zoomerang/Zoomerang.htm

and here is a link to the over-the-shoulder task monitoring that I did
http://students.lisp.wayne.edu/~ar1314/Presentation/OverTheShoulderResults.htm

You can see the front pages of the "old" and "new" versions by
visiting
http://students.lisp.wayne.edu/~ar1314/informationarchitecture.htm

but the site itself is behind a firewall.

The final report for the project is not on-line unfortunately, but you
can get a sense of the questions I used and how I monitored results.
Feel free to contact me if you need more information. Good luck!

Sonya Norris
Library of Michigan

Date: Wed, 15 Sep 2004 13:47:22 -0400
From: "MacKenzie Stewart" <mstewart at wellesley.edu>
To: web4lib at webjunction.org 
Subject: Focus groups and survey questions
Message-ID:
<fc.006640d80cd28332006640d80cd28332.cd28be1 at firstclass.wellesley.edu>

Dear Web4Lib readers - 

We are about to begin an evaluation of our Library web site. This Fall
,
we'll be working on ways to obtain useful information, including focus
groups and surveys. We are looking for feedback in regard to
usability,
content and navigation specifically. 

Have others done anything similar ? Did you receive useful feedback ?

Thanks,
Mac