This week, my colleague Jacqui and I have been visiting members of our cooking skills study group and discussing what we have been finding out so far from some of their course evaluation materials.

All members of the group are aiming to collect more robust evaluation materials from their courses than they might do usually.

This week we asked study group members from Edinburgh Community Food and NHS Forth Valley to tell us about the evaluation methods they had found particularly useful or easier to use.

Both agreed that taking observation notes (i.e. observing the course session and writing up observation and reflection notes afterwards) has been useful and fairly straightforward.

Observation has been useful because:

It may back up, or explain information that has been collected from self-reporting methods such as questionnaires, for example:

• it may help explain a poorly completed questionnaire if you note that someone has literacy issues
• if someone reports in the questionnaire they are eating more fruit and veg, and they spontaneously talk about what fruit and veg they are eating during the course: this helps back up their questionnaire
• If someone gives a high score in their baseline questionnaire for their level of cooking skills and in the final questionnaire their score reduces: your observation notes may help you work out if their skills have really decreased, or did they find out there was more to cooking that they first thought?

How is observation being carried out and written up?

Study group members are carrying out observation in different ways, including:

• an ‘observer’ sits in on a colleague’s course; watches, listens and take notes and doesn’t get involved in running it or helping out
• those running the course write up short notes and participant comments whilst running the course (obviously this method will be easier if there is more than one person running the course or/ and there are not too many participants
• those running the course write up notes afterwards

What makes taking observation notes easier?

Using a checklist may help construct your notes. A few study group members have used a list of their planned outcomes and indicators to structure theirs. Some of these might just require a yes/ no answer (e.g. did X taste the food?) others might require reflection notes. (e.g. ‘X would only taste the food after his mum had said how nice it was’)

What can go wrong?

Study group members have been sending us observation notes as part of their evaluation materials we will analyse. So far, the main challenges for us have been:

• There may be less information written about ‘quieter’ course participants compared to those who have more to say.
• Some notes anonymise participants to the extent that we cannot track them (i.e. notes that say ‘some participants’ ‘one participant …’ ) As we are hoping to track the progress of each individual participant, we need to be able to identify them in the notes. – It’s not surprising that this is a challenge: it can be second nature for practitioners to ensure participants anonymity. However, this can be ensured by coding participants rather than using their names.
• Finally, it’s not always easy to distinguish between spontaneous and prompted comments. For example has a participant spontaneously said ‘ I made the soup again when I got home’ ? Or have they been asked if they made the soup again? The former is observation and the latter is self –reporting’

However you choose to undertake your evaluation, makes sure you get participants permission first.

kim.newstead@nhs.net