Counselling, Research in the School

Researching psychotherapy using questionnaires

What methodologies are appropriate in the development of new questionnaires in psychotherapy? Dr Gina Di Malta from the School of Psychology and Counselling is lead author of a new article that introduces an in-depth qualitative method called the Three-Step Test Interview (TSTI) in order to refine a psychometric test that can be used in large-scale studies of therapeutic relationships.  

Scale development methodologies tend to emphasise the use of statistical analysis to establish reliability and validity in scales. These methods are indeed necessary, yet it may be the case that in-depth qualitative methods are equally important to support and guide the refinement of scale items. 

Three-Step Test Interview (TSTI) is a method of cognitive pretesting, which was piloted in the development of new scales and showed that it could help identify problems resulting from a mismatch between theory and participants’ understanding of items. Such a tool is useful in that it can extract and organise in-depth feedback on scales and their items. The method enables localising problems in the response process, their effects as well as their causes. 

In the context of the validation of a psychotherapy scale, the TSTI was particularly useful in highlighting subjective content that could impact on the reliability of the scale, as well as potential ethical issues associated with observations of participant’s distress when responding to some of the scale items. Its application produced knowledge on the scale that could not be captured through statistical methods alone. 

Gina Di Malta, Mick Cooper, Joel Vosand Kees van der Veer(2020) An Application of the Three-Step Test-Interview (TSTI) in the Validation of the Relational Depth Frequency Scale Journal of Humanistic Psychology


The main objective of this study is to evaluate the utility of a new qualitative scale development methodology—Three-Step Test-Interview (TSTI)—in its first application in the validation of a psychotherapy scale: The Relational Depth Frequency Scale (RDFS). The TSTI is a cognitive pretesting method designed to uncover potential problems in scale construction. The RDFS is a six-item unidimensional scale of in-depth therapeutic relating, designed for use in large-scale outcome studies. Following the creation of an item pool and “expert ratings,” a purposive sample of four therapists and four clients (five females, three males, mean age: 49 years) was recruited to take part in the TSTI with the view to refine the original 36-item RDFS prior to psychometric exploration. Structured observations pointed to problems in test-takers’ patterns of responses in relation to theoretical knowledge of the relational depth construct. Issues uncovered and addressed included some misinterpretations of instructions and items, redundant content, double-barreled items, and test-takers’ reactions to intimate content wording. The method supported the refinement of the RDFS including amendment to its instructions and the removal of problematic items. TSTI results produced knowledge on the scale which could not be captured with statistical methods.

Read more about Gina Di Malta’s work here:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s