Validity testing of new components
Validity is often confused with reliability but it is important to distinguish between the two concepts.
Validity relates to how well a device or procedure measures what it is intended to measure. The source of the confusion is that one cannot have validity if there is little reliability and over repeated measurements where one keeps getting different answers.
I will return to the issue of reliability in later blogs, but for now let’s focus on validity and how to test it for the new pain scoring component of SortED tablet (OUCH™).
Construct validity To demonstrate this one needs to show that results agree with theoretical expectations. An example suitable for validating OUCH would be that we expect pain levels to reduce when analgesia is given, so before and after analgesia tests with OUCH should show reduced scores.
However this may be simplistic. Inspection of the Oxford Analgesic League Table, which expresses the relative pain reducing power of different analgesics, makes the point. It expresses pain reduction as NNT (Number Needed to Treat); the number of patients needed to be treated for one patient to achieve at least a 50% reduction in maximum pain. Virtually no analgesics have NNT =1 where everybody responds, most NNT are 2-5. Many even higher.
Known group construct validity This approach compares pain levels for different conditions where the pain score should differ. Obviously the groups must be assigned before the results are captured. Since this is needed before the tests are run, a large range of possible conditions must be included to cover a reasonable patient sample. However most conditions carry an expectation of how bad the pain is likely to be. OUCH attempts to reduce the subjective element of pain scoring and may be able to reduce this bias.
Predictive validity This type of validity testing measures some future independent event. Comparing the OUCH score obtained by one observer to the analgesia prescribed by ED staff is one approach. Careful design would be needed here so that the OUCH score could not influence the analgesia prescribed.
Simulated clinical use OUCH now delivers analgesia suggestions and we shall of course be checking whether these meet with clinicians approval on a patient by patient basis before the system is actually used in the clinic.
Gillie Francis – Feb 2018