Validation of BrainCheck Digital Cognitive Assessments (BC-Assess): Test–Retest Reliability & Concurrent Validity Compared to Traditional Paper-Based Tests

Introduction

Digital cognitive assessments (DCAs) provide a scalable, accessible approach to early cognitive screening, offering automated scoring, remote delivery, and reduced clinician burden.

The BrainCheck Assess (BC-Assess) battery consists of six digital assessments targeting key domains of cognition:

• Immediate Memory: Immediate Recognition (BC-IMR)
• Delayed Memory: Delayed Recognition (BC-DLR)
• Attention: Trail Making A (BC-TMTA)
• Cognitive Flexibility: Trail Making B (BC-TMTB)
• Executive Function: Stroop (BC-STP)
• Processing Speed: Digit Symbol Substitution (BC-DSS)

Previous studies have demonstrated that BC-Assess effectively distinguishes individuals with normal cognition, mild cognitive impairment (MCI), and dementia [1]. This study examines the test–retest reliability and concurrent validity of BC-Assess compared to traditional neuropsychological tests.

Methods

In conjunction with the HeadSMART II trial (NCT04423198), 521 cognitively healthy adults were recruited from four U.S. sites by BRAINBox Solutions.

Two analytic subsets were defined for this validation study:

1. Test–Retest Reliability Subset

• n = 60 (28 females; age 18–84, M = 48, SD = 18)
• 30 participants with ≤12 years of education
• Participants completed BC-Assess twice, ≥7 days apart
• Reliability was evaluated using Intraclass Correlation Coefficients (ICCs)

2. Concurrent Validity Subset

• n = 68 (34 females; age 18–82, M = 48, SD = 18)
• 33 participants with ≤12 years of education
• Participants completed BC-Assess and matched paper-based assessments in the same session

Correlations were calculated between:

Results

High Test–Retest Reliability for BC-Assess

Intraclass correlation coefficients (ICCs) ranged from 0.72 to 0.89, demonstrating good reliability across most subtests.

• For BC-IMR/DLR, moderate ICCs were observed for accuracy alone (0.61/0.34), likely due to ceiling effects.
• When incorporating response time (accuracy / median RT), ICCs increased to 0.77/0.78 (not shown).

Moderate to High Concurrent Validity

BC-Assess showed moderate to high Pearson correlations with corresponding traditional paper-based assessments, supporting its validity as a cognitive assessment tool.

Conclusion

BrainCheck Assess provides reliable and valid digital cognitive testing across diverse adults. Scores showed strong test–retest reliability and moderate to high correlation with traditional assessments, validating BC-Assess as an efficient and scalable tool for early cognitive screening and monitoring.

Interested in piloting BrainCheck?
Apply for the BrainCheck Pilot Grant at: https://braincheck.com/pilot-grant

References

1. Ye S, Sun K, Huynh D, et al. A Computerized Cognitive Test Battery for Detection of Dementia and Mild Cognitive Impairment: Instrument Validation Study. JMIR Aging. 2022;5(2):e36825.
2. Benedict RHB, et al. Hopkins Verbal Learning Test–Revised: Normative data and reliability. The Clinical Neuropsychologist. 1998;12(1):43–55.
3. Reitan RM. Validity of the Trail Making Test as an indicator of brain damage. Perceptual and Motor Skills. 1958;8(3):271–276.
4. MacLeod CM. Half a century of research on the Stroop effect. Psychological Bulletin. 1991;109(2):163–203.
5. Joy S, Kaplan E, Fein D. Speed and memory in the WAIS-III Digit Symbol–Coding. Archives of Clinical Neuropsychology. 2004;19(6):759–767.