QAS Self Study: Measurement

1. The general principle to use in consolidating results of pilot testing is to group times in such a way that the CPE hours for the course are not distorted. Consider the following examples.
   
   Five pilot testers report times of 51-56-83-100-150.The three closest grouped times are taken as representative; that is, 51-56-83. These total 190 minutes which divided by 3 equals an average of 63 minutes. 63 divided by 50 yields 1.0 CPE (rounded to the next lowest hour). If all five times were used the CPE would be 1.5 which would be distorted by the unrealistically high time of 150.
   
   Five pilot testers report times of 37-48-49-90-90. If we chose 48-49-90 we get 1.25 CPE rounded down to 1.0 CPE. If we do 37-48-49 we do not get even 1 CPE. If we do 49-90-90 we get 1.5 CPE. The most realistic choice is 48-49-90 yielding 1.0 CPE or to use all five times which again results in 1.0 CPE.
   
   Six pilot testers report times of 32-55-63-66-72-120. If we remove the low and high outlying times we have a fairly good grouping of times 55-63-66-72 which yields 1.0 CPE. If we use all times we still arrive at 1.0 CPE. If we removed the low outlier we would compute 1.5 CPE which would distort the number.

1. What is considered a substantial change requiring re-piloting testing of a self study program is at the discretion of the sponsor. Some examples follow:
    

• A change or update that affects 20% of the course material
• Following three smaller changes or updates
   

Grammar and spelling should not be a factor of substantial change. The consideration should be driven by program content change.

2. Because the self study platform can track the time a participant spends on course completion, data shows that actual time spent by participants to     complete the course is more than the current CPE credit recommendation. What is the next step?

    The program should be re-pilot tested using a sample of the target audience with the intended knowledge level/background. If results of re-pilot testing     support a different amount of recommended CPE credits, use the new CPE credit recommendation prospectively only.

1. The target audience for a course is non-CPAs. If CPAs are in the audience but not the target audience, do pilot testers need to be CPAs?

Answer:  No.  Sample group of pilot testers must be comprised of individuals with the background/knowledge of the target audience.

1. The pursuit of an alternative to pilot testing for determining the recommend CPE credits for a self study program was inspired by National Association of Insurance Commissioners (NAIC). The NAIC publishes three methods for calculating self study credit time that do not involve pilot testing. While pilot testing is a practical method for determining CPE credit, the samples used in most pilot tests are not of sufficient size to yield statistically valid results and are only an approximation of the actual average completion time for self study courses.
    

    The primary formula used by the NAIC is the basis for the word count formula with the modification to eliminate the adjustment for difficulty level of the     course used by the NAIC and additional adjustments to the reading speed and time per question based on data analysis to determine an optimum value for     technical courses—accounting, auditing and tax courses.

    One hundred courses of varying degrees of difficulty of five different CPE providers were used in the data analysis and research. The data analysis showed     that the word count formula provided reasonable results of determining the recommended CPE credits for a self study course.