Main Body

Clinical Chemistry and Hematology Laboratories

Therapeutic Substance Monitoring (TSM) Proficiency Testing Program

• Therapeutic Drug Monitoring Proficiency Database

The TSM proficiency testing program was established in 1977. The laboratory discipline augmented existing programs in clinical chemistry, toxicology, hematology, microbiology, and diagnostic immunology which were created in response to Public Health Laws of 1963 (10 NYCRR Part 58), authorizing the Department of Health to license clinical laboratories that provided analytical services to healthcare providers in the State. Only laboratories that maintain standards of good laboratory practice and demonstrate analytical proficiency are permitted to provide services to the healthcare provider.

Currently, there are about 425 laboratories that either hold or are attempting to qualify for the Therapeutic Substance Monitoring laboratory permit. These laboratories are challenged at least three times each year with specimens prepared by Wadsworth Center staff. Laboratories process the specimens as patient specimens and report their test findings to the Center. Test results are evaluated using performance criteria specified by the Clinical Laboratory Improvement Amendments of 1988 and by the Wadsworth Center. Substandard analytical performance that may compromise patient care is investigated and resolved through a coordinated effort between the testing laboratory and program staff.

---

Proficiency Testing Program Activities

The program is designed and managed by Wadsworth Center staff and comprises the following activities:

• specimen design
• specimen preparation and validation
• test kit preparation and shipment
• results review
• results processing and performance evaluation
• test reports
• consultation and remediation

Specimen Design

The objective in specimen design is the formulation of a test material that is stable, contains analytes at levels encountered in testing patient populations, and behaves as a patient specimen in the analytical system (i.e., no untoward effect of test material matrix on analytical performance characteristics).

Technological advances have produced remarkably stable and highly automated analytical systems that require microliter volume of test specimen for analysis. Unfortunately, many of these analytical systems are not adequately robust to produce accurate test results when challenged with test materials that are not characterized as freshly collected patient specimens. The preparation of suitable test materials and the parsing of analytical error as either clinically significant or non-relevant matrix effect is a continual challenge to the PT program.

Specimen Preparation and Validation

We use normal human serum derived from plasma as the test specimen matrix. Exact amounts of analytes (drugs) are transferred to the serum pools. Five lots of test specimens are prepared for each proficiency test event to challenge the analytical methods over the range of clinically relevant drug concentration. Aliquots of test specimens are bottled under sterile conditions and are stored at -70 degrees Centigrade until they are shipped to laboratories for processing. The accuracy of preparation and the stability of test materials are validated by program staff using immunoassay and/or high-performance liquid or gas chromatography and by statistical analysis of data produced by laboratories participating in the program.

Test Kit Preparation and Shipment

Three proficiency test events are scheduled each year. Test kits are prepared for each event and include the following materials:

Test Kits

• five test specimens that are shipped frozen in an insulated container with a frozen gel pack
• instructions for specimen handling and processing
• glossary of instrument and reagent codes
• form used to record and report test findings

The test kit is typically shipped to the laboratory via United Parcel Service and is delivered within 24 to 48 hours.

Results Review (Panic Values)

The clinical laboratory must postmark its report of test findings within eleven days of the test kit shipment date. Results are reviewed immediately upon receipt for results that are so discrepant from expected results as to pose imminent danger to patient care. Program staff immediately telephone the contact person identified on the report form and the discrepant result(s) are discussed. The laboratory is required to investigate immediately and to determine whether erroneous patient results were likewise reported to a healthcare provider. The laboratory must verbally notify program staff of its findings that same day and follow up with a written report (Panic Values).

Reports are also reviewed for completeness (director signature, instrument and reagent codes, laboratory identification codes) and for clarity prior to release for data entry.

Results Processing and Performance Evaluation

Process used for the evaluation of participant performance:

Process for Evaluation

• assign the weighed amount of drug as the preliminary target value
• judge appropriateness of preliminary target through robust statistical analysis of participant results
• assign target value as either the weighed amount of drug or the robust estimate of mean drug concentration
• investigate sources of analytical bias should results from an analytical system differ significantly from assigned target value
• assign method-specific target value should source(s) of analytical bias not be clinically relevant or significant
• use allowable range ascribed by both CLIA'88 and the Wadsworth Center around the assigned target value to judge laboratory performance

Target Value Assignment

The ideal target value is the analyte concentration determined from the amount of pure substance added to the defined volume of matrix. The use of a gravimetrically determined target value is achievable under the following conditions: 1) test specimens are prepared accurately, 2) test specimens are stable, 3) the analytical method is calibrated for (and is specific for) the drug added to the matrix (no interference from clinically insignificant metabolite(s)), and 4) the analytical method is robust and insensitive to specimen matrix.

We assign a preliminary target value to a specimen analyte concentration as the amount of drug added. The gravimetrically assigned value is compared to the mean concentration determined from statistical analysis of participant results, and if agreement is within 3%, the gravimetric target value is used for the evaluation of laboratory performance. Otherwise, the statistical estimate of mean drug concentration is used as the target value.

Statistical Analysis

Program staff list the drugs included in the test specimens and the laboratory is required to quantify those drugs for which service is provided to physicians. The number of program participants reporting results ranges from about 33 for free phenytoin to about 350 for phenytoin. Numerous instrument/reagent combinations are used among participants (as many as 24 for digoxin), and the distribution of test results for some analytical systems may differ significantly from that of others, primarily due either to assay calibration or to test specimen matrix effects. The potentially large number of peer groups, some with a small number of laboratories, and the possibility of multi-modal distribution of composite results poses a challenge to statistical analysis and characterization of laboratory performance.

We use robust statistical methods to estimate the location (mean) and dispersion (standard deviation) of peer group and composite proficiency testing data. The robust statistical analysis does not assume a gaussian distribution of data and is tolerant of small sample size.

Peer-Group Target Value Assignment

In some instances, an analytical method may produce results that are biased significantly from an assigned target value which may result in a high failure rate for the analyte challenge among users of the method. Program staff, the method manufacturer, and laboratories coordinate efforts to investigate whether the source of analytical error is clinically relevant. The investigation is typically accomplished through parallel testing of patient and proficiency test specimens by the method under investigation and by a referee method. Should the analytical bias documented through analysis of PT specimens not be evident in the analysis of patient specimens, the program will evaluate proficiency test results using the statistical mean of method results as the target value. Investigations of the source of analytical bias ensue, and should the source of interference be identified as clinically relevant, the use of the peer method mean as the analyte target value is suspended.

Allowable Range of Analyte Concentration Around Target Value

Laboratory performance is judged using specifications published by the federal Department of Health and Human Services (CLIA'88) and by the New York State Department of Health (DOH). The DOH has been evaluating laboratory and analytical systems performance in therapeutic drug monitoring since 1977. Until 1982, the program used the standard deviation of participant results as the index of acceptable performance, i.e., results that fell within plus or minus 2 SD were judged acceptable. This was standard practice among proficiency testing programs. However, we recognized significant shortcomings in this approach to judging performance: 1) the use of the standard deviation of test results to establish ranges of acceptable results accommodated poor performance and the use of inferior methods -- poor performance among program participants was rewarded with larger ranges of acceptable results; and, 2) program performance standards had no defined relationship to clinical requirements for good patient care.

The therapeutic substance monitoring PT program in 1982 introduced a fixed performance standard to judge laboratory analytical proficiency. The state-of-practice among quality laboratories was an interlaboratory relative standard deviation (CV) of about 20%. All laboratories participating in the PT program were expected to achieve and maintain that level of performance, i.e., determination of drug concentration to within 20% of the target value. The state-of-practice has improved, and today, laboratories are expected to determine analyte concentration to within 15% of the assigned target value. In the context of clinical requirements for analytical proficiency, no consensus has been reached on allowable analytical error. However, using statistical models that estimate the contribution of analytical error to the clinical misinterpretation of theophylline monitoring data, we have proposed that total error should not exceed 9% (Jenny, RW. Clin Chem 1991;37:154-8). Critiques of laboratory performance in the PT program include an evaluation of whether the laboratory achieved a level of performance that does not compromise the clinical utility of drug monitoring data

Enter CLIA'88.

The Clinical Laboratory Improvement Amendments of 1988 establish performance evaluation criteria to be used by proficiency testing programs approved by the federal Department of Health and Human Services. The rule-making is silent on the basis for these criteria (Jenny RW and Jackson KY. Clin Chem 1994;38:496-500). Nonetheless, federally approved PT programs including New York State programs are required to evaluate laboratory performance in the context of published evaluation criteria. We have adopted the federal evaluation criteria as a minimum standard of performance, and continue to judge analytical proficiency using a standard that will continue to evolve as technology advances and clinical demands are better understood.

Program Analytes and Performance Standards

TDM/Toxicology PT Program Performance Standards (target ± stated ranges)
Analyte	DHHS CLIA'88	NYS DOH
	Satisfactory (100%)	Satisfactory (100%)	Marginal (50%)
acetaminophen	not stated	15%	20%
carbamazepine	25%	15%	20%
digoxin	0.2 ng/mL or 20% (greater)	0.2 ng/mL or 15% (greater)	0.3 ng/mL or 20% (greater)
ethanol	25%	10%	15%
ethosuximide	25%	15%	20%
free phenytoin	not stated	20%	25%
gentamicin	25%	15%	20%
lithium	0.3 ng/mL or 20% (greater)	0.2 ng/mL or 15% (greater)	0.3 ng/mL or 20% (greater)
procainamide/ NAPA	25%	15%	20%
phenobarbital	20%	15%	20%
phenytoin	25%	15%	20%
primidone	25%	15%	20%
quinidine	25%	15%	20%
salicylate	not stated	15%	20%
theophylline	25%	15%	20%
tobramycin	25%	15%	20%
valproic acid	25%	15%	20%
vancomycin	not stated	15%	20%

 

Definition of Terms

Unsatisfactory proficiency testing performance means failure to attain the minimum satisfactory score (80%) for an analyte or overall test event.

Unsuccessful proficiency testing performance means a failure to attain the minimum satisfactory score (80%) for an analyte or overall test event for two of three consecutive test events.

Program Reports

Program participants are provided several reports to assist them in the review and evaluation of test performance.

1. Performance Evaluation Report -- the official record of laboratory performance in the test event. The report includes the analytes tested for, performance standards put forth by both CLIA'88 and the NYS DOH, analyte and event test scores, and a statement of satisfactory or unsatisfactory performance.
2. Performance History Report -- the official record of analyte and event scores over the four most recent test events. Analyte and specialty status is listed as Successful, At Risk (for unsuccessful status), or Unsuccessful.
3. Report of Laboratory SPC Data -- a supplementary report that characterizes laboratory performance and errors in the context of clinical and PT program performance specifications.
4. Statistical Critique -- a tabulation of the mean, standard deviation, and relative standard deviation (CV%) for each of the analytical methods used used by program participants.

---

Please mail comments, corrections or suggestions to: clinchem@wadsworth.org