Early inter-laboratory evaluations provided important information on the performance of new toxicity test methods and laid the foundation for establishing procedures and criteria for formally validating the performance of new test methods. The current criteria and principles of test method validation have evolved from the principles established at meetings and workshops in the 1990s (Balls, et al., 1995; Balls, Blaauboer, Fentem, et al., 1995; Balls & Karcher, 1995; Fentem & Balls, 1997; NIEHS, 1997; OECD, 1996). For those interested in the historical evolution of test method validation, excellent narratives include those by Spielmann (2000) and Spielmann & Liebsch (2001).
Validation criteria for new toxicological test methods in use today were developed by three organizations: the Organisation for Economic Cooperation and Development (OECD), the European Centre for the Validation of Alternative Methods (ECVAM), and the Interagency Coordinating Committee on the Validation of Alternative Methods (ICCVAM). These organizations are sometimes called validation authorities. They have worked to harmonize their validation criteria so that there are no major differences between them (Worth & Balls, 2001).
The following are examples of criteria that should be adequately addressed in any validation exercise:
- The purpose of the test method
- The mechanistic basis of the test method
- The relevance of the test method
- The performance of the test relative to the existing regulatory test and relative to the response in the species of concern
- The reproducibility of the test within and among laboratories
- Whether data was generated using coded chemicals and Good Laboratory Practice (GLP)
The OECD, ECVAM, and ICCVAM also provide guidance on quality control issues relevant to the conduct of inter-laboratory validation studies, such as Good Laboratory Practice (GLP) (Cooper-Hannan, et al., 1999; NICEATM-ICCVAM, 2007b; OECD, n.d.) and Good Cell Culture Practice (GCCP) (Coecke, et al., 2005; Hartung, et al., 2002).
The specific validation criteria and/or guidelines for each organization can be found as follows:
- ECVAM Guidelines
- ECVAM. (2002). The principles and procedures of validation. (Eds. A.P. Worth & M. Balls). Altern. Lab. Anim. 30, Suppl. 1, 15.
The OECD, ECVAM and ICCVAM have similarly defined toxicological test methods validation:
OECD (2005): "the process by which the reliability and relevance of a particular approach, method, process or assessment is established for a defined purpose"
ECVAM (Balls, et al., 1995): "the process by which the reliability and relevance of a procedure are established for a specific purpose"
NICEATM-ICCVAM (2003): "the process by which the reliability and relevance of a procedure for a specific purpose are established"
Reliability and relevance are further clarified as follows (OECD, 2005):
Reliability: "Measures of the extent that a test method can be performed reproducibly within and between laboratories over time, when performed using the same protocol. It is assessed by calculating intra- and inter-laboratory reproducibility and intra-laboratory repeatability."
Relevance: "Description of relationship of the test to the effect of interest and whether it is meaningful and useful for a particular purpose. It is the extent to which the test correctly measures or predicts the biological effect of interest. Relevance incorporates consideration of the accuracy (concordance) of a test method."
Most validation studies benefit from the process of prevalidation, which is a small inter-laboratory study conducted prior to the larger inter-laboratory validation study. The prevalidation study provides the opportunity to optimize the test method protocol and to obtain a preliminary assessment of the test method's performance and reproducibility (Curren, et al., 1995). A logical sequence of events for the validation of a test method is described below in Figure 1.
The OECD is an inter-governmental organization whose work is conducted through committees and working groups composed of member country delegates. There was considerable international debate on test method validation among many parties in the early 1990s. The OECD's work on guidance for test method validation was begun by 1994 in an attempt "to internationally harmonize the various published and advocated concepts for the validation of alternative test methods" (OECD, 2002) and resulted in the development of the 1996 OECD Solna Report (OECD, 1996). The OECD validation and acceptance criteria have been refined over time, and the current guidance is the OECD Guidance Document on the Validation and International Acceptance of New or Updated Test Methods for Hazard Assessment (OECD, 2005). OECD also provides guidance on the validation of (Quantitative) Structure-Activity Relationship ((Q)SAR) models (OECD, 2004; 2007). According to the OECD, a test method can be validated prior to being considered for an OECD Test Guideline (TG), but validation of a test is not a prerequisite for initiating the development of a TG (OECD, 2005, p. 47).
The European Commission established ECVAM in 1991 to coordinate the validation of alternative test methods within the European Union (EU) and to promote the acceptance of validated methods by regulatory authorities at the EU and international levels. ECVAM participated in early validation studies and international efforts to define validation principles. Based on this experience, ECVAM published validation principles for alternative test methods in 1995 (Balls, et al., 1995). ECVAM's current principles and procedures for validation have been published (ECVAM, 2002, p.15), and ECVAM Guidelines have been provided for submitting a test method. Additional ECVAM publications on validation describe specific approaches or principles such as integrated testing strategies (Blaauboer, et al., 1999), the modular approach to validation (Hartung, et al., 2004), or principles of weight of evidence validation (Balls, et al., 2006).
An assessment of scientific validity by ECVAM can be based either on the outcome of a prevalidation and/or validation study, or on a weight-of-evidence approach (Worth & Balls, 2001). A weight-of-evidence approach can be used when there is sufficient published information on a method; otherwise, a prevalidation and/or validation study would be conducted. Regardless of who conducts a validation study, a method evaluated by ECVAM must fulfill ECVAM's validation criteria. ECVAM also emphasizes the use of a prediction model (PM) as an integral component of the test system (ECVAM, 2002). The PM is the algorithm used to convert the test data from the new method into a prediction of the reference method endpoint currently accepted by regulatory authorities.
The ECVAM Scientific Advisory Committee (ESAC), which supports the activities of ECVAM, assesses the validity of a method that has undergone validation (formal or weight of evidence) and issues "a statement endorsing the scientific validity of the method." ESAC statements are published in the journal Alternatives to Laboratory Animals (ATLA) and on the ECVAM website. ESAC members are selected by the European Commission from individuals nominated by member countries and include one representative from each EU country as well as other stakeholders.
Methods considered scientifically valid following their evaluation by ECVAM and the ESAC are described on the ECVAM website and in AltTox's table of Validated/Accepted Alternative Methods.
ECVAM has processes in place for assessing the outcomes of validation studies conducted by other organizations (Worth & Balls, 2001) and has continuously worked with other organizations to internationally harmonize validation and acceptance of alternative methods. Due to the urgent need for animal alternatives to address the testing needs of recent EU cosmetics (Council Directive 2003/15/EC) and chemicals policies (REACH Regulation (EC) No 1907/2006), ECVAM has restructured into key areas that directly target the animal tests to be replaced, such as systemic toxicity, topical toxicity, and sensitization (Hartung, et al., 2003).
ICCVAM was established in the US "to facilitate the validation, review, and adoption of new scientifically sound test methods, including alternatives that can refine, reduce, and replace animal use (Stokes, et al., 2002). ICCVAM is an interagency committee composed of representatives from 15 US Federal agencies. Both regulatory and non-regulatory agencies "that generate or use toxicological data to carry out their responsibilities" are represented on ICCVAM. Additional activities of ICCVAM include facilitating interagency and international harmonization of test methods, and coordinating other interagency issues related to test methods. ICCVAM activities are announced by publication in the Federal Register.
ICCVAM published its original validation and regulatory acceptance criteria and processes in 1997 (NIEHS, 1997). Since that time, a submissions guidance document has been developed (NICEATM-ICCVAM, 2003) to be used in conjunction with information in the 1997 validation and regulatory acceptance criteria document, which was updated in 2001. Schechtman (2006) provides an overview of these documents and the activities of NICEATM-ICCVAM.
The National Toxicology Program (NTP) Interagency Center for the Evaluation of Alternative Toxicological Methods (NICEATM) provides administrative and technical support for the activities of ICCVAM. ICCVAM and NICEATM work collaboratively on validation-related activities. ICCVAM is also supported by the Scientific Advisory Committee on Alternative Toxicological Methods (SACATM), which is composed of 15 scientific representatives from various stakeholder groups. The SACATM, unlike the ESAC, does not directly participate in assessing the validity of test methods but advises ICCVAM and NICEATM. ICCVAM and the SACATM are committees that meet only several times per year, while the NICEATM has an office and staff at the NTP.
NICEATM-ICCVAM conduct reviews of new or revised test methods that are nominated by government agencies and other stakeholders when sufficient information is available. Members of ICCVAM and the NICEATM staff are willing to meet with and provide feedback to test developers and validation sponsors, and they encourage the participation of regulators in all steps of the validation process. At their inception, NICEATM-ICCVAM did not directly support test method validation studies, but in collaboration with ECVAM and JaCVAM they are now participating in international validation studies for in vitro cytotoxicity test methods and an in vitro estrogen receptor transcriptional activation assay.
NICEATM assembles independent scientific peer review panels for the review of proposed test methods. The peer review panel develops a statement on the scientific consensus of the validity of the method under review (NICEATM-ICCVAM, 2007a). ICCVAM prepares recommendations on a proposed method, based on the peer review panel report and SACATM comments. Public comment is also considered. If ICCVAM judges the method as being validated, it forwards recommendations on the validity and potential acceptable applications of the method to relevant US agencies for their consideration in accepting the test method.
ICCVAM has also developed procedures for considering test methods endorsed by ECVAM and for working collaboratively to facilitate test method review, validation, and international harmonization of alternative test methods (ICCVAM, 2001; Schechtman & Stokes, 2002).
Methods that have been peer reviewed and/or validated by ICCVAM are described on the ICCVAM website and in AltTox's table of Validated/Accepted Alternative Methods.
Other Countries and Regions
Other countries and regions are also involved in developing and validating non-animal toxicity test methods. Typically, regulatory authorities in other countries will partner with ICCVAM or ECVAM for test method reviews. Japan has conducted validation studies and participated in international validation and harmonization activities (Kojima, 2006). The Japanese Center for the Validation of Alternative Methods (JaCVAM) is a part of Japan's National Institute of Health Sciences (NIHS) (Ohno, 2004). Alternative methods in Korea are developed and validated at the National Institute of Toxicological Research (NITR) (Park, 2006). NITR has plans to establish a Korean Center for the Validation of Alternative Methods (KoCVAM).
Stakeholder Programs to Promote Validation
There are many cases of industry and industry trade associations supporting the development and validation of alternative methods. A widely-known example is the early validation studies for alternatives to the Draize rabbit eye test supported by members of the US Cosmetic, Toiletry, and Fragrance Association (CTFA).
The European Partnership for Alternative Approaches to Animal Testing (EPAA) is a recent EU initiative among the European Commission and various industry sectors to facilitate the validation and acceptance of alternative test methods.
Timeline and Rate Limiting Steps
The prevalidation/validation process for each new or revised test method typically takes about 4-6 years but could be less than two years using techniques such as ECVAM's "catch-up validation" (defined as an abbreviated study for a method significantly similar in structure and performance to one already rigorously validated) (Worth & Balls, 2001).
An expert panel of stakeholders developed the following time-frame for each major step in the validation and regulatory acceptance of alternative methods (ECVAM, 2005, p.8):
|R&D through test optimization
||undefined number of years|
|Peer review & validation authority endorsement
The development of new test methods and the regulatory acceptance process are the rate limiting steps in moving new methods through the validation process (ECVAM, 2002; Worth & Balls, 2001). Greater efficiencies are expected upon the infusion of more resources and the development of persons with validation study expertise. Further harmonization of validation procedures and mutual acceptance of validated methods would also facilitate new test adoption.
Following successful validation, a new or revised test method can be considered ready for regulatory acceptance and implementation for regulatory decision-making purposes.
Figure 1. Evolution of a new or revised toxicity test method for regulatory purposes. Ideally, test developers, validation centers, and regulatory officials work together to design a relevant test method and the prevalidation/validation studies to assess its performance.
Need for a new test method identified
R&D to develop a relevant test method
Inter-laboratory prevalidation study to optimize the protocol and access the performance
Data analysis/management team decision to validate method
Validation study to gather sufficient inter-laboratory data for determining assay performance and reproducibility and to provide results on sufficient chemical classes
Validation study data analysis and recommendation of management team
Validation study results published in peer-reviewed journal
Transparent and independent peer review of the validation study results and data
Validation authority review of the data and peer review recommendations published
Results and recommendations forwarded to regulatory authorities for acceptance consideration, and to the OECD for new or revised Test Guideline (TG) development
[Review and decision by regulatory authorities; development of OECD TG]