Quality control strategies for effective AI in digital pathology

Digital pathology, the process of digitizing pathology slides for analysis, has revolutionized the field of pathology. Artificial analysis on digital pathology slides involves the application of machine learning algorithms and artificial intelligence (AI) techniques to assist pathologists in diagnosing diseases accurately and efficiently.

To ensure reliable and accurate results, it's beneficial to follow best practices for workflow processes and maintain strict quality control measures. This approach helps guarantee long-term success in AI analysis of digital pathology slides, avoiding significant drift in protocols, procedures, and data sets that could affect an AI model's performance. Establishing these principles and quality controls before starting AI model training will help maintain good performance, both in the short and long term.

This article outlines some of the best practices covered in more detail in our whitepaper, "Best practice guide to ensure quality and long-term success using artificial intelligence to analyze digital pathology slides.” These best practices are part of the "Good Machine Learning Practice" guidelines from the US FDA, UK, and Canadian governments. 

The guidelines are divided into the following four categories: 

1. Project initialization and planning
2. Upstream workflows
3. Downstream workflows 
4. Long-term model maintenance and sustainability

Overview of the control steps involved in producing and maintaining a long-term AI model

1. Project initialization and planning

Begin each AI project with a comprehensive initial assessment, evaluating the needs of the participating scientists, pathologists, IT professionals, and stakeholders. This will help you understand the specific requirements and challenges for your AI project.

When planning the project, pay special attention to data collection and annotation: Gather a diverse and representative dataset with proper metadata covering various pathologies, tissue types, and staining techniques. Review the data regularly to manage changes over time. Finally, engage expert pathologists/scientists to ensure accurate annotations and set clear guidelines for consistent labeling (consider producing a ground truth document).

2. Upstream workflows

Upstream covers the time period before running the model. Digital scanning and quality control measures at this stage are crucial for reliable AI analysis further in the process. 

Sample collection and processing

To maintain consistency, protocols for tissue acquisition and cytologic preparations should be standardized. This includes standardizing sampling techniques, the time between tissue harvesting and fixing, and the fixation duration. Moreover, it is important to document all details about sample origin, processing time, and fixation methods.

Further guidelines for sample collection and processing include: 

• Tissue sectioning: Use sharp, well-maintained microtomes for uniform tissue sections to reduce artifacts. Regularly calibrate equipment and monitor section thickness to ensure uniform slides. Perform quality control checks for sectioning, processing, and embedding artifacts (such as folding or tearing).  
• Staining procedures: Follow standardized staining protocols for consistent results. Regularly test staining reagents and implement standard dehydration and clearing steps to avoid artifacts. Automation of IHC and vital stains can help minimize variation.
• Slide preparation: Standardize slide mounting and drying processes to avoid bubbles or smudges. Maintain consistency in consumables like glass slides and cover slips.
• Quality assurance checks: Conduct regular quality audits to assess staining accuracy and consistency. Implement double-check mechanisms for stained slides. Commercially available quality control tests are available for vital stains such as HE.
  
  

Slide digitization

Digital slides are the prerequisite for AI-assisted image analysis. Keep these things in mind for the best results: 

• Scanner calibration: Regularly calibrate digital scanners to maintain color accuracy and sharpness. If possible, use standardized calibration slides. Ensure software versions, compression, file types, and pixel density are consistent within AI projects.
• Slide preparation for scanning: Ensure slides are clean and free of debris or stains before scanning to prevent artifacts. Inspect for overhanging labels, tape, or excess mounting medium. Wipe slides with a lint-free cloth and check for cracks or chips. Wait until the slides are fully dry before loading into the scanner.
• Image acquisition: Set scanning parameters consistently for uniform image capture. Place slides flat in the scanner and engage any securing mechanism. Randomly inspect scanned images to detect anomalies or artifacts. Implement a slide tracking system to ensure all slides are scanned and accounted for. Ensure the objective magnification and image resolution is suitable for AI analysis.
• Data annotation and labeling: Expert pathologists should oversee the annotation of slides, marking regions of interest and labeling specific structures. Use standardized protocols and controlled vocabularies for consistent labeling. Implement double-check mechanisms to validate annotations for accuracy. Use techniques like stain normalization to mitigate color variations between slides.
  
  

3. Downstream workflow

Downstream includes all changes after the AI model has been produced. The important quality control measures in this phase include: 

• Annotation quality control: Conduct periodic reviews of annotations to ensure consistency and accuracy. Organize regular training sessions for annotators to maintain annotation standards. Implement inter-annotator agreement checks and resolve discrepancies through consensus.
• Model performance monitoring: Establish a system to track the model's performance over time. Implement feedback loops with pathologists and scientists to refine the model based on real-world feedback.
• Compliance and security: Adhere to data protection regulations and standards to ensure patient data confidentiality. Implement secure data transmission protocols and encryption methods for data transfer and storage.
  
  

4. Long-term model maintenance and sustainability

Finally, to ensure the long-term success and sustainability of the AI model, consider the following: 

• Model performance monitoring: Using a monitoring system, continuously assess model performance on new data. Implement feedback loops with pathologists for real-time model refinement.
• Documentation and knowledge transfer: Maintain comprehensive documentation covering data sources, pre-processing steps, model architectures, and training strategies. Facilitate knowledge transfer within the team and to new members to ensure continuity.
• Regular updates and retraining: Update the model regularly based on new data and evolving pathology practices. Retrain the model periodically (if necessary), incorporating the latest advancements in AI and deep learning techniques. 
• Compliance and ethical considerations: Adhere to regulatory frameworks, data protection laws, and ethical guidelines to ensure patient privacy and compliance. Obtain necessary approvals from relevant regulatory bodies before deploying the AI system in clinical practice.

Implementing these best practices for upstream and downstream workflow processes, along with stringent quality control measures, is essential for accurate and reliable AI analysis of digital pathology slides. Following these guidelines can enhance the efficiency of pathological diagnosis, leading to better patient outcomes and overall healthcare delivery.

Further learning materials

For additional information on the topic, check the following websites:

• US FDA - Good Machine Learning Practice for Medical Device Development: Guiding Principles
• UK Gov - Software and Artificial Intelligence (AI) as a Medical Device

References

Abels et al. (2019, September 3). Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association. The Journal of Pathology, 249(3), 286-294. https://doi.org/10.1002/path.5331 

Aeffner et al. (2019, March 8). Introduction to digital image analysis in whole-slide imaging: a white paper from the Digital Pathology Association. J Pathol Inform., 10(9). https://doi.org/10.4103/jpi.jpi_82_18 

Louis et al .(2016, January). Computational pathology: a path ahead. Arch Pathol Lab Med, 140(1), 41-50. https://doi.org/10.5858/arpa.2015-0093-sa 

Niazi et al. (2019, May). Digital pathology and artificial intelligence. Lancet Oncol., 20(5), e253-e261. https://doi.org/10.1016/s1470-2045(19)30154-8 

Zarella et al. (2023, September). Artificial intelligence and digital pathology: clinical promise and deployment considerations. J Med Imaging (Bellingham), 10(5). https://doi.org/10.1117/1.jmi.10.5.051802 

Zarella et al. (2019, February). A practical guide to whole slide imaging: a white paper from the Digital Pathology Association. Arch Pathol Lab Med, 143(2), 222-234. https://doi.org/10.5858/arpa.2018-0343-ra