ISTQB® AI Testing (CT-AI): validate and test artificial intelligence systems

Module 1: understand AI and testing challenges

• Defining the AI effect and distinguishing narrow, general and super AI.
• Comparing conventional and AI-based systems.
• Overview of technologies (machine learning, deep learning) and development frameworks.
• Using AI as a Service (AIaaS) and pre-trained models.

Module 2: analyse AI system quality characteristics

• Analysing specific attributes: flexibility, adaptability and autonomy.
• Managing system evolution and algorithmic and societal bias.
• Model ethics, transparency, interpretability and explainability (XAI).
• Safety and undesirable side effects (reward hacking).

Module 3: understand machine learning (ML)

• Distinguishing supervised, unsupervised and reinforcement learning.
• The ML lifecycle: understanding objectives, algorithm selection, training and deployment.
• Model tuning and managing underfitting and overfitting.

Module 4: manage machine learning data

• Data preparation: cleaning, augmentation and feature engineering.
• Dataset splitting strategies: training, validation and testing.
• The impact of data quality and labelling errors on the final model.

Module 5: master functional performance metrics

• Using the confusion matrix to evaluate classifiers.
• Calculating and interpreting key metrics: precision, recall, F1 score and accuracy.
• Specific metrics for regression (MSE, R-squared) and clustering.

Module 6: explore neural networks and coverage

• Deep neural network (DNN) structure and how layers work.
• Neural network-specific coverage measures (neuron, threshold and sign-change coverage).

Module 7: test AI-based systems

• Requirements specification and test levels (model testing versus system testing).
• Test data management and automation bias.
• Detecting and managing concept drift in production.

Module 8: evaluate specific characteristics and explainability

• Challenges in testing probabilistic, non-deterministic and complex systems.
• Evaluating AI decision explainability and transparency.
• Defining test oracles suited to autonomous systems.

Module 9: apply AI testing methods and techniques

• Protection against adversarial attacks and data poisoning.
• Applying pairwise testing to manage combinations.
• Using metamorphic, A/B and back-to-back testing when no oracle is available.

Module 10: set up AI test environments

• Specific test environment requirements for self-learning systems.
• Using virtual environments to simulate complex or dangerous scenarios.

Module 11: use AI to optimise testing activities

• Automated defect analysis and prediction of high-risk areas.
• Intelligent test case generation and regression suite optimisation.
• Using computer vision to test graphical user interfaces (GUIs).