IEEE 3128:2025

New IEEE Standard - Active - Draft.
An evaluation framework for the intelligent capabilities of artificial intelligence (AI) dialogue systems such as chatbots, consulting terminals, or operation interfaces is established in this recommended practice. The recommended practice classifies the intelligent capabilities of AI dialogue system into three, as cognitive intelligence, emotional intelligence, and system completeness. For each class, the capabilities are subdivided into five intelligent levels, from the lowest Level 1 to the highest Level 5, with the performance criteria clarified at each level by an evaluation checklist.

This recommended practice establishes an evaluation framework for the capabilities of artificial intelligence dialogue systems such as chatbots, consulting terminals, or operation interfaces. The recommended practice defines and classifies the types and levels of the intelligence capabilities according to a checklist of criteria. The checklist tables describe the criteria used to determine the level that a dialogue system achieves based on the analysis of behavior and performance.
This recommended practice provides a framework for the evaluation of AI applications in dialogue systems, where both industry and academia can achieve consensus on professional terms and definitions, and share a common understanding on test methods as well. This recommended practice can also be used as the test guide for AI dialogue systems in smart manufacturing and smart city construction.