On June 23, Beijing International Studies University (BISU) unveiled the "Big Language Model Translation Quality Evaluation Report BISU-AiTQA (v1.0)". This evaluation marks a breakthrough in the current international evaluation landscape, which predominantly focuses on English, by establishing a comprehensive evaluation system based on Chinese and encompassing multiple languages and domains.

BISU releases the BISU-AiTQA(v1.0) on June 23. 

This pioneering effort is crucial for fostering cultural exchanges between China and the world and enhancing the global expressiveness of the Chinese language.

Research for the evaluation was conducted on six major language models, ranging from ChatGPT to Claude, Gemini, and domestic models like Qwen and DeepSeek.

The rapid advances made in large language models in recent years has brought them close to human-level translation capabilities in multilingual tasks. This progress significantly enhances the naturalness of translations, consistency in terminology, and coherence in discourse, thereby revolutionizing the translation industry, educational methods, and research paradigms.

Despite these advances, large language models still face challenges in understanding complex language structures, deep semantics, domain-specific knowledge gaps, and prominent issues in Chinese comprehension and generation.

Consequently, there is a pressing need to establish a systematic and interpretable evaluation procedure to assess the true capabilities of these models in multilingual translation tasks and further improve their translation abilities.

Based on BISU's extensive expertise in foreign language disciplines and teaching, and drawing on previous translation evaluation research achievements, the university mobilized faculty and students who are proficient in English, Japanese, Russian, French, and Arabic to evaluate the translation quality of six major language models from domestic and international sources across the domains of contemporary literature, Party and government documents, and foreign affairs news.

Built on Chinese-specific texts, BISU-AiTQA (v1.0) developed a six-dimensional evaluation framework that incorporates lexical diversity, syntactic complexity, discourse coherence, and linguistic acceptability, among other indicators.

With a focus on the linguistic characteristics of translations, the evaluation effectively identifies a model's capabilities in lexicon, syntax, discourse coherence, semantic understanding, and language expression, as well as its handling of Chinese discourse, cultural connotations, and policy expressions.

This initiative fills a crucial gap in multilingual evaluations based on Chinese and represents a solid step forward in promoting cultural exchanges between China and the world.

The rise of AI technology presents unprecedented challenges for foreign language education, yet despite its capabilities, it still cannot fully handle complex languages.

When utilizing AI technology, users must have judgment skills, and fostering such skills is a crucial aspect of foreign language education.

In the age of artificial intelligence, foreign language learning is not only necessary but imperative. Based on this premise, BISU-AiTQA (v1.0) emerged from the intersection of liberal arts and sciences, enriching language teaching.

The multilingual, multidisciplinary language corpora and translation evaluation metrics within the project can be widely applied in translation courses, language testing, language analysis, and other educational scenarios, providing a vital platform for the digital transformation of translation education and language research.

Language serves not only as a means of communication but also as a building block of human thought processes. Language proficiency is the cornerstone of foreign language talent, and the advancement of large language model technology serves as a catalyst for foreign language development, expanding pathways for foreign language teaching and translation research.

In classrooms, efforts should be made to guide students in collaborating with models, training them in human-machine interaction, language judgment, and critical thinking skills to cultivate students as guides of AI.

Amid the national push for the "AI +" strategy, where does the charm of foreign language disciplines lie, and how can foreign language institutions undergo significant transformations? BISU has taken the lead in addressing these questions.

The School of Artificial Intelligence and Language Sciences at BISU, for instance, leveraging the university's strengths in foreign languages, tourism, and regional studies, has recruited faculty in computer science, mathematics, and linguistics.

Focusing on the innovative fusion of language science and artificial intelligence technology, the school targets key "AI + X" fields supported by the Beijing municipal government, such as foreign language services, brain-computer interfaces, care for and rehabilitation of the elderly and intelligent cultural tourism.

The origins of the school can be traced back to the establishment of the "Language Cognition Laboratory" at BISU in 2008, which was reorganized into the "Artificial Intelligence and Language Cognition Laboratory" in 2018.

Currently, the school has assembled professionals from cognitive neuroscience, biomedical engineering, information processing, natural language processing, computer science and technology, linguistics, and other fields, focusing on research areas such as artificial intelligence and language, brain science and intelligent computing, and intelligent intersections. The school is dedicated to nurturing high-end talent with language proficiency and artificial intelligence technology skills.

BISU students participate in research projects at the School of Artificial Intelligence and Language Sciences.

In the future, BISU will continue to explore talent development models around "AI + foreign languages", providing an effective "BISU solution" for the development path of foreign language institutions in the era of artificial intelligence.

 BISU-AiTQA(v1.0): http://aitqa.bisu.edu.cn/