XAI LANGUAGE TUTOR - A XAI-BASED LANGUAGE LEARNING CHATBOT USING ONTOLOGY AND TRANSFER LEARNING TECHNIQUES

International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 DOI: 10.5121/ijnlc.2020.9501 1 XAI LANGUAGE TUTOR - A XAI-BASED LANGUAGE LEARNING CHATBOT USING ONTOLOGY AND TRANSFER LEARNING TECHNIQUES Nuobei SHI, Qin Zeng and Raymond Lee Division of Science and Technology, Beijing Normal University-Hong Kong Baptist University United International College, Zhuhai, China ABSTRACT In this paper, we proposed a XAI-based Language Learning Chatbot (namely XAI Language Tutor) by using ontology and transfer learning techniques. To facilitate three levels of language learning, XAI Language Tutor consists of three levels for systematically English learning, which includes: 1) phonetics level for speech recognition and pronunciation correction; 2) semantic level for specific domain conversation, and 3) simulation of “free-style conversation” in English - the highest level of language chatbot communication as “free-style conversation agent”. In terms of academic contribution, we implement the ontology graph to explain the performance of free-style conversation, following the concept of XAI (Explainable Artificial Intelligence) to visualize the connections of neural network in bionics, and explain the output sentence from language model. From implementation perspective, our XAI Language Tutor agent integrated the mini-program in WeChat as front-end, and fine-tuned GPT-2 model of transfer learning as back-end to interpret the responses by ontology graph. All of our source codes have uploaded to GitHub: https://github.com/p930203110/EnglishLanguageRobot. KEYWORDS NLP-based Chatbot, Explainable Artificial Intelligence (XAI), Ontology graph, GPT-2, Transfer Learning 1. INTRODUCTION Chatbot is one of the hot topics in Natural Language Processing, normally, it considered as the by-product of Question-Answer (QA) system. With the same procedures to understand and give response to human users, machine will go through these FIVE steps to complete one round dialogue. All have comprised of automatic speech recognition (ASR), natural language understanding (NLU), dialogue management (DM), natural language generation (NLG), speech synthesis (SS). Figure 1 shows the system flow of a typical human-machine interaction system. Figure 1. Flowchart of a typical chatbot system In decade ago, Natural Language Processing (NLP) only a topic related to the linguistic of human language, where the QA system depends on the human collection and manual operation resemble the expert system. When the computational linguistic related different disciplines, such as International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 2 statistical engineering, machine learning, data mining and human voice recognition. [9] The procedures and processing system are different with before. In the meantime, NLP update to a technology belongs to Artificial Intelligence related to cognitive science of psychology, philosophy and biology for structure simulation. Researches on rule-based matching chatbot were incited since the first chatbot was invented and tried the Turing Test in 1950s. To build a chatbot in such pattern require tremendous amount of human dialogues as knowledgebase. Traditional chatbot with sufficient corpus can generate suitable responses for human questions in both grammar and matching rates due to responses are natural conversations produced by human. Additional matching words signified the betterselected responses. Thus, AI-based NLP technology challenges nowadays are machines’ capability to generate responses which is the focus of our XAI Language Tutor. Neural network as language model in Natural Language Processing (NLP) supports machine to generate appropriate responses in recent years. Recurrent Neural Networks (RNN) with popular framework like TensorFlow and Keras are the mainstream for Language Model generation. In late 2018, Google published a basic language model called Bidirectional Encoder Representations from Transformers (BERT) with outstanding performance in 11 common NLP tasks, which concentrated on Encoder scheme. Few months later, Open AI released another transformer based on unsupervised learning with pre-trained model focusing on Decoder scheme for text generation, the Open AI GPT-2. By using unsupervised learning as pre-training scheme, the bi-directional Transfer Learning model can serve as a promising Language Model framework in NLP. With the pre-trained language model, relatively small dataset can achieve better performance than traditional language models. Based on GPT-2 with fine-tuned model [1], our language agent has fluent and syntactic response as a virtual AI English tutor for industrial usage. No matter of how excellent performance of these models, the essential neural networks are always in needed of big data as data source due to technology of Neural Network currently. However, Human minds make inferences that go far beyond the data available. The reverse engineering of human learning and cognitive development helps the engineering of humananalogous machine learning system [2]. Neural Network originally is the mimic of human brain. However, Artificial Neural Network outputs are the mathematical computation results of neurons layers. Except the computation process, the structure of topology for result generation and the meaning of result are independent with each other for human. Thus, it always considered as a black box, but the basic concepts in bionics inspired by the structure of neural network biologically for the human thinking and learning processes. We mimic the way of human learning and reasoning to explain the output of the artificial neural networks, which also witness the development of search engines. That related to another question: How do human get information and knowledge? Due to the relation of keywords, the ranking done by search engines are more suitable for human justifications. In this paper, we use ontology graph (OG) to simulate the connection part of neural networks, the axons. The ontology graph is the tree of real-world concepts in different areas acquired by raw data, which focus on the relation between different nodes of the ontology graph. Just like neural networks, the tree also has the characteristic of synapses, which can inspire the relation extraction in ontology graph as memory in human brain. The nodes of entities in Ontology Graph and the bodies of cell in biological neural network are similar for memory activation mechanism. Also, the interaction of agent with human use natural language rather than query language of database or mathematical distance computation for similarity. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 3 Facing the barrier of machine can understand the natural language, we use ontology graph to explain the biological neural networks, where the artificial neural network also inspired. Thus, the key-point is the language model with learning algorithm to make the machine can understand natural language. To some extent, the ontology graph has ability to reason and generate new knowledge when it has sufficient knowledgeable and capable ontology graph that can “absorb” and “generate” new knowledge. From the implementation perspective, XAI Language Tutor constructed based on QuestionAnswer-type of conversation as fundamental interactions between human and machine. In general, such Question-Answer system with knowledge base is better than the system without database, such as Information Retrieval-based Question-Answer (IRQA) by crawler or search engine or rule-based pattern recognition distance matching. The knowledgebase divided into two parts, where the task-oriented knowledge aims at special-domain knowledgebase like expert system, however, the chatbot for daily chats need open-domain knowledge to answer unpredicted routine questions. For example, the customer service chatbot like Ali Xiaomi [3], which is the typical example of E-commerce online support staff to substitute human online customer service, for which the most of conversations expected in E-commerce. The more specific domain, the more suitable for chatbot to predict and shape personality preference for different users. The opposite is open-domain KBQA such as Siri of Apple, Xiao Ice of Microsoft, the interaction provides a 24 hours personal assistant for users including database and APIs with search engine and other apps within one terminal to answer questions of opendomain knowledge. In our XAI Language Tutor, we use unstructured data, English text, from daily dialogue to construct knowledge base with dictionary and graphs (ontology graphs) from fine-tuned dataset. The language model also the fine-tuned Transfer Learning model based on Open AI GPT-2, the 40G pre-trained data of natural language. With Python’s AI ecosystem development platform, researchers will obtain more ideas between neural network and cognition to find highly accurate answers from massive unstructured data. For the aspect of ensemble implementation perspective, we propose a mini-program in WeChat for real-world usage, with the fine-tuned GPT-2 model [1] and speech recognition service from Google, whose three levels systematically English Learning method provide an efficient way in natural language learning. Simultaneously, the ontology graph visualized on Neo4j, graph database, to explain the generated response from agent. The main contributions and originality of this paper include: 1. Following the eXplainable Artificial Intelligence (XAI) concept to explain the output natural language from Neural Network language model to prove the output of neural network are explainable and readable for human. 2. The local explanation using the relatively small dataset with dialogue random sample to build Ontology graph. 3. The introduction of GPT-2 framework, a transfer-learning framework, with dialogue format [1] as a language model for our XAI Language Tutor. 4. The processing of natural language analysis in linguistic for Ontology. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 4 5. The creative idea in users’ aspects to develop an AI NLP-based English agent into miniprogram of WeChat as intelligent mobile XAI Language Tutor and successfully implement in real-world usage for systematically English learning. This paper presented as follows: Section 2 is the literature review to review the contemporary chatbot system from technology companies such as Microsoft, Alibaba and Hugging face in respect of function design and technology component to analyse existing idea and optimize our idea for agent. Also, the research direction and related work for XAI and our practical method of using ontology graph with NLP. Section 3 states the framework and methodology in theoretical and practical of agent, which include THREE levels for XAI Language Tutor for users, connectionism in language model with GPT-2 and ontology. Section 4 presents the implementation of agent at mini-program and the testing of constructed XAI Language Tutor in real world to analyse NLP raw data and interpreter, the ontology graph. Section 5 is to evaluate the performance combined with ontology for conclusion. Last Section is the future work to optimize the agent in both QA system and XAI field. 2. LITERATURE REVIEW In big data era, not only the Seq2Seq model of neural network promote the industrial applications, but also the traditional rule-based system upgrades their back-end and integrate more technologies to optimize the response from system. We will track back the functions of system to reflect the history and development in recent years. Furthermore, use definition currently to attribute these products by functions and technologies to assist the design and implementation of our agent. Other aspects are Linguistic and Ontology of NLP, we will go through the procedure of ontology and analysis the construction via linguistic and cognitive science. Due to the particularity of our XAI language tutor, we encountered situations that from user perspective, where requires unstructured raw data as source text for natural language learning. In technology, with the help of AI ecosystem, we need to develop a multistep functionoriented learning method as language agent. This is a process from data pre-processing to the end of user interface to implement our idea for agent. Although AI technology provide lots of framework to generate conversation responses, we always adhere to research direction and related work of XAI methodology and Ontology Graph (OG) with NLP. 2.1. An Overview of Chatbot 2.1.1. Extend contents for meaningful: Knowledge-Grounded Neural Model (seq2seq RNN) [4] Since the origin of seq2seq model generated by Recurrent Neural Network (RNN), neural network based chatbots had engaged in both industrial and academic communities. In 2018, Microsoft extended their industrial conversation system to make responses from the system to avoid brief and illogical contents as compared with human responses. In respect of design, Microsoft already possessed functions to give simple responses in open domain. Figure 2 shows the extension exist only in the branch of Encoder to add facts into response. Both versatility and scalability in open-domain and external information knowledge of textual and structured are combined in this system, which has the recommendation system function for restaurant but not task-oriented [4]. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 5 Figure 2. Response from Conversation model knowledge grounded After knowledge grounded, the system has more meaningful and logical contents in responses, which only infuse the knowledge information into the trained data-driven neural model. Based on this framework, they use 23M open-domain conversation from Twitter and 1.1M Foursquare tips to get a significant improvement over seq2seq model. Different with traditional content filling, which add the contents pre-defined and fill the slot in the sentence, this knowledge grounded architecture display in figure 3, which also using seq2seq model. Together with basic generation model, the source sequence and dialogue history, connecting to the texts by matching or entity links. Thus, both of them are in the same tuple for decoder, it viewed in Figure3 that dialogue encoder and facts encoder. Figure 3. Architecture of Knowledge-Grounded Model To be specific for the architecture and technique, the dialogue encoder is independent with facts encoder in system, it means the dialogue just choose a supplement of domain knowledge for specific the response contents in practical. 2.1.2. AliMe: Update rule-based model and integrate with Neural Network generation (rule-based IR+seq2seq) [17] In order to improve response quality to satisfy the requirements from 5% user, whose questions exceed the special-domain of AliMe, the commercial chatbot specialize in E-commerce industry, developers integrate both traditional information retrieval (IR) based and Seq2Seq neural models, so it called A Sequence to Sequence and Re-rank based Chatbot Engine. AliMe keeps their knowledge base with 9,164,834 QA pairs, meanwhile, add the neural network generation and optimizer for open-domain questions. After that, Seq2Seq model of GRU with SoftMax and Beam Search algorithms provided better response from neural network in both open and special International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 6 domain. Also, the Seq2Seq model used second time to re-rank candidate responses. The integration and performance show on figure 4 [17]. Figure 4. Top-1 accuracy of candidate approaches 2.1.3. Xiao Ice: Empathetic Social Chatbot（IR+seq2seq+KG） [3] Compared with chatbots mentioned before, Xiao Ice focus on the emotion analysis, which shows on figure 5. The key-points of Xiao Ice is comprised of Intelligent Quotient (IQ) and Emotional Quotient (EQ) into the system design. Figure 5. Xiao Ice Emotion analysis Figure 6. Examples of inconsistent responses generated using a seq2seq model In figure 6, the comparison shows that chatbot with emotion provided more natural and humananalogous responses. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 7 Generally speaking, classification for response, chatbot has divided into retrieval-based and generated-based. Xiao Ice has both candidate generator and candidate ranker. For the generator, it uses rule-based matching with real-world conversation collected and stored by natural language, another one is deep learning model trained by paired dataset to simulate human dialogue to build human-like system, the other is query to knowledge graph to get related entities. The candidate ranker also corresponding to the generator but including the semantic computation in NLP and empathy matching of Xiao Ice personality. It gives us inspiration to combine our language learning neural model with knowledge graph. In general, the knowledge graph used for retrieval response by query from knowledgebase. However, our research direction is cognition of XAI, we will use knowledge graph to explain the output sentence from neural network. We can combine these two directions by generator response of neural model but rank the sentence by information retrieval in knowledge base to match the relations to optimize the final response. It means that knowledge graph not only an explainer but also an optimizer with complete sentence for our response suitable for user preference. 2.1.4. Transfer Learning with GPT-2: Neural Network Based Conversational Agents [1] The above chatbots discussed are basically adopting the Seq2Seq model, after several years, Seq2Seq model already a mature technology in NLP as a language model or act as an optimizer. Since neural network is a data-driven model, it means that the performance are heavily relies on the amount and quality of the big data. In order to enhance the relations in sentence to get the conditional probability of next word, Seq2Seq model already use attention mechanism. Based on attention mechanism, Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. [6] Transfer Transfo we used as back-end language model in our agent is a fine-tuned dialogue format language model with a high-capacity Transformer model. [1] The fine-tune procedure also the process to tackle the natural language format from long-text to dialogue with short response. The prototype model of Transfer Transfo is pre-trained on document-level contiguous sequence and paragraphs with long-range information. After that, the fine-tune is augmented input representation and use a multi-task learning scheme for adjustment. During input representation, every input token including the embedding of word and position. Figure 7. TranferTransfo’s input representation In figure 7, the input token is concatenating all the persona sentences of same speaker as utterance and track back the history utterance of the dialogue to construct a transformer. By using the persona-chat dataset to fine-tune the model, it helps to shape communicator’s background to further define the topic and better understand user’s input sentence. It shows the priority in the different AI language tutor with different personality resemble Speaking module in IELTS. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 8 Figure 8. Example dialog from the PERSONA-CHAT dataset In figure 8, compare with traditional model, transfer learning shows strong empirical improvements in discriminative language understanding tasks. In neural network model, for QA system as language model, it really a big progress. This is why we chose transfer learning rather than seq2seq model to apply XAI. 2.2. eXplainable Artificial Intelligence (XAI) eXplainable Artificial Intelligence (XAI) is the methodology for the problem that the model or data cannot be understand or explain by human users, which is the human perception problem for the real-world applications. XAI is also a requirement for the models currently to suit the step of explainable with model high-quality performance. The flourishing of AI brings out the question that how we can make the progress for AI technology keeps the same step with its explanation on the outputs or decisions of algorithm rather than opposite. XAI is an interpreter to solve the question that the machine can understand the input data and get the output data human readable that means machine has ability to thinking like human. In NLP, it means Natural language Understanding (NLU). The NLU improvement is the progress of encoder part corresponding to end to end model. Natural Language Generation (NLG) is the verification to get expected responds from human. We will focus on NLU and NLG to make own agent more intelligence following the XAI methodology. To some extent, XAI decide the application direction of AI, which should be reliable and understandable, where reliable depends on the human perception in usage scenario. The research for visualize language model outputs and interaction of ontology and language model makes big progress in NLU and NLG. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 9 Figure 9. Different purpose with different target audience in scenario In figure 9, when we use AI technology on NLP, whether such algorithm can be trust and explainable for different audience and scenario is an important factor, which is also the core meaning of XAI. The following 5 parts covers the related XAI components in applications. [8] a) Transparency, which focus on readable format by human; b) Causality, whether Data-driven model provide both accurate inferences with decision background; c) Bias, Black-box cannot calculate model bias against complicated computation; d) Fairness, does AI system always operate in a fair manner to users. e) Safety, AI system output are understandable and trustworthiness in regulatory sectors. XAI transparency and compliance should be taken into account in association with a given related prediction [8]. In this article [8], Hani Hagras explained the Fuzzy Logic Systems (FLSs) and human understandable AI, FLSs tried to mimic human thinking and research on the approximate way to thinking rather than limit in human brain resemble to neural network, the precise way. It has upgraded to the philosophy to build numerous of if-then reasoning rules to describe given human behaviour in human-readable way. The rules are the highest reasoning format for inference called OWL in ontology. For KBQA system, knowledge reasoning is a developing area in ontology and natural language generation. From existed knowledge to generate new knowledge based on logical rules to make up the contents of dialogue system is to be solved. What we do visualize neural network using ontology is figure out the complicate relationships among numerous inputs and their interactions. That is a way of fuzzy logic to solve the question of XAI for the lay user in a human readable format and for researchers to analysed and augmented. 2.3.Natural Language Processing in Linguistic Natural language is relative to machine language for different audience, in Computational Linguistics, Natural Language Processing is the automatic processing of computer to deal with natural language of human. In abstract, NLP related to cognitive science, psychology and philosophy both epistemology and ontology. International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 16 4.2. Ontology Graph In order to build the Ontology corresponding to the random sample of human-machine interaction dialogue, we cooperate neo4j as graph database for visualization to create the nodes and links automatically. The dataset of Open AI GPT-2 is json format, so we first transform the json data to txt data for entities and relations extraction. Then we use Spacy internal language model to extract the Subject-Predicate-Object (SPO) triple as our potential knowledge, the information. Every entity contains a property or an extended range to expand knowledge to get more information based on web, where we choose DBpedia as our entity links. Because it is the constructed data from Wikipedia, which already build an OWL database. It will help our ontology graph link to the internet semantic web to enlarge our own database and extend the entities. After we have extracted SPO triple, the node and relations will connect to DBpedia and link with each other at neo4j platform. Figure 18. Test of Level 3 free-style conversation. Figure 18 shows the conversation in Level 3 free-style conversations. In this stage, the response from language model are truly unexplainable for human, where we cannot understand why the model give such meaning of response but it really a dialogue to response the questions from International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 17 users. As we see, first three sentences in figure 18 are general speaking about hobbies in music and reading. When we talk about work, the contents from general to special domain about animals. Considering the break out, with the help of search engine and relatively high IDF (inverse-document frequency) of document, so the TF-IDF score is higher, we find that the special noun ‘animal shelter’ is one of the entities to be explained. After that, we find the most similar paragraph with dialogue content of the response in figure 19, which will be the input to be extracted triple with Subject-Prrdicate-Object (SPO) form. Figure 19. the SPO sentence unextracted by Spacy. Figure 20. Extracted Triple. The extracted Triple shows on figure 20. After the Spacy deal with its pipeline tasks of the model to pre-processing the data for triple extraction, such as tokenizer, Tagger, lemmatization, coreference resolution, textacy will extract the SPO tiple and print all of them as list. The colourful text is the text following the S-P-O syntax but not recognized by Spacy. With my experience of protégé of Stanford University to build ontology graph manually, we decide to add the absent triple which sentence included in the demo paragraph with the Cypher commands in neo4j. The syntax of English has five basic sentence patterns: S-P-O, S-P-O-O, SP-O Object complement, S-intransitive verb, Subject-linking verb-predicative (SVP). The handcraft triple will obey these five basic rules. Even though the 40G pre-trained dataset of Open AI GPT-2, it only equips the transfer-learning model with fundamental grammar to generate suitable response with correct syntax. However, the meanings in linguistic is from the relations rooted by fine-tuned dataset between a linguistic form and communicative intent. Also, the contents and the length of sentence are controlled by fine-tuning part of Transfer Transfo with daily dialogue dataset. As we mentioned before, ‘animal shelter’ is the break out for Ontology construction. With data mining and search-engine technologies to find the paragraph, which contains most keywords. Ontology is the construct imitation of neural network in bionics and cognitive science in International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 18 philosophy. In addition, Ontology is the visualization of training dataset in both macroscopic for magnitude of dataset and microcosmic for the details of SPO triple meanings. The Figure 21 show the magnitude of the paragraph with corresponding ontology graph, which can be analysed at linguistic level for meaning in Ontology by NLP technology for industrial projects. Figure 22 is the details of similar meanings of output response for the comparison in fuzzy logic, where the advantage of transfer learning is the ability in syntax and semantic understanding for the pre-trained procedure with Big Data. Figure 21. ontology graph with whole triples In training dataset sample, it shows the paragraph we extracted 30 relationships also contains the content of dialogue. Such as the animals of horse, volunteer at animal shelter. The specialization is that the Subject entities (the brown bubble) is too little when compared with Objects (green bubble). Figure 22. Ontology graph visualizes the generated response International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 19 In order to prove the output sentence from XAI Language Tutor is explainable, we enlarge the related Triple extracted from sample paragraph. Due to our dialogue based on completely training data, the 200M json format data for fine-tuned. In general, only the rule-based pattern recognition can match the high-similarity sentence in database as response. However, the transfer learning Transfer Transfo model trained on end-to-end based neural network can generate similar response to express same meaning in Natural Language Understanding (NLU). However, compared with rule-based pattern matching, neural network considered as a black box. Table 1. Comparison of XAI Language Tutor generated and SPO triple XAI Language Tutor generated training data triple at ontology graph I’m a volunteer of the animal shelter I volunteer at animal shelter I like to ride horses I taking care of horse. I ‘ve been around dogs I am getting dog I have a big collection of books and artwork. ( the reminder from user has collect) I read book that is awesome that is awesome … … From the aspect of our research direction connectionism, neural network is the bionic research, our ontology belongs to philosophy but the structure resembles the connection in humans’ brain for the fuzzy definition of entities. As we see, the green bubbles represent different entities, but due to the same object, it will activate by the objects. At the beginning of this paper, we mentioned that we use ontology graph mimic the neural network. The local explanation visualization of training data shows the explanation of the output sentence and simulate the relationship in human’s brain for inference and cognitive science. The comparison of responses from XAI language tutor and ontology shows on Table 1. The size of ontology graph and language model restricted by the hardware. If the ontology extended with the completely training data, to some extent, with the help of TF-IDF for special words, we can explain more about the black box. 5. CONCLUSIONS In this article, following the eXplainable Artificial Intelligence, we design and implement a XAIbased Language Tutor using Ontology Graph (OG) and Transfer learning. For methodology, we apply connectionism into neural network and ontology for the structure and activation simulation. With the THREE layers in methodology, we complete two aspects tasks. For industrial, our XAI Language Tutor techniques refer to Natural Language Processing (NLP) especially in NLU and NLG, Transfer Learning and WeChat mini-program for the top layer of industrial application for systematically learning English. In research, we use Ontology, Search Engine technique to explain the random sample of human-machine interaction dialogue of realworld conversation. At experiment, we test several times with the three level to make sure that XAI Language Tutor can be used to English Spoken training in a silent environment. For the functions of language model, we get some experience to be verified that the pre-train stage for language model only equip with basic ability in English syntactic focusing on the form of natural language, in addition, the meaning of relations in connectionism and linguistic is from fine-tune dataset. We can conclude that the pre-trained is for grammar, but fine-tune is for meaning. From research contribution, our idea that use Ontology Graph to explain the outputs of International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 20 neutral network model of natural language sentences in fuzzy logic makes a progress in NLU. It means the neural network model besides the feature that Open AI GPT-2 generate text according to the reminder written in algorithm, the content of output sentence can be explain and visualize at the ontology graph with the same training dataset. The larger ontology graph contains more training data, the better and detailed explainable of output sentence. 6. FUTURE WORK There are several aspects for us to self-improvement, we will discuss about the agent system for English learning and Ontology for eXplainable Artificial Intelligence (XAI) separately. For the agent system implementation, as we see, the performance of agent is qualified to be a language tutor to communicate with users in real-world. The XAI language tutor currently, is an XAI-based Language Tutor. To be specific, it has much space to upgrade to a knowledge-based Question-Answer System (KBQA). In the future, with larger and meaningful dataset to be extracted as Triple to refresh Ontology and make inference with KBQA to generate new knowledge via OWL. Our object is to improve the agent at Java-based platform where it will clone itself for free-style conversation with different users. Following the XAI, for implement, we interpret random sample outputs from agent. To some extent, it means the neural network can explain for human thinking. Due to the dataset are daily dialogue of USA, it is hard to extract meaningful information as knowledge, so, the most of Triple we extracted are oral expression. If the fine-tune dataset in system implementation more grounded and meaningful, the ontology will be larger and can generate new knowledge with logical rules. Ideally, ontology will refresh with free-style conversation simultaneously, where NLP tasks extract the new conversation sentence from the dialogue to refresh new knowledge. In technical, with the lack of hardware and server, our XAI Language Tutor performance and capability to be further improved. In the future, our work to optimize the system display below: 1. Find a better server to push the project implement in real world. 2. Find or develop better triple extraction tool – due to Spacy cannot extract all triples and refresh the Ontology Graph with the human-machine interaction simultaneously 3. Inference new knowledge by fuzzy logic systems. 4. Train the GPT-2 even GPT-3 with meaningful dataset to be a KBQA system. 5. Construct Java Platform for agent. ACKNOWLEDGEMENTS The authors would like to thank for UIC DST for the provision of computer equipment and facilities. This project is supported by UIC research grant R202008. REFERENCES [1] Thomas Wolf, Victor Sanh, Julien Chaumond & Clement Delangue, (2019 ) “Transfer Transfo: A Transfer Learning Approach for Neural Network Based Conversational Agents,” in USA, Association for the Advancement of Artificial Intelligence, arXiv:1901.08149v2 International Journal on Natural Language Computing (IJNLC) Vol.9, No.5, October 2020 21 [2] Joshua B. Tenenbaum, et al, (2011) How to Grow a Mind: Statistics, Structure, and Abstraction,” Science 11 Mar 2011: 1279-1285. [3] Li Zhou, Jianfeng Gao, Di Li, Heung-Yeung Shum, (2019) “The Design and Implementation of XiaoIce, an Empathetic Social Chatbot.” arXiv:1812.08989v2 [cs.HC] [4] Marjan Ghazvininejad et al., (2018) “A Knowledge-Grounded Neural Conversation Model,” arXiv:1702.01932v2 [cs.CL] [5] Jay Alammar, https://jalammar.github.io/illustrated-gpt2/, access in 2020 [6] AshishVaswani et al, “Attention Is All You Need,” (2017) 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, arXiv:1706.03762v5 [cs.CL] [7] Barredo Arrieta, A., Díaz -Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., . . . Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115. [8] Hani Hagras, (2018) “Toward Human Understandable, Explainable AI,” Computer, 51 (9). 28 - 36. [9] Raymond S. T. Lee, (2020) Artificial Intelligence in Daily Life, Springer. [10] DB-Engines, accessed in 2020 https://db-engines.com/en/ranking/graph+dbms [11] Jiaxuan Li (2017) TensorFlow Technical analysis and practice, Posts and Telecom Press. [12] Raman Kishore, (2019), “https://medium.com/analytics-vidhya/a-knowledge-graphimplementationtutorial-for-beginners-3c53e8802377”, access in 2020 [13] Spacy, https://spacy.io/usage/spacy-101#architecture, access in 2020 [14] Neo4j, https://neo4j.com/developer/neo4j-browser/, access in 2020 [15] Edward H.Y.Lim, JamesN.K.Liu,and Raymond S.T.Lee (2011), KnowledgeSeeker– Ontology Modelling for Information Search and Management, ISBN 978-3-642-17915-0 [16] Jason Weston, Sumit Chopra & Antoine Bordes. (2015). Memory networks. ICLR, arXiv:1410.3916 [cs.AI] [17] MinghuiQiu et al., (2017) “AliMe Chat: A Sequence to Sequence and Rerank based Chatbot Engine,” Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 498–503 AUTHORS Clarissa N. B. Shi is graduated from Beijing Normal University - Hong Kong Baptist University United International College (UIC) awarded M.Sc. (Data Science) with distinction from Hong Kong Baptist University in 2020. Her research interests covering Ontology graph and Chatbot in Natural Language Processing, Neural Network with Mathematical and Explainable Artificial Intelligence (XAI). Qin Zeng is graduate from Beijing Normal University-Hong Kong Baptist University United International College (UIC) awarded his M.Sc. (Data Science) from Hong Kong Baptist University in 2020. Qin Zeng had worked in the software industry for decade before obtaining the master degree with rich experience in Linux environment, crossplatform, multi-language development and deployment. His research interests covering natural language processing, recurrent neural networks and intelligent mobile applications in WeChat platform. Dr. Raymond S. T. Lee (M’98) attained his B.Sc. (Physics) from Hong Kong University in 1989, M.Sc. (IT) and PhD (Computer Science) from Hong Kong Polytechnic University in 1997 and 2000 respectively. Dr Lee had worked at the Department of Computing of Hong Kong Polytechnic University as Associate Professor 1998 - 2005. During the past 20 years, Dr. Lee has published over 100 publications and the author of 8 textbooks and research monographs covering the fields of artificial intelligence, quantum finance, e-commerce, pattern recognition, intelligent agents and chaotic neural networks. Dr. Lee is now the Associate Professor in Beijing Normal University-Hong Kong Baptist University United International College (UIC) working in the field of quantum finance, quantum anharmonic oscillators, chaotic neural oscillators, fuzzy-neuro financial systems, chaotic neural networks and severe weather modelling and prediction.