Dialogue Management in a Home Machine Environment: Linguistic Components over an Agent Architecture

Dialogue Management in a Home Machine Environment: Linguistic Comp onents over an Agent Architecture Jose F. Quesada, Federico Garca, Esther Sena, Jose Angel Bernal, Gabriel Amores Grup o de Investigacion Julietta Universidad de Sevilla Abstract This pap er presents the main characteristics of an Agent{based Architecture for the design and implementation of a Sp oken Dialogue System. From a theoretical p oint of view, the system is based on the Information State Up date approach, in particular, the system aims at the management of Natural Command Language Dialogue Moves in a Home Machine Environment. Sp ecically, the pap er is fo cused on the Natural Language Understanding and Dialogue Management Agents, and discusses their integration over a global agent architecture (which includes Action and Knowledge Managers, Sp eech Input/Output comp onents and HomeSetup controllers). 1 1 Introduction Sp oken Dialogue Systems have b ecome one of the most prominent research areas in Computational Linguistics. Over the last few years, several pro jects all over the world have focused on this topic. This pap er describ es the main characteristics of a sp oken dialogue system aimed at the control of a lightning scenario in a home machine environment. From a functional p oint of view, the system may b e describ ed as a:  Sp oken Dialogue System, which acts as a natural language interface b etween users and  Home Devices installed in a Home Machine Environment  designed and implemented over an Agent Architecture (sp ecically, 1 This work has b een supp orted by D'Homme (Dialogues in the Home Machine Environment), EC Pro ject IST-2000-26280. The authors would liketo thanks the rest of the pro ject partners for their comments and suggestions on the work. OAA - Op en Agent Architecture [Martin et al 99]). Section 2 describ es the overall architecture and agents, mainly the HomeSetup, KnowledgeManager, ActionManager, Text- To-Sp eech and Dialogue Manager Agents. One of the main goals of the Natural Language Understanding comp onent of the system is b oth to detect the command and the device that the user wants to control and to set the level or p ercentage that the user sp ec- ies at a precise moment. For instance, the sentence "Set the kitchen light at level 5" , must be transformed into a semantic representation, where the main features will b e:  Command :Set  Device : Light  Lo cation : Kitchen  Level :5 Taking this information as its input, the Dialogue Manager must be able to detect if the semantic structure is:  Functionally complete (Which light do you want to turn on if you have just received "Switch on the light" ?)  Pragmatically coherent and technically p ossible (you cannot switch on a do or, for instance). Besides, the Dialogue Manager must clarify ambiguous expressions, such as "Switch on the living room light" , when there are several lights in the living ro om. In order to solve these problems, the Dialogue Manager is closely connected to b oth the Natural Language Understanding and the Knowledge Manager mo dules. Next, we present the linguistic comp onents of the architecture, that is, the Natural Language Understanding and the Dialogue Manager Agents. At this level, it is worth noting that the system allows the following functionality:  Natural Command Languages  Exp ectations  Multiple Commands  Default Interpretation A detailed description of the kernel of the Dialogue Manager may be found in [Quesada et al 00, Quesada, Torre & Amores 2000 ], including the sp ecication language for the description of dialogue systems and the strategies for the implementation of the previous functions, mainly the manipulation of exp ectations and the use of default interpretations. Section 3 describ es the Natural Language Understanding mo dule, paying sp ecial attention to the grammatical level: a unication{ based semantic{oriented grammar sp ecication. The interface b etween the Natural Language Undertanding and the Dialogue Manager agents is based on the DTAC proto col ([Amores & Quesada 2000 ]). Section 4 includes an interesting set of examples incorp orating for each example the DTAC representation of the sentence obtained by the grammar presented in section 3. Finally, section 5 highlights one of the main challenges of the system: to obtain a natural language interface to the Home Machine Environment. This means that a user must be able to install, plug or unplug, and con- gure new or previous devices and to work and talk with them on the y. This section includes some examples which illustrates the main functions supp orted by the system so far. 2 Overal l Architecture and Agents The system has b een implemented using the Op en Agent Architecture [Martin et al 99]. Figure 1 illustrates the main agents apart from the Facilitator, the central agent in charge of the communication between the rest of agents. Figure 1: Overall agent architecture.  HomeSetup : This agentisincharge of the simulation of a Home Machine Environment. This agent mainly allows the installation of new devices and their con- guration. As part of the conguration the user may instantiate the set of descriptors related to the device.  KnowledgeManager : This agent contains b oth the static (structure of the house, general ontology, ...) and dynamic (installation, conguration and status of the devices) knowledge involved in the task.  ActionManager : The current devices will supp ort the actual functionality of the system. This functionality (turn on/o the device, ask the current status, change the level of a dimmerable device, ...) must b e accessed through the Home- Setup agent, in charge of the simulation and monitoring of the scenario. Nevertheless the devices incorp orate primitive functions to the system. The Action- Manager incorp orates new higher level functionality that will b e translated into sequences of primitive actions.  Text-to-Sp eech : In charge of the generation of the sp eech output. In order to do so we have develop ed an OAA- wrapp er for Festival, allowing in this way the use of this system as part of the general architecture.  Dialogue Management : This agent includes the User Interface (which controls at a high level the communication with the user), the Parser, the Semantic Interpreter and the Dialogue Manager comp onents. The Knowledge Manager is in charge of storing and organizing the information related to the devices installed in the house regarding ubication, characteristics and relations with other devices. The approach adopted for organizing this information is based on the conformation of a semantic network. The prop erties of this network are:  No des organized in a graph top ology  Nondirected connections between no des.  Nonconexed.  Noncyclical.  Parent-child relation dened in each conection b etween no des. Each no de contains a list of comp onents. A comp onent consists of a pair of strings: a lab el and a value. We can describ e thereby the structure of the network as having two relations:  A comp onent b elongs to a no de. component  node  A no de includes other no de. node 1  node 2 e.i. node 1 is parentof node 2. Let see an example of semantic network: ground < indoors ground < outdoors indoors < kitchen indoors < den indoors < corridor indoors < living_room outdoors < front outdoors < back where the symbol  is represented by < . The gure 2 represents the graph conforming the network sp ecied. Figure 2: Graph representation of a semantic network example. 3 Natural Language Understanding Dialogue systems enable users to interact with computer systems via natural and intelligent dialogues, as they would with human agents. Development of such systems requires a wide range of sp eech and language technologies, including natural language and dialogue pro cessing, to determine the meanings and intentions of the utterances and to generate a text-to-text interaction between the user and the system. As regards this do cument, what we have done is to develop an interface for b oth English and Spanish for the sp ecication of different functions in an intelligent house scenario. In order to do so, we have made use of Natural Command Languages. Natural Command Languages comprise the set of input and output natural language expressions which are acceptable in a given application domain. The most outstanding characteristics of these NCLs are the use of natural language vo cabulary and syntactic structure. Also, NCLs reect natural syntactic order and more precisely the patterns of sp oken language. In fact, they are a sub-language, with reasonably clear b oundaries but also one in which new functionality can b e added without altering the structural prop erties of the language. Next subsection presents the main rules of the unicationbased grammar that has b een implemented. At this level it is worth noting that the grammar incorp orates Non{terminal typication, and includes rules to construct a Natural Command Language Dialogue representation mo del, based on Dialogue Moves where DMOVE represents the dialogue move itself [Amores & Quesada 2000 ], TYPE the sp ecic typ e of dialogue move, ARGS the functional sub categorization mo del of the typ e of dialogue move and CONT the actual contents of terminal structures. 3.1 Unication{based and Semantic{oriented Grammar RootsOfGrammar: // Parameters Value Device // Commands Command This is a conguration parameter used by the parser, in fact, it is the only one that the parser uses. On the other hand, the unier is the one in charge of the rest of the parameters. By means of these Ro otsofGrammar, all grammar symb ols that may b e treated as ro ots are sp ecied. This in turn let us have grammars with several ro ot symb ols. NonTerminalTypes: // Parameters Device = (DMOVE: specifyParameter, TYPE: Device) Number = (DMOVE: specifyParameter, TYPE: Value, SUBTYPE: Number) Level = (DMOVE: specifyParameter, TYPE: Value, SUBTYPE: Level) Percentage = (DMOVE: specifyParameter, TYPE: Value, SUBTYPE: Percentage) // Commands CommandOn = (DMOVE: specifyCommand, TYPE: CommandOn, ARGS: [Device]) CommandOff = (DMOVE: specifyCommand, TYPE: CommandOff, ARGS: [Device]) CommandIst = (DMOVE: specifyCommand, TYPE: CommandISt, ARGS: [Device]) CommandSet = (DMOVE: specifyCommand, TYPE: CommandSet, ARGS: [Device,Value]) CommandInc = (DMOVE: specifyCommand, TYPE: CommandInc, ARGS: [Device,Value]) CommandDec = (DMOVE: specifyCommand, TYPE: CommandDec, ARGS: [Device,Value]) AskHelp = (DMOVE: askHelp, TYPE: AskHelp) Quit = (DMOVE: quit, TYPE: Quit) Under the lab el NonTerminalTyp es, we have b oth parameters and commands. Both typ es will b e the ones that will app ear in the DTAC structures that the Dialogue Manager will take as its input. This can be clearly noticed in section 4. As regards parameters, there are four, namely device, p ercentage, numb er and level. The lab el device may stand for words such as \kitchen", the lab el number stands for a plain number such as \5", and the word level along with p ercentage, indicate either a level such as \at level 5", or a p ercentage such as \to 25 p ercent" resp ectively. These parameters will b e instantiated as the arguments of the commands, but they will b e enclosed under the lab el \value", that is, this value together with the device, will be what will actually app ear as the argument of the command. Therefore, p ercentage, numb er and level will b e value subtyp es. Wehave created this construct to avoid ambiguities in b oth the grammar and the semantic representation. Commands are divided into eight typ es. The rst group of commands (CommandOn, CommandO, and CommandIst) takes a device as its argument. At this p oint, the user may ask the system to switch on a light, to switch it o, or he may just b e asking for the status of a light. The second set of commands are the ones used with dimmerable devices, therefore they take a device plus a value. They are used to set a lightat any level, p ercentage or numb er, to increase it, or to decrease it. The third set consists of two commands, one of them for asking help, and the other one for quiting the system. As it can be noticed b elow, the do not require any parameters. /////////// PARAMETERS (A100 : AuxDevice -> LAuxDevice) f @up = @self-1; g (A101 : AuxDevice -> AuxDevice LAuxDevice) f @up = @self-2; g (P100 : Device -> LKeyDevice ) f @up = @self-1; g (P102 : Device ->AuxDevice LKeyDevice ) f @up = @self-2; g (L100 : AuxPerc -> LAuxPerc ) f @up = @self-1; g (L101 : AuxLevel -> LAuxLevel) f @up = @self-1; g (L102 : Number -> LNumber) f @up = @self-1; g (L104 : Level -> AuxLevel LNumber ) f @up = @self-2; g (L105 : Level -> LNumber AuxLevel ) f @up = @self-1; g (L106 : Percentage -> LNumber AuxPerc ) f @up = @self-1; g (L107 : Percentage -> AuxPerc LNumber ) f @up = @self-2; g (L110 : Value -> Number ) f @up = @self-1; g (L111 : Value -> Level ) f @up = @self-1; g (L112 : Value -> Percentage ) f @up = @self-1; g (L120 : Args -> Device ) f @up.Device = @self-1; g (L121 : Args -> Value ) f @up.Value = @self-1; g In this part of the grammar, we can see the rules with the functional equations asso ciated to them. All of them are parameter rules. Words b eginning with L are terminal categories. For instance: LAuxDevice, LAux- Perc,LNumb er,LKeyDevice. We have words such as \level", \p ercentage" and \light", which b elong to the corresp onding categories \AuxLevel", \AuxPercentage" and \ AuxDevice". Also, as device is a non-terminal typ e, \LkeyDevice" is the lexical category standing for words such as \kitchen". An Args can b e either a value or a device, and a value can b e either a level or a p ercentage. The creation of these constructs and the way in which the rules have b een implemented, allows us b oth to have any kind of word order and to avoid grammatical ambiguity. /////////// COMMANDS (C200 : CommandOn -> LCommandOn) f @up = @self-1; g (C201 : CommandOff -> LCommandOff) f @up = @self-1; g (C213 : CommandIst -> LAuxIstB) f @up.TYPE =a CommandISt; g (C214 : CommandIst -> LAuxIstA LAuxIstB) f @up.TYPE =a CommandISt; g (C215 : CommandIst -> LAuxIstB LAuxIstC) f @up.TYPE =a CommandISt; g (C202 : CommandSet -> LCommandSet) f @up = @self-1; g (C203 : CommandInc -> LCommandInc) f @up = @self-1; g (C204 : CommandDec -> LCommandDec) f @up = @self-1; g (C211 : XCommand -> CommandOff ) f @up = @self-1; g (C212 : XCommand -> CommandIst ) f @up = @self-1; g (C213 : XCommand -> CommandOn ) f @up = @self-1; g (C220 : XCommand -> CommandSet ) f @up = @self-1; g (C221 : XCommand -> CommandInc ) f @up = @self-1; g (C222 : XCommand -> CommandDec ) f @up = @self-1; g (C223 : Command -> XCommand ) f @up = @self-1; g (C224 : Command -> XCommand Args) f @up = @self-1; @up = @self-2; g The way in which the command rules have b een arranged allows us to have any combination of words, no matter the order in which they are either said or typ ed. This is a crucial p oint, sp ecially if we have into account that this is intended to be sp oken, and sp oken language is by no means predictable or xed as regards word order. If we start from the lexical categories, which are the ones b eginning with L, we get the Command*. The * stands for On, O,Set, Inc, Dec. As we can notice, all these are commands with arguments and they can b e of the same typ e as the ab ovementioned parameters.They can b e describ ed as sp ecic commands. We may have any combination of words, sp ecically when the user wants to ask for the status of a light. He may simply say \Estado de la co cina", or \en que estado se encuentra la co cina".In the later, the two keywords would be \estado" and \encuentra", and in the former, the keyword would b e would b e \estado". All the Command* are XCommands. These are commands without arguments. Finally, the XCommand is inserted within a more general command which is just a Command. /////////// ERROR REPAIR (N001 : Command -> Command No Args) f @up = @self-1; @if (@self-3.Device) @then f @up.RepairedDevice = @self-1.Device; @up.Device =a @self-3.Device; g @if (@self-3.Value) @then f @up.RepairedValue = @self-1.Value; @up.Value =a @self-3.Value; g g (N002 : Command -> Command No Command) f @up = @self-3; @if (!(@self-3.Device) && (@self-1.Device)) @then f @up.Device = @self-1.Device; g @if (!(@self-3.Value) && (@self-1.Value)) @then f @up.Value = @self-1.Value; g @up.RepairedCommand = @self-1; g These rules are used when the user makes an error and he inmediately corrects himself. He may b e repairing an argument whichmay b e a level, p ercentage, or a device, he maybe changing the command, that is to say,hemay be asking the system to turn o the kitchen light when he has just realized that that light is already o, and he may be changing b oth the command and the argument. This is clearly exemplied in section 4, in examples 4, 5 and 6. In numb er 4, the user is changing the command, in number 5, he is changing b oth the device and the value, and in number 6, he is repairing the command and the value. 4 Semantic representation of dialogue moves Next, we present some examples which illustrate the use of the previous grammar. These results have b een obtained with the Episteme NLP to ol ([Quesada & Amores 2000 ]). The goal of this section is to illustrate the generality and simplicity of the representation formalism with a wide range of phenomena: incomplete commands, commands with dierent arguments, dierent typ es of arguments (numb ers, descriptors, ...) and error repairs, b oth of commands and parameters: 1. Turn on 2 4 DM OV E : specif y C ommand TY PE : S w dC ommandO n ARGS : [ S w dD ev ice ] 3 5 2. Turn on the kitchen light 2 6 6 6 6 6 6 6 6 4 DM OV E : : specif y C ommand TY PE : : S w dC ommandO n ARGS : [ S w dD ev ice ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : k itchen 3 5 3 7 7 7 7 7 7 7 7 5 3. Dim the kitchen light down to 25 p ercent 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand TY PE : S w dC ommandD ec ARGS : [ S w dD ev ice ] ; [ S w dV al ue ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : k itchen 3 5 S w dV alue : 2 6 6 4 DM OV E : specif y P ar ameter TY PE : S w dV alue CON T : 25 SU BT Y P E : S w dP er centag e 3 7 7 5 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 4. Increase the kitchen lamp no no decrease it 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand TY PE : S w dC ommandD ec ARGS : [ S w dD ev ice ] ; [ S w dV al ue ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : K itchen 3 5 E r r or C ommand : 2 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand TY PE : S w dC ommandI nc ARGS : [ S w dD ev ice ] ; [ S w dV al ue ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : K itchen 3 5 3 7 7 7 7 7 7 7 7 5 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 5. Increase the lamp in the kitchen to level 10 no no no the light in the dinning ro om to 25 p ercent 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand TY PE : S w dC ommandI nc ARGS : [ S w dD ev ice ] ; [ S w dV al ue ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : l iv ing r oom 3 5 S w dV alue : 2 6 6 4 DM OV E : specif y P ar ameter TY PE : S w dV alue CON T : 25 SU BT Y P E : S w dP er centag e 3 7 7 5 E rrorS wdDevice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : K itchen 3 5 E rrorS wdV alue : 2 6 6 4 DM OV E : specif y P ar ameter TY PE : S w dV alue CON T : 10 SU BT Y P E : S w dLev el 3 7 7 5 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 6. Set the light in the kitchen at level 5 no no no increase it to 25 p ercent. 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand TY PE : S w dC ommandI nc ARGS : [ S w dD ev ice ] ; [ S w dV al ue ] CON T : S w dV alue : 2 6 6 4 DM OV E : specif y P ar ameter TY PE : S w dV alue CON T : 25 SU BT Y P E : S w dP er centag e 3 7 7 5 S w dD ev ice : 2 4 DM OV E : specif y P ar ameter; TY PE : S w dD ev ice CON T : k itchen 3 5 E r r or C ommand : 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 DM OV E : specif y C ommand; TY PE : S w dC ommandS et ARGS : [ S w dD ev ice; S w dV al ue ] CON T : S w dD ev ice : 2 4 DM OV E : specif y P ar ameter TY PE : S w dD ev ice CON T : k itchen 3 5 S w dV alue : 2 6 6 4 DM OV E : specif y P ar ameter TY PE : S w dV alue CON T : 5 SU BT Y P E : S w dLev el 3 7 7 5 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 5 Natural Language Dialogues Finally, we present some dialogue examples which illustrate the main functions of the system from a linguistic p oint of view: 5.1 Natural Command Languages and Exp ectations Scenario: Two lights: one in the kitchen and the other in the corridor. Both are turnedo at the beginning Dialogue 1  User: Turn on.  System: What device would you like to turn on?  U: The kitchen light.  S: The light in the kitchen has just b een turned on. In this example, the system detects that in order to turn on a device, the user must supply the descriptor of the actual device. This functionality is implemented as a exp ectation of the dialogue manager ([Quesada, Torre & Amores 2000 ]). 5.2 Multiple Commands Dialogue 2  U: Turn o the kitchen and switch the corridor on.  S: The light in the kitchen has just b een turned o.  S: The light in the corridor has just b een turned on. Althought the output generated by the system must be improved in order to obtain just one sentence, this example illustrates one of the most complex phenomena in dialogue management: the user asks the system to execute several commands in one utterance. 5.3 Exp ectations, previous commands and default interpretation Dialogue 3  U: Turn on the corridor.  S: The light in the corridor is already on.  U: Sorry, the kitchen light.  S: The light in the kitchen has just b een turned on. This example shows the integration of the dialogue manager with the knowledge manager, which has detected that the corridor lightwas already on. Besides, the example illustrates an ambitious dialogue strategy: the default interpretation of a parameter as part of a previous command. In this scenario, the sp ecication of the parameter is analised as an exp ectation of the previous command. 5.4 Plug & Play Dialogue 4  U: What is the current state of the living ro om light?  S: The light in the living ro om is not available at the moment.  Using the HomeSetup Agent, the user instal ls, congures and plugs a new light at the living room.  U: Now, turn the living ro om on.  S: The light in the living ro om has just b een turned on. Taking into account that the 4 previous sub dialogues b elong in fact to just one dialogue obtained from the real system, it is worth noting the ability to supp ort plug & play capabilities without restarting the dialogue manager nor the rest of agents. 6 Conclusion A dialogue system designed to supp ort natural and exible dialogues intended to control a lightning system in a home machine environment has b een presented. The pap er has concentrated on the linguistic comp onents of the agent{based architecture: the grammar and the semantic representation of complex sentences. A set of dialogue examples illustrating the linguistic capabilities of the system have b een presented and describ ed. References [Amores and Quesada 1997] Amores, J.G. and J.F. Quesada. 1997. Episteme. Procesamiento del Lenguaje Natural 21 . pp. 1-16. [Amores & Quesada 2000] Amores, J. G., Quesada, J. F. 2000. Dialogue Moves in Natural Command Languages. Deliverable 1.1. Siridus pro ject.