Connected Intelligence Engine

Role

User Research, Development and Interaction Design

Supervised by

David Karger, MIT CSAIL/ Kamal-Youcef Toumi, MIT Mechanical Engineering / Areej Alwabel, Center for Complex Engineering Systems / Sattam Alsubaei, Center for Complex Engineering Systems

Funded by

King Abdulaziz City for Science and Technology

Jumana Almahmoud, Najat Alrashed and David Karger. Lost and Confused in a Complex System: Based on a True Story from a Collaborative Research Lab. CHI 2018 Submitted. pdf

Challenge 

One of the main challenges that we face at the Center for Complex Engineering Systems is the integration of multiple models by feeding output of one as an input to another. The way this is currently solved at the center is to bring in model developers to write special purpose code to combine the models. However, our goal is to eliminate the need for the model developers and make it possible to specify those combinations visually. This motivated us to look into the challenges and investigate the possible solutions.

Solution

The proposed solution is the connected Intelligence engine (CIE) is a complex system that allows for joint data analytics such as simulation and modeling. It addresses both usability and performance issues that model developers at the center usually face, as mentioned in the introduction section. CIE aims to bridge the gap between the three phases workflow that is typically used when performing  any data analytics task: managing the data, analyzing it, and visualizing the analysis results.

As a first step, we did an exercise in which we have collected all the models that were developed at the center and documented the interactions between them. The following figure shows the documentation frame work to collect the list of models and the status of interaction between them.

The next figure bellow show the Models Interaction Matrix. The grey column and row are the list of models developed in the center, the colored cells between them are the interactions. The yellow cell represents “integrated” as in models that were connected by the model developers to solve a certain problem, the purple represents “potential interaction” as in models that the center wants to connect but not yet done, and the red represents “not applicable” meaning models that could not be connected together due to the incompatibility of their inputs and outputs. From that matrix, we could see the large number of models that have potential interaction between them to produce a valuable output. The potential interactions between the models were documented by listing all the inputs that feeds into one model, followed by the outputs that are produced from that model and used by the other model for the integration.

CIE

The CIE has three main interfaces the users need to go through in order to solve problems. The first interface that is presented to the users is the domains interface (see Figure 1.a) which includes the nine domains tackled at the center (Environment, Water, Labor etc.). This helps the users filter out the domains of interest. Users can choose one or more domains and then proceed to the next interface. 

The second interface is the network interface (see Figure 1.b). The users will be presented with a network for each domain they have selected. The network includes all the models and datasets in that specific domain. Each model added to the system has a unified metadata describing the model’s inputs and outputs to help the models connect together in the network. The metadata includes the model’s name, description, default values and a list of inputs and outputs.

The nodes of this network are the models and the datasets, the edges represent the interactions between those models showing the flow of dataset between the models. Those edges indicate to the user the possibility of connection between models i.e. there is an edge from model “X” to model “Y” because model “X” outputs dataset “Z” which is input to model “Y”. The way models are presented in the network and the direction of the edges imposes the order in which the models need to be selected and executed.

The network interface gives the user visibility of all the models and datasets in the relevant domain and the connections between them. Users interact with the network by selecting the nodes (models or datasets) they are interested in. After selecting the nodes, users double click on the last node of interest creating a path of models and datasets. The system will automatically trigger the models starting from the first node that was selected and ending with the last node that was double clicked, including the nodes in between whether the users have clicked on them or not. Other nodes that are not in the selected path will be ignored.

After selecting the models and datasets, users will continue to the next interface to control the decision variables (see Figure 1.c). In this step, users are asked to fill the input form or use the default values provided by the model developer(s) in order to trigger and run the models to produce the final output. To provide guidance to the users, CIE provides hints on each interface and the ability to hover over the nodes in the network for description about the model or dataset.

System Design 

The design started with classic HCI design approaches by observing the main prospect actors of the system. Followed by collecting data and documentation for the system components (models and datasets). After that the requirements were gathered and CIE was developed to be tested to generate design reflections for the first iteration. Some of the main design requirements are to show the users possible paths to reach a solution, to provide visibility of all the models and datasets in the system, to help reduce the cognitive load on the user, to assist the users in making sense of the datasets, to enable users to trace and understand paths they chose for solving their problems and finally to minimize errors caused by users. Throughout the design process, a framework inspired by the work done by Chilana et al. in which we involved the domain expert(s) throughout the design process.

Usability Study

A task-based observational study was conducted with ten participants with varied educational backgrounds and levels of experience in data modeling and visualization (see Figure 4). They all were expert computer users. Four tasks were selected to represent the available functions in the platform (running models from one domain or multiple domains): 1- You are an engineer who is interested in knowing the traffic in Riyadh during rush hour. What would you do using the platform? 2- You are interested in looking at the default design of the power grid covering Riyadh, what would do using the platform? 3- You want to visually compare between social behavior and traffic, using the platform what would you do? 4- You are interested to see how people move around and how does that affect energy consumption.

 The numbers and our observations show that as the level of experience increased the number of tasks completed and SUS (System Usability Scale) scores increased (figure 5). The researchers encoded the participants’ reactions to the user experience of the three main interfaces into two main categories: Satisfied and Confused. The first interface of the domains and the last interface of the input forms received satisfied responses. However, the interface with the network of models and datasets received more confusion by the participants.

The expectations of a network page were not met when the users interacted with the interface. Some affordance problems were detected when users did not know how to choose from the network. For example, this user expressed while performing task # 2: “I don't know which node is considered the final node! but since my final chosen node is power grid I’ll double click on it”. This user was able to visually explore the network but in order to interact with it he read the instructions provided at the header of the interface, in which they were asked to” double click on the final node of interest to proceed”. We expected that the network will help the users trace the data flow between the models through the direction of the edges and be able to interact with the models in a certain order. However, interacting with the network was a huge source of confusion in our study.

Other users were surprised when they discovered that the system will provide the integration for them. For example, this user after clicking on two nodes while performing task # 4 surprisingly expressed “Ah the system will handle the connection for me?!”. This user was on her fourth task and she had already successfully completed her previous tasks. However, this task involved two domains (see Figure 3) that were colored differently within the same network which could be the source of her confusion.

Although some users relied on the cues that we have provided on the interface to help them navigate the system. Most of the hints ended up not being noticed except for the network interface in which some users noticed after being frustrated. Some users expressed that it was not easy to locate the cues, for example “Too much visuals made me not read”. Finally, the tasks that were given to the users were either to control models from one domain or multiple domains. The tasks required connecting two domains received more number of errors and less number of tasks completed.

As shown in the results of our study, the interactions observed between the users and the network interface indicated moments of discomfort. An interesting pattern merged indicating a deficiency in the level of collaboration provided by that interface. The network of models and datasets was designed to give the users the ability to view all the models and datasets in a certain domain, yet users found it confusing and had to go through training in order to understand how to interact with it. Users were able to visually explore the network but in order to interact with it they read the instructions provided at the header of the interface. We expected that the network will help the users trace the data flow between the models through the direction of the edges and be able to interact with the models in a certain order.

Unfortunately, while the network approach was aligned with the three main pillars of a complex system, the usability of this approach was not what we hoped for. Trying to simplify the navigation in a complex system of connected models by using the network representation added complexity to some users while they were interacting with the system.

For More Info check out our published case study in CHI, here