Beyond The Conversation: Context-Fluid Experiences and Augmented Cognition

Have you ever felt like you were having a one-sided conversation with someone? It feels as if you are exerting much effort with minimal feedback or response in return.

When we use an application, we can think of this experience as a conversation between the user and the technology. Sometimes, it feels as if we are having that same one-sided conversation with the technology we are using. As modern people, we learn the ins and outs of the tech we are interacting with, from the information architecture to the layout of the UI elements. Because of this, we adapt to the technology. Just as we adapt to the technology, the technology should also adapt to us. This conversation should not be one-sided.

The metaphor of a “two-way conversation” isn’t new within the field of human-computer interaction; however, we shouldn’t take the metaphor at face value. Just think, when you are talking with a friend, the conversation you are having is most likely being affected by a specific state of affairs. The conversation is sensitive to a scenario. For example, the language, the cadence of speech, the volume, and other parameters may be more or less appropriate depending on the context of the conversation.

Context is KEY

As UX designers in a world with exponential advancements in sensing technology, we ought to have context at the top of our list of things to consider in any design process—especially if we are designing for a “context-fluid” experience.

A context-fluid experience is achieved when a person uses a product in a variety of scenarios and the actual experience of the product is heavily affected by the context of use. As designers, we are (or should be) aware of the importance of contextual design, such as how a UI might be presented differently to a person who is running compared to a person who is relaxing on a couch.

What if we take the idea of contextual design a few steps further? Context is not only defined by the time of day, activity, or physical location of the user. A very important component of context is the user’s psychological state. The context of taking a relaxing night drive to grab some ice cream is completely different than rushing to the hospital for an emergency in the middle of the night. How can we create truly context-sensitive systems for context-fluid experiences?  

Enter augmented cognition

The field of augmented cognition strives to manifest computer systems that can “sense” and interpret a user’s cognitive and physiological information, enabling the system to present information accordingly.

Just as humans have built machines to overcome our physical limitations, the field of augmented cognition aims to overcome our cognitive limitations within a human-computer environment. Our cognitive limitations lie within our inability to process massive amounts of information received from every direction (Wladawsky-Berger 2013). With consumer-focused augmented reality on the horizon, we can expect to be designing for not only AR but also for augmenting the human-computer experience by utilizing feedback loops and sensing technology.

Current sensing technologies

Cognitive sensing

For a technological system to adapt to a user’s current cognitive and physiological state, the system must be able to gather this information. A few methods of obtaining cognitive information from a user include functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and positron emission tomography (PET) (Ayaz et al., 2010).

However, these methods need cumbersome equipment and are very restrictive to the user. These methods are also extremely expensive, making it difficult and impractical for use outside of a laboratory setting. A more practical method of acquiring cognitive information is by means of functional near infrared spectroscopy (fNIR). fNIR can be employed to monitor changes in oxygenated and deoxygenated hemoglobin at the cortex of the brain.

With the implementation of fNIR, it is now possible to create more practical, portable, obtainable, and relatively inexpensive monitoring systems. There are newer systems available such as the Artinis OctaMon, with a headband-like form factor. Artinis also produces an even smaller monitoring device that is simply an electrode, connecting to a small battery pack. Both of these systems can wirelessly transmit data for real-time monitoring. These new developments would make it seem like fNIR is a suitable method for acquiring cognitive data in controlled laboratory settings as well as mobile contexts.

That said, these systems need to become even more affordable and further miniaturized to be adopted outside of their niche market.  For the time being, a more practical means of acquiring psychophysiological data is necessary for use in a consumer setting.  

Physiological sensing

Measuring a user’s physiological state is an easier and less invasive process than measuring cognitive states. There is a plethora of wearables on the market today that collect biometric data such as motion, heart rate, skin temperature, perspiration, and more. The advantage of physiological sensing technologies is that they’ve already been widely adopted—think Fitbit, Apple watches, and even Polar heart rate monitors to use at the gym. Furthermore, these sensing methods can also be used to infer a user’s psychological state. For instance there are products on the market today that measure a user’s rate of breathing to infer mood.  

With that said, biometric data may only result in a high-level inference of context. For example, by utilizing data such as heart rate, perspiration, and rate of breathing, I still may not be able to discern if the user’s psychological state is that of distress or exuberance. With this data, I may only be able to discern between “calm” and “excited” psychological states. This level of granularity may not be fine enough to design truly context-fluid experiences, so further research in this area is warranted.

Bringing it all together now

Currently, there isn’t a very cost-effective, widely used method of easily discerning a user’s psychophysiological state. That said, as forward-thinking designers and developers, we should be anticipating the ability to acquire and make use of this psychophysiological information.

What we can do currently, however, is think about how to best make use of the available data acquisition methods to create context-sensitive applications for context-fluid experiences.

As designers, it is our job to continue to facilitate and improve the two-way conversation between our technology and its users.  Let’s work toward creating meaningful feedback loops between human and computer, thus optimizing the context-fluid experience.


Ayaz, H., Willems, B., Bunce, S., Shewokis, P. A., Izzetoglu, K., Hah, S., Deshmukh, A., Onaral, B. (2010). Cognitive Workload Assessment of Air Traffic Controllers Using Optical Brain Imaging Sensors. In T. Marek, W. Karwowski & V. Rice (Eds.), Advances in Understanding Human Performance: Neuroergonomics, Human Factors Design, and Special Populations (pp. 21-32): CRC Press, Taylor & Francis Group

Wladawsky-Berger, Irving (2013) “The Era of Cognitive Computing” July 01, 2013. Accessed on May 20, 2014 from

Posted in Big Ideas | 1 Comment »

1 Comment

  • Tyler Lindell

    October 4, 2016 at 2:50 pm

    Very interesting indeed! To successfully produce context-fluid experiences, it could also be helpful to include eye tracking which is quickly advancing and may be widely available within the next 3 – 5 years. Having the ability to marry all of these data points together will likely also require heavy use of AI techniques, one that stands out here would be deep learning which requires very large data sets and can predictively evaluated a scenario based on limited information after training of the AI model has reached a certain point.

    Great article Cam!

Leave a comment