Interactive Technologies and Future Living Spaces

Carsten Busch, André Selmanagić and Martin Steinicke

  • Applied Interactive Technologies denote hardware and software technologies from the games and consumer electronics industry that can be or have been used individually or in combination to solve a problem or to serve a use case in a non-entertainment context.
  • APITs show great potential in various non-entertainment contexts, e. g. in future living spaces (wearables as data-sources for smart cities); industry (smart glasses to assist maintenance works); health (motion sensors to support the rehabilitation process of stroke patients).
  • The Creative Media R&D Group is always looking for partners and challenges for future projects.

Abstract

Interactive technologies have mostly been developed for the games and entertainment industries. But these technologies offer enormous possibilities for other fields of application as well and can influence work and life beyond gaming contexts. This paper describes the concept of Applied Interactive Technologies (APITs) and outlines various potential fields of applying interactive technologies in non-entertainment contexts – from the industrial sector to future living spaces. It presents existing prototypes as well as practical and potential use cases for these fields of application. The paper closes with introduction of the application centre “creative Applied Interactive Technologies (cAPITs)” and the “ProWear” cooperation network. Both will foster the application of APITs and provide training and network facilities for the games sector and/with commerce & industry to enable co-creation in cross-cluster projects.

1. Introduction

The “Forschungs- und Weiterbildungszentrum für Kultur und Informatik” FKI (Research and Training Centre for Culture and Computing) at the HTW Berlin is the geographical and mental home of the Creative Media Research and Development Group led by Prof. Dr.-Ing. Carsten Busch. Creative Media (CM) focuses on themes and challenges that connect to at least one of the three dimensions: interactive technologies and media, learning and teaching as well as brand creation and communication.

Projects that fall into this three-dimensional space are of inherent interest to CM. Through this space we generally weave the application of game concepts and technologies to support the developed concepts and solutions. The interest in games as a cultural media as well as their application to other contexts – e.g. for teaching and learning – has always been present in Creative Media.

The research project “Realitätsnähe und symbolische Interaktion bei Computerspielen und Online-Games” (RSI, 2007–2010) furthered this interest and the conviction that concepts and technologies from the games industry hold a huge, even disruptive, potential in quite a number of sectors and fields of application (Busch, 2010). In RSI a number of approaches had been combined, but especially the semi-structured interviews of managers and experts hinted at low-hanging fruits as well as future scenarios – from using game engines for virtual tourism to using wearables in future living spaces.

Elsewhere (Busch, 2014) we have argued for a comprehensive definition of gamification that encompasses the application of both game concepts, or methods, and technologies. This was due to our longstanding belief in the innovative potential of using all of them – potentially, but not necessarily combined – to enrich non-entertainment contexts.

2. Gamification

In the past few years, gaming concepts that were originally developed in and for the games industry have increasingly been deployed in other industries and non-gaming environments. There they are used to reshape and unitize processes of production, work, learning and teaching. In its simplest form “players” can gather points or collect badges for completing tasks, compete against or collaborate with each other to advance to another, more complex level of the gamified system. Through these concepts, formerly tedious steps of work can be gamified and thus may become simply more fun, which again leads to increased motivation of employees and customers alike, heightens customer retention, or makes learning outcomes more long-lasting (Burke, 2014).

Quite a number of definitions focus on a rather narrow approach to gamification, but there are exceptions, too: “Gamification is using game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems.” (Kapp, 2012)

No matter whether a narrow or broader approach is applied, gamification in general is an experience and motivational (design) approach (Busch and Steinicke, 2017).

3. Applied interactive technologies (APITs)

In the course of the Gamification hype and its problems as well as due to the discussed focus on designing a gamified experience, we started to focus on an alternative concept, when considering the potential of applying game technologies. We would like to shortly introduce the – i.a. in Busch and Steinicke (2017) – developed concept in the following paragraphs.

3.1 APITs – a definition

As stated above, through Gamification, gaming concepts have found their way into non-gaming industries and contexts. But beyond these concepts, there is also a wide range of gaming technologies that have not yet been applied equally successfully outside the realm of gaming itself. At the current state, the full potential of these gaming technologies, which we will hereafter call Applied Interactive Technologies or APITs, has not been tapped yet in the same way that Gamification has been.

But which technologies should be considered – potential – APITs in the first place? Because new (problem) contexts and new technologies emerge all the time and the unique potential of individual technologies changes in the course of their lifecycle, it is not feasible to create a final list of technologies that could be applied to non-gaming contexts. Thus we proposed the following definition of APITs: “Applied Interactive Technologies denote hardware and software technologies from the games and consumer electronics industry that can be / have been used individually or in combination – often contrarily to their original purpose – to solve a problem or serve a use case in a non-entertainment context.” (Busch and Steinicke, 2017.)

Figure 1: The synthesis of Applied Interactive Technologies (Busch and Steinicke 2017)

Below the individual parts of this definition will be highlighted and potential fields of application and the potential of applying APITs in use-cases that originate in these fields will be described.

3.2 APITs – hardware game technologies

Due to the current Virtual Reality (VR) hype, the games industry’s most prominent hardware is certainly the Oculus Rift and its competitors. Developed for experiencing games and applied to an increasing degree to other entertainment products like 360° videos, VR hardware also enters prominently into other sectors and markets. But the field is much broader: All kinds of sensors and controllers (e.g. the Kinect or the Wii) that can now, amongst other things, be used to track employees in high risk spaces in factories, enable robots to perceive and to interact with humans, or calculate the heartbeat of a person without wiring them (Microsoft, 2015) are open to creative exploitation. In general (successful) hardware from the games sector is very well designed for its purpose and due to positive scale effects extremely cheap compared to developing such hardware in-house.

3.3 APITs – software game technologies

Today the production of digital games is a billion dollar business with teams of some hundred or even thousand individuals involved in producing AAA games (Middelberg, 2013; Gilbert, 2014). Naturally, the professionalization of the games industry led to the development of quite a number of different tools, frameworks and middleware that specialize in the support and acceleration of such development processes. The most prominent of these are certainly the engines. Graphics engines, sound engines, game engines and world editors can help developing immersive and interactive media content, but can just as easily be used e.g. as a technology for rapid prototyping. Additionally, there are now several specialized middleware and frameworks that excel in providing one specific feature to game developers. These kinds of solutions are ideal candidates to be applied to non-entertainment contexts, too (Busch and Steinicke, 2017).

Due to the fact that the goal in developing software game technologies in the first place is to accelerate development, these are generally designed for ease of use and an adequate learning curve. Additionally – with advent of the Indie-game development movement and the resulting battle for market shares – these software solutions are now quite reasonably priced or even free of charge.

3.4 APITs – consumer electronics

In addition to these – one might say – core gaming technologies, Busch and Steinicke (2017) proposed to include the application of consumer electronics to non-entertainment context into the APITs concept, too. This is due to their equally big potential (Busch et al., 2012b), but also to the relatedness of both fields (Busch and Steinicke, 2017).

Important technologies to mention here are certainly the ubiquitous mobile devices, such as smartphones or tablet PCs. They contain an extensive range of sensors and quite powerful graphic chips, which is why they can be used as controllers and Augmented Reality devices as well. In addition to these general purpose portable devices (GPPD) a number of products entered the market, which are tailored to specific consumer use cases (or value propositions) – e.g. fitness trackers and head-mounted displays. Other devices may once be independent GPPDs, but are too constrained yet – such as Smart Watches, Smart Clothing as well as Smart Jewelry.

It should be obvious that wearable devices in particular have a huge potential in the context of APITs. This is due to their close proximity to the human body. Thus wearable devices provide unique opportunities for enhancing and extending human abilities as well as providing the carrier with additional information, such as standard operating procedures (SOPs) and contextualized in- and external status or instructions.

Interactive technologies can be applied to directly foster co-creation or enable it in the first place. This is especially relevant when participants do not share the same location.

Furthermore, current developments in the segment of consumer electronics – among those especially microcontrollers like Arduino or single-board computers like Raspberry Pi – have led to a wide range of technical components. These are not only of interest to individuals of the Do-It-Yourself (DIY) or maker scene, but make a shift from off-the-shelf solutions towards customized and possibly unique machines and gadgets feasible – even for small companies.

4. Fields of application

The portrayed interactive technologies can be applied in numerous fields. The following exemplary fields of application are chosen, because they show the most economic and social potential in the short to medium term. They also capture general conditions that apply to a wide range of scenarios and use-cases in these application fields.

4.1 Co-creation

In the business, management and marketing literature the concept of co-creation underlines that the interaction of enterprises and their customers (Prahalad & Ramaswamy, 2004; Payne, Storbacka & Frow, 2008) as well as suppliers and distributors (Perks, Gruber & Edvardsson, 2012) is a (central) source of value creation.

Additionally co-creation can be applied to creative tasks and work in groups (see Ranta-Meyer, Chapter X). The effective sharing of information about individual actions as well as within interactions can be assumed to be a general prerequisite for co-creation to occur. In addition to this basic need, interactive technologies can be applied to directly foster co-creation or enable it in the first place. This is especially relevant when participants / group members do not share the same location. In this case e.g. avatar-based telecommunication may foster presence and thus strengthen the interaction. But even in onsite meetings APITs may yield relevant results – e.g. when applied to foster brainstorming, ideation and strategy planning in a digital aesthetic learning context (see e.g. Busch et al., 2012a).

Furthermore, APITs may enable or support local and remote co-creation in design tasks – such as exhibition design (Meyer and Steinicke, 2017) – with technologies like game engines, Virtual Reality and Augmented Reality in combination with alternative views of the design space displayed on Head-Up-Displays (HUDs), tablets, (multi-) touch tables and/or PCs.

4.2 Future living spaces

When considering future living spaces relevant themes – such as smart cities (Bowerman et al., 2000) – the potential of APITs becomes obvious. Smart Cities generally are considered to be smart due to analysing “Big Data” as well as interoperate with this data and the cities ICT and IoT systems. But both data and interaction can be assumed to be dependent on mobile and stationary consumer electronics.

Wearables as mobile APITs are on the one hand a very promising source of data that is relevant to a smart city – e.g. for the optimization of (public) transport routes or smart light management in public facilities. On the other hand, APITs and its subset wearables will be one of the pillars for e-participation and e-government as well as co-design and co-creation in public spaces. Furthermore APITs will play a central role in concepts that try to interact with the general public – e.g. for the sharing of experiences as discussed in Bauters (Chapter X).

No matter whether these experiences are initiated via tangible or intangible means, APITs like wearables or Augmented Reality devices will work as an enabler for most forms of user interaction in smart cities.

4.3 Industry

In the context of „Industry 4.0“ and the increasing digitalization of all industries, the usage of APITs within the industrial sector shows grand economical potential. Initial pilot projects like DHL Group’s usage of AR glasses for logistics support (Glockner, Jannek, Mahn and Theis, 2014) and the ProGlove (ProGlove, 2017) – a glove equipped with a barcode scanner – already demonstrate the potentials of (certain) wearables.

In cooperation with the X-Visual GmbH, the Creative Media Research & Development Group created a prototype to assist the maintenance of production plants (Busch et al., 2015). A head-mounted display in the shape of data glasses provides the worker with additional information about the machine (e. g. the maintenance procedure, the machine’s maintenance history and current check values) and allows him to perform the necessary steps hands-free. Using proximity markers, for example NFC tags or Bluetooth beacons, the glasses can automatically identify the machine that is being maintained.

4.4 Health

As a field of great social and economic importance, the health sector can also benefit from applied interactive technologies. The range of possibilities is vast: better diagnoses due to long-term measurements using cheap sensors, more extensive training of the staff (e.g. training surgeries using VR simulations), overlaying CT scans and X-ray images directly onto the patient using AR eye-wear or optimizing hospital management.

MindMotionPro (MindMaze, 2017) shows how interactive technologies like motion sensors can be used in the rehabilitation process of stroke patients. Based on existing therapies, the application includes various game-like exercises, where the patient’s movements are tracked and displayed within a virtual world.

While using VR for surgical training is a logical step, “The Alfred Lab” (Embodied Labs, 2017) goes a different path by using VR to show medical students how living as an elderly person with audio-visual impairments feels like and in consequence foster empathy.

4.5 Communication

The usage of interactive technologies for customer communication is growing rapidly. The wide distribution of powerful mobile devices as well as the introduction of new technologies like VR and the initial hypes they induce are certainly reasons for this development. VR glasses already seem omnipresent on trade fairs like the ITB 2016 (ITB Berlin, 2017). Travel operators and other service companies have begun sending VR sets to their customers, enabling them to examine the hotel (or any other place or thing) beforehand.

Sensors, gesture recognition and small-scale robotics allow for interactive exhibitions. Additionally integrating user devices (Bring-Your-Own-Device-Principle) makes personalized and / or extended exhibition experiences possible and potentially reduces costs on the exhibitor’s side.

4.6 Work & learn

In conjunction with exciting teaching and learning materials, interactive technologies show great promise in various educative scenarios from classical (pre-) school education to professional training.

In the spirit of “learning by doing”, APITs support training on the job scenarios. Building upon the maintenance example from the “Industry” section, the maintainer could learn about the specifics of a machine while working on it. AR glasses would display the instructions of the maintenance procedure and highlight the according interaction points using object recognition. Context-sensitivity can be used to make the system more intelligent, e.g. inferring the correct moment for displaying the next instruction using hand or gesture recognition, f. ex. by recognizing a screwing motion.

A similar system could be used for training the welding of metals, where the correct positioning and movement of the welding tool are of utter importance. AR glasses can be used to overlay visual guides and give feedback, when the velocity and / or working angle are suboptimal – giving the user the chance to correct the motion and (with enough repetitions) store it in the motoric memory.

5. Re- & preview: Creative Media’s current R&D on APITs

The “Innovationsforum Interactive Technologies” – a series of events initiated by the Creative Media Research and Development Group – focused on different aspects of APITs and their use with the goal of fostering research on the full potential of APITs, finding new fields of application and promoting their use in various fields of business. Representatives from companies and industrial firms met researchers and game developers to exchange experience as well as discuss and develop new use cases for APITs. In the discussions and workshops during the events of the “Innovationsforum”, many creative ideas like those above have sprung up and will be followed upon. They outline the changes in working, living and learning conditions our modern societies are about to face. Industry, service and private lives are equally suitable fields of application for interactive technologies. Building on the success of the “Innovationsforum”, the Creative Media Research and Development Group recently established the “Anwendungszentrum creative Applied Interactive Technologies (cAPITs)”. This application centre is funded by the European Union and the Berlin Senate (EFRE) and supported by the trade association of the games industry (Bundesverband der Interaktiven Unterhaltungssoftware (BIU) e.V.) and Berlin’s business and technology development agency (Berlin Partner GmbH). Its mission is to foster the application of interactive technologies in non-entertainment contexts as well as serve as a training and network facility for the games industry and/with commerce & industry.

Bringing together representatives from various disciplines, the “Innovationsforum” also lead to the creation of the ZIM network “ProWear”. This cooperation network’s main goal is to research the usage of wearables in professional contexts and to develop prototypes and solutions accordingly. Using consumer electronics – and wearables in particular – in non-entertainment contexts, opens up several fields of research, which are currently being worked on by the networks actors:

  • Physical-technical connection of device combinations with respect to their according contexts, e.g. connecting devices and sensors using fixtures, plug connections, etc. with each other and / or with clothes or the body.
  • Setup and robust operation of (ad-hoc) networks between wearables and local as well as non-local interaction points and data sources, e.g. setup of network connections via Bluetooth or similar standards and alignment of data formats and exchange protocols.
  • Adaptive presentation and interaction methods that adhere to existing and use-case-specific usability requirements and bridge the technological side with the fields of application, e. g. enabling hands-free operation of a given device combination in the context of an industrial application.

Our research and development carried out both in the cAPITs application centre and the ProWear cooperation network, as well as the input and experience from the participating companies, will help us advance the concept of APITs and explore the possibilities they create within the various fields of application – from the industry and health sector to future living spaces.

References

  • Bowerman, B., Braverman, J., Taylor, J., Todosow, H., & Von Wimmersperg, U. 2000. The vision of a smart city. In 2nd International Life Extension Technology Workshop, Paris (Vol. 28).
  • Burke, B. 2014. Gamify: How Gamification Motivates People to Do Extraordinary Things. Boston, Bibliomotion. ISBN 978-1937134853.
  • Busch, C. 2010. Final Report of the Research Project “Realitätsnähe und Symbolische Interaktion in Computerspielen und Online-Games” (Reality Concepts and symbolic interaction in digital games). Berlin.
  • Busch, C.; Conrad, F. and Steinicke, Martin 2012a. Digital Aesthetic Learning: Using In-Game Artefact-Creation to facilitate Learning in Change and Innovation Workshops. In: Proceedings of the Online Educa, S. 126–128, ICWE, Berlin. ISBN 978-3-941055-16.
  • Busch, C.; Kramer, A; Meyer, R. and Steinicke, M. 2012b. Angewandte Consumer Electronics in Lehre und Entwicklung. In: Sieck, J. (Ed.) Wireless. Communication and Information – Mobile Gesellschaft, 5–15, Verlag Werner Hülsbusch, Göttingen. ISBN 978-3-8648-8029-2.
  • Busch, C. 2014. Gamification – Techniken und Methoden digitaler Spiele als Innovationstreiber in der Kreativwirtschaft und anderen Branchen. In: Holl, F. and Kiefer, D. (Eds.) Creative Sprint – A Collaborative View on Challenges and Opportunities in the Creative Sector. (German Version), 70–81, Fachhochschule Brandenburg, Brandenburg/ Havel. ISBN 978-3-00-045295-6.
  • Busch, C. (Ed.) 2015. Inside Creative Media – Werkstattbericht. Glückstadt: Hülsbusch, W.Embodied Labs (2017).
  • The Alfred Lab: Aging with macular degeneration and hearing loss. Retrieved 1.3.2017 from http://www.embodiedlabs.com/work/.
  • Busch, C. and Steinicke, M. 2017. Gamification vs Applied Interactive Technologies – Distinctions and Mappings. In: Schönbohm, A (Ed.) Spielräume: Facetten von GAMIFICATION in Unternehmen und Weiterbildung, Flying Kiwi Verlag, ISBN 978 3 940989 32 1 (in print).
  • Gilbert, B. 2014. Why Activision is spending $500 million on Destiny. Retrieved 20.5.2016 from http://www.engadget.com/2014/08/13/destiny-500-million/.
  • Glockner, H., Jannek, K., Mahn, J. and Theis, B. 2014. Augmented Reality in Logistics: Changing the way we see logistics – a DHL perspective. DHL Customer Solutions & Innovation.
  • ITB Berlin 2017. Keynote-Panel: Virtual Reality – Marketing Powertool oder Branchenbedrohung? Retrieved 1.3.2017 from http://www.itb-berlin.de/de/Besucher/Events/Eventsuche/Eventdetail.jsp?eventDateId=381270&pageTitle=Keynote-Panel%3A+Virtual+Reality+-+Marketing+Powertool+oder+Branchenbedrohung%3F.
  • Kapp, K. M. 2012. The Gamification of Learning and Instruction. Game-based Methods and Strategies for Teaching and Education. San Francisco: John Wiley and Sons. ISBN 978-1118096345, 10.
  • Microsoft 2015. Detecting heart rate with Kinect. Microsoft Developer Kinect for Windows Product Blog. Retrieved 20.05.2016 from https://blogs.msdn.microsoft.com/kinectforwindows/2015/06/12/detecting-heart-rate-with-kinect/
  • Middelberg, A. 2013. GTA V kostet 265 Millionen US-Dollar für Entwicklung und Werbung. IGN.de Retrieved (20.05.2016) from http://de.ign.com/grand-theft-auto-v-ps3/86554/news/gta-v-kostet-265-millionen-us-dollar-fur-entwicklung-und-werbung
  • MindMaze 2017. MindMotionPRO. Early Rehab. Made Simple. Immersive Virtual reality solutions for early motor rehabilitation in Stroke. Retrieved 01.03.2017 from http://www.mindmaze.com/healthcare-and-neuro-rehabilitation/
  • eyer, R. and Steinicke, M. 2017. Using the Hololens for Mixed Reality Exhibition Design. Accepted for publication in Proceedings of XV. Konferenz Kultur und Informatik: Mixed Reality, vwh Verlag Werner Hülsbusch, Glückstadt.
  • Perks, H., Gruber, T., and Edvardsson, B. 2012. Co‐creation in radical service innovation: a systematic analysis of microlevel processes. Journal of Product Innovation Management, 29(6), 935–951.
  • Prahalad, C. K. and Ramaswamy, V. 2004. Co-creation experiences: The next practice in value creation. Journal of interactive marketing, 18(3), 5–14.
  • ProGlove 2017. 1st smart glove for industries. Retrieved 01.03.2017 from http://www.proglove.de/