RESEARCH ARTICLES

 

Disappearing boundaries? Reality, virtuality and the possibility of 'pure' mixed reality (MR)^†

 

 

Daniel O'Shiel

FONDECYT and the Instituto de Filosofía, Universidad Diego Portales, Santiago, Chile https://doi.org/0000-0002-5521-2903

Correspondence

 

 

---

ABSTRACT

This article argues that reality and virtuality are still very much phenomenologically distinguishable, although this might not be the case forever. I argue for two main types of virtuality - one inherently involved in the dynamic horizons of perceptual experiences, while the other is all of our experiences of digital images - in order to show that a particular possible instantiation of the latter type, namely "pure" mixed reality (MR), might come to blur and collapse various experiential categories in the future, not least real and irreal, like never before. To show this, there are three main sections. First, I outline my understanding of the two basic types of virtuality, as understood from a classical phenomenological analysis. Second, I give an account of the most important family of "virtual technologies" relevant to the question at hand, namely virtual, augmented and mixed reality (VR, AR and MR respectively) technologies. After homing in on MR, I explain what "pure" MR is and how, through tactile holograms, this might change even basic experiential distinctions going forward, and not necessarily for the better.

Keywords: holograms; image-consciousness; imagination; mixed reality (MR); perception; phenomenology; reality; virtuality

---

 

 

Introduction

This article investigates the phenomenological natures of reality and virtuality and argues for two kinds of the latter, as well as the fact that reality and digital virtuality - that is, one of the types of virtuality - are still very much phenomenologically distinguishable, although this might not remain the case forever.

We will see that the first type of virtuality is involved in the dynamic spatial and temporal horizons of perceptual experiences themselves, whereas the second is all of our experiences of digital images. I will also show how "pure" mixed reality (MR) must be considered as a particular possible instantiation of this latter type, which might even come to blur and collapse various experiential categories - like between real and irreal, present and absent, genuine and fake - in the future, and like never before.

To demonstrate all of these points, there are three main sections. Firstly, I present my understanding of the two basic types of virtuality, as understood from a classical phenomenological analysis of perception, "phantasy" and "image-consciousness" . Secondly, I give an account of virtual, augmented and mixed reality (VR, AR and MR respectively) technologies, which I consider as the most important family of "virtual technologies" relevant in this context. I then home in on MR, explain what "pure" MR is and how, through tactile holograms, this might change even basic experiential distinctions going forward, and not necessarily or always for the better.

 

Two types of virtuality: Horizonality in perception and digital images

"Virtuality" is an exceedingly difficult term to pin down with any precision. Through my research, some of which has already appeared (O'Shiel, 2019), I have landed on two main meanings. These are distinct in that they take place on different experiential planes, one on the level of reality and perception, the other as a particularly powerful sub-category of our imagination. They are still, however, both connected by virtuality's basic nature to be on the cusp of the real and present without actually being them, therefore as a crucial experiential bridge between what is fully there and what is not but could quite easily be. This is the underlying nature of virtuality thus manifest in two different types: on the edges of our immanent and immediate perceptual fields in a way which makes these edges possible in the first place; or as manifest in the nature and dynamics of lively and physical digital images which can captivate us immensely and even come to supplant our perceptions both in the order of significance and value as well as the amount of time spent.

To explain a bit further, I need to briefly summarise my classical phenomenological understanding of three basic types of experience - namely perception, phantasy and image-consciousness (Bildbewusstsein) - as collected from my studies of Husserl, Fink and Sartre (O'Shiel, 2019). Once this is done, I will be able to situate and relate the two types of virtuality I am speaking of.

From the Husserlian phenomenological perspective, perception is decidedly about what is real and present (for example, a cup right in front of you that you can drink from); "phantasy" or imagination in a narrow sense - designated in classical German phenomenology technically by "phantasy" with a "ph-", which includes but is not exhausted by more everyday operations of fantasy (of unicorns and the like) - is about what is irreal and not physically present (for example, imagining a unicorn); and "image-consciousness" (Bildbewusstsein) is a middle category that presents us with images through external, physical materials (examples: a painting; a picture of a horse on a screen; watching a tennis match on TV). What is key to emphasise here is that classical Husserlian phenomenology has an incredibly rich theory and debate about the experiential nature and differences between our perceptions, our phantasies and imaginations, and our engagement with external, physical images (that is, "image-consciousness"), the latter two groups making up imagination in a broader sense. The first are always about reality, the second about irreality. Regarding the third, there is some debate about the status of image-consciousness. Husserl ([1898-1925] 2005) has it as its own category with its own tripartite structure of "physical image" (the physical material, for example a TV), "image object" (one's immediate phenomenological experience of it, in this case the TV emissions) and "image subject" (what the TV is showing, for example a tennis match at Wimbledon); Fink ([1930] 1966) has it as a special, transcendent type of perception; and Sartre ([1940] 2004) as a special and powerful sub-category of "the imaginary". These differences notwithstanding, each finds it between pure perception and reality on one hand, and pure phantasy and irreality on the other.

In all of this, I have become convinced that there are two rather unique types of virtuality that nevertheless are concrete manifestations of a more universal general dynamic between what is real and not, fully present and not. The first is on the level of perception and is basically Husserl's whole theory of perceptual consciousness as essentially horizonal consciousness - that is, in perception it is impossible to have experiences that are not always already structured by the infinite potential for both inner (that is, looking closer) and outer (looking beyond) horizons. Although Husserl never uses the term "virtual" himself, it quite clearly fits perfectly for that which is not but could easily become present in my perceptual field within his highly detailed phenomenological theory. A very simple example is the implicit virtuality of the room next to from where I currently am, and then the exterior of the house beyond that, then the road, the town and so on. In this manner, only a very small portion of our experiential lifeworld is actually present to us at any given moment; all the rest is virtually so, radiating out from great and easy potentiality which is frequently and habitually actualised close to where we are, all the way to immensely distant climes, regions, worlds and galaxies which have an increasingly zero possibility of being actualised, and are even difficult or impossible to imagine due to our perceptual or even epistemic ignorance of them.

In short, in the perceptual realm, the virtual is an absolutely essential element, as the continuously almost or soon-to-be physically present, like the next room if I get up and actually go there. In this way, our perceptual lives are a constant interplay between the virtual becoming actual - that is, perceived - and the actual fading back into virtuality (again), often to be reignited when the right circumstances and motivations arise.

On a different plane, we discover virtuality in the second main meaning. If the first meaning of virtual as potentially "real" or "actual" (which in German is the same term, wirklich) has a long and rather complex philosophical history going back to medieval thought and concepts like Aristotle's "dunamis", this second variant is decidedly more recent and coincides with the rise of contemporary computerised and digital technologies and societies. Here "virtual" simply means digital, computerised and networked phenomena; all the "images" - that is, visual but also audial and otherwise (for example, a vibration on a game controller) - that impinge upon our senses through our electronic and networked devices. In this manner, although one could vaguely talk about the "virtuality" of a unicorn in phantasy, or how Cézanne's painting "The Basket of Apples" (c. 1895) can put you virtually in touch with a bunch of apples that are not actually there like a real one is, "virtual" and "virtuality", as we now ordinarily refer to these terms, are all those experiences facilitated by our networked digital technologies, and which thereby put us in direct and quite lifelike and realistic touch with information, people and things that are still nevertheless not actually present like we or the screen or the apple right in front of me is.

In this manner, the two meanings of virtual cover, on one hand, that which is almost or potentially present in the perceptual mode and, on the other, that which is digitally so through our computers and devices. Moreover, the latter is, in structural phenomenological terms, a particularly strong and ever-developing category of image-consciousness (compare Liberati, 2012). Underlying all of this, in both concrete manifestations, is virtuality generally as that crucial conceptual and experiential bridge between the fully real and present (the perceptual) on one side, and the fully irreal or absent (phantasy or imagination in the narrow sense) on the other.

I think, until now, that intuitively and experientially most of us most of the time can still very much distinguish between what we perceive and what we imagine, namely between what is real and actually there, and what is a mere wisp of our phantasy. There are borderline cases like dreams, illusions and the like (see Smith, 2002), but these are usually corrected diachronically precisely because we have a more basic comprehension of what is real and what is not. However, the crucial and interesting thing with new "virtual technologies" - including technologies like social media and online gaming but also especially VR, AR and MR technologies - is that they are all starting to increasingly blur the lines between what is simply real and actual and what is not. This latter distinction, moreover, is already perhaps inverting in terms of importance and value for some, and therefore it might even collapse in the future, not only theoretically but also experientially and evaluatively. Just how this could happen and what this might mean is the focus for the rest of this article, and I can consider this issue to its maximum potential by homing in on the case of "pure" mixed reality (MR).

 

VR, AR and MR technologies

MR is the latest and ongoing development from a family of virtual technology that began some decades ago. Virtual reality (VR) has been around, on and off and in various forms, since the late 1960s (Baudisch, 2015; Plasencia, 2015; Friedman et al., 2016; Cipresso et al., 2018; Nunes de Vasconcelos et al., 2019). However, in recent years, due to technological, functional and economic advances, it has augmented in stature and use considerably, not least in the gaming industry. Augmented reality (AR), for its part, is a more recent phenomenon but, as we will see, has already showed signs of surpassing VR in its scope, popularity and use. Finally, mixed reality (MR) is a rather new development that is not yet really fixed, neither in concept nor in a particular physical piece of technology. Nonetheless, it is crucial to figure out, especially with regard to our basic experiences of perception and imagination, including where all this might be heading.

In the last decade or so, literature on VR, AR and MR technologies has burgeoned. There have already been quite staggering recent efforts (e.g. Cipresso et al., 2018) to catalogue most, if not all, highly indexed academic writings on the matter, as well as countless particular studies. Predominant lines of research seem to be the technologies' potential effects and uses in education (e.g. Barbalios et al., 2013; Bujak et al., 2013; Yilmaz, 2016), medicine and health care (Kleinsmith et al., 2015; Lee et al., 2019), as well as the massive realm of retail, industry and manufacturing (West et al., 2015; Flavián et al., 2019; Coles, 2020; Malik et al., 2020; Sun et al., 2020). In all of these, as well as hundreds, if not thousands, of other articles, the overarching interest is based on decidedly scientific and technical grounds, with the aims primarily practical and economic.

When it comes to the philosophy of virtuality and virtual technology, there is relatively little literature. There is the odd phenomenological attempt here and there, whether it be regarding embodiment and the arts (Morie, 2007; Katan-Schmid, 2020), the nature and influence of (a particular type of) AR technology (Liberati, 2012; 2018), or a more wholesale attempt to understand the nature of "virtual fictions" through particular concrete analyses (de Warren, 2014), or an interdisciplinary treatment with sociological accounts emphasising the various dimensions of virtual space in certain online experiences like gaming and Skype (Berger, 2020). Metzinger (2018) also has a useful article demonstrating how VR can be of interest to a whole host of philosophical domains, listing and explicating them; although he does overstate how high the interest might be, actually and potentially.

Perhaps the most dominant and well-known issue in philosophy is about the specific status of "virtual objects". Here Chalmers's quite well-known claim (2017) that virtual objects are "real" has been met with responses and criticisms (notably McDonnell & Wildman, 2019) and relates to a more general debate about (ir)realism and fictionalism with regard to many mental or virtual objects (compare also Laas, 2015; Van de Mosselaer, 2018; Beisbart, 2019; Juul, 2019). If virtual objects are "real", it is because those in favour take, wittingly or not, a more metaphysical path - like Bergson ([1896] 2005) and Deleuze ([1966] 2011) do - where virtuality is seen to have "reality" just as actuality does. From this perspective, reality and actuality are not the same, whereas in classical phenomenology they are (wirklich). This different stance is often due to a rather wholesale representationalism - in Sartre's terms the "illusion of immanence" ([1940] 2004, p. 6) - where all, even perceptions themselves, are still just ultimately "in the head" (see Noë, 2010), or are even just "images" of a deeper world of matter (compare Bergson, [1896] 2005). From these epistemological, metaphysical and ontological standpoints, claims can then even be made that perception is somehow just the most "close-to-perfect VR experience we currently know" (Metzinger, 2018, p. 3). Once one is in this mindset, one is indeed far down the rabbit hole of representationalism (compare also Wiesing, [2009] 2014).

Recently there have been acknowledgements and developments of "4e cognition" in cognitive science and the philosophy of mind, where the overlapping ideas of embodied, embedded, enactive and extended have gained significant traction and allowed these traditions to escape the head somewhat. Classical phenomenology has, however, known, studied and explained these elements for over a century, and thus although this new discipline certainly can complement and build upon phenomenological insights, some of the latter might also run the risk of being corrupted by rather predominant representationalist proclivities.

Regarding the main debate: if "real" just means "there are such things as virtual objects" - and sometimes Chalmers seems to speak this way himself - then it is not even trivial. Of course, phenomenologically and otherwise, there are virtual objects for the simple fact that we experience them. What is also quite obvious but not noted by many, however, is the fact that they are given to consciousness through screens and other devices, and are thus already, by this very fact, experientially and even ontologically not the same as straightforward perceptions. This is because digital virtualities are not fully there in front of you in the same manner as a normal case of perception; virtual digital phenomena act precisely as media for a transcendent endpoint, object or world that cannot be perceived and experienced with all the senses like this apple right in front of me can.

This is indeed the main point: no matter how elaborate VR technologies are or become, they remain of the digital, image-consciousness order and structure, and are thus virtual in the second sense presented here. This will remain so as long as they do not cover all the complexity of perception and we remain aware of implicitly "stepping into" or entering them from our everyday perceptual lives - currently an impossible task. Nevertheless, their capabilities and appeal are already such that they can even supplant the real and perceptual for quite some time for many. What is more, perhaps even more so than any social media platform or any digital, online game, VR is a technology that tries to be as real, lifelike and thus perception-like as possible. Indeed, generally one may say that VR systems, as they continue to develop, are trying to cover the lived body as much as possible - they are trying to not only attain visual and audial verisimilitude and exactitude, but also cover other elements of our psychophysiology, as Spielberg's 2018 film Ready Player One rightly speculates. VR vision and audio are already at high levels, with some head and limb motion also already well incorporated for certain games or programmes. The next challenges will be to incorporate more haptic, proprioceptive and wider movement elements. Beyond this, although some original VR in the 1960s simulated phenomena like wind and even certain smells (see Stein, 2016) in a highly specialised setting, incorporating widespread touch, smell and taste do not yet seem to even be on the radar of mainstream and affordable VR technologies.

The interesting phenomenological and philosophical point is that VR will only ever cease to be VR if every single facet of our perceptual experience is covered to the level of indistinguishability, including an erasure of the "stepping-in" moment to the technology. With the rise of supercomputers and artificial intelligence (AI), perhaps uploading our minds into a digital paradise (or hell) - as the former is portrayed well in the "San Junipero" episode of Black Mirror (a TV series) - is more theoretically possible than a one-size-does-all VR bodysuit. Nonetheless, in the event of the latter, it would need to accommodate not only all of our bodily movements and sensations, both inner and outer, but also all of our basic biological and bodily functions like drinking, eating, sleeping and expelling waste, all in a seamless manner and with no recollection of a former or other world. This is a radical, far-fetched scenario and is one, moreover, that the gaming world at least seems not to even have as a goal. Nevertheless, it must be the general, or at least one, end template of VR experiences as such: create the technology to cover all of our perceptual capacities and life. Then, and only then, would one have a perfect VR system where the "V" is no longer even recognised.

Considering these points, such a possible pretension must be seen more as a theoretical guideline than an actual goal, at least for now. However, in a not-so-distant future it is conceivable to have Ready Player One-style VR bodysuits and treadmills that would take the immersion and completeness of at least four key elements (vision, sound, some touch and various bodily movement) to new levels. Where the developments go after that cannot be foreseen, but by this logic they would try to gradually match up with the capacities of the perceptual world, even to taste and smell - while also, not least in game scenarios, adding in fantasy elements and scenarios as designers so choose.

VR is more mainstream nowadays, but it is still also something that the majority of people have not experienced. This might be for reasons ranging from high price or lack of interest, to the rather simple but seemingly powerful fact that the devices are still rather bulky and with wires, and thus decidedly immobile and uncool, in contradistinction to smartphones and their apps. Some smartphone companies, like Samsung, have tried to market smartphone-VR hardware and experiences, but it does not seem to have taken off. Taking this into account, VR has not (yet) revolutionised every facet of modern life like smartphones have. Regarding these latter, they are even more than mainstream because they are the devices which constitute and maintain the mainstream itself. With VR's relatively marginal status and success, therefore, it might come as unsurprising that a good deal of research and focus (e.g., Carter, 2020) is already moving on from VR to its younger and potentially more explosive cousin, AR.

Generally speaking, augmented reality (AR) is when a primarily real image - that is, an image that has real, perceptual content, usually captured by a (video) camera - is overlain with some digitally or otherwise projected or manipulated element(s) (e.g. a Pokémon). Although one could argue for images being manipulated and doctored before the rise of digital technologies, it is widely accepted that it is in these latter situations where these "augmented" realities do occur. In this research, a good number (for instance, Boland & McGill, 2015; Cipresso et al., 2018) posit a reality-virtuality continuum which was originally introduced by Milgram and Kishino (1994; see also Flavián et al., 2019). This is where, on one extreme, you have "pure reality", namely the perceptual, real world without any mediation through digital screens or similar devices; and on the other side you have "pure virtuality", namely a wholly digital experience where perception has been fully bracketed and blocked out, as is the case when one puts on a VR headset and fully engages with the game or programme. Between these two extremes are mixtures of reality and digital virtuality, which is indeed what many (for instance, Boland & McGill, 2015; Liberati, 2018) refer to as "MR", "mixed reality". In this in-between land, there are two general categories (Boland & McGill, 2015; Chalmers, 2017; Flavián et al., 2019): "augmented virtuality" (AV) is a predominantly digital, virtual environment with a few real, perceptual objects called in and overlain when required (for example, a keyboard while gaming - see Boland & McGill, 2015); and AR is when the opposite occurs, namely an image of the real and perceptual world that, however, also has a few digital and virtual objects projected or integrated into it. Both categories are of course images; they have the structure of image-consciousness simply because they use screens or headsets. However, AV is a digitally created one with some perceptual items brought in, whereas AR is a perceptually faithful image, at least in terms of vision and sound, with some digital objects superimposed.

Pokémon Go (2016; see also Liberati, 2018) is one of the better-known instances of AR: you search, through your phone and its camera, the real, actual world for digital, virtual characters which are superimposed and projected into this very same world through your phone. On top of this, I would propose instances that are already quite prevalent outside gaming. One is some new car windscreens. These allow you to see the road ahead while also protecting you from wind, rain and other natural forces and events, meaning they are and have always been designed based upon real, perceptual and practical concerns. However, now in certain higher-tech models, classic screens have become partially digitised and are thus overlain with virtual images and pieces of information, such as the speed one is going, whether one is in the lane and so forth. The supposed advantage is that important pieces of information can now be in your eyeline, meaning you do not have to take your eyes off the road. This, then, is a small everyday example where some realities already have superimposed information and images on them, supposedly to aid us in an important and potentially dangerous real task like driving.

AR keeps its base in reality by either recording, with a camera, that reality, or by providing a transparent screen (see also Feiersinger et al., 2018) and thereby allowing us to continue perceiving reality while at the same time creating the possibility to project and display data, information and images onto that screen or surface. Pokémon Go is an example of the first; the digitised car windscreen is a case of the second.

There have already been advanced AR technologies which have failed commercially, none more so than Google Glass. However, although it may be a long time, or even never, before everyone has a piece of AR eyewear like a smartphone, at least in certain domains like education, medicine, manufacturing and more, world-leading AR technology looks to have taken hold. In fact, AR is already a significant part in many of our lives, perhaps without us even noticing. This is due to our smartphones and their applications. For instance, an Instagram filter is nothing other than a piece of AR because it takes an accurate representation of something (for example, one's face) and then adapts it through various digital and virtual manipulations. Even further, many memes and online posts, from funny pieces of entertainment to dangerous pieces of misinformation, use the same tactic: they take visual or factual things, images and information and then adapt them for various ends, from entertainment to deception. We are thus already living, through our phones and social media accounts, in an AR universe, where that which has a real source, and that which does not, has already become quite fused and even indistinguishable at times.

This is one reason for the latest and last term, "mixed reality" (MR), which is also the end point of the investigation here. The more one looks at AR and AV, the more one sees a continuity between technologies that have a predominant basis in perceptual reality (AR), and those with a predominantly digital base (AV). Ultimately anything on the reality-virtual continuum, if it is not a simple case of reality or virtuality, is a mixture between the two. "MR" would then be any image that is a combination of real and digital elements. This is indeed where the technology seems to be heading, with some of the most promising being the Microsoft HoloLens and the Magic Leap 2, currently devices which can capture, create and develop a wide range of phenomena on the reality-virtuality continuum. In this manner, MR comes to cover both AR and AV because it is any technology that uses both real and virtual elements together.

Crucially here, Flavián et al. (2019, p. 549; emphasis in original) argue for a "pure mixed reality" supposedly right in the middle of the continuum, where the effect "is not superimposed on the physical environment (as in AR) but virtual objects are rendered so that they are indistinguishable from the physical world". This, for me, would be a mixed reality of a different phenomenological order, one that I can only finally address in the last section of this article. For now, I think it is important to note the power of AR as a type of MR too because it situates the experience squarely in one's home universe. To paraphrase Bujak et al. (2013), AR does not, unlike VR and even AV, separate the user from their current perceptual reality; if VR is about transporting one into a fantastic digital world in order to play, escape and transcend, and AV is the same with the appearance of real aids from time to time, then AR as well as MR which keeps one situated in one's actual environment both bring the transcendent and fantastical into one's own classroom, home environment and phone. This is the virtual in the strongest sense of this second type, and although it may seem a rather trivial point, the very fact that these technologies allow the virtual and digital to quite freely infiltrate and inhabit our immediate lives, habits and surroundings, is already a testament to its power. Nowhere has this occurred more seamlessly than with our phones: the easy use of filters in a simple social media app, or altered data in a piece of fake news, are precisely testaments to AR's early success, once one thinks about the true meaning and scope of the term. Indeed, conceived broadly, AR already has, through our phones and other digitised screens, facilitated a great deal of fun and novel presentation of information, but it has also already been highly misleading and dangerous. Excuse the pun, but this situation looks set to augment in the future as well.

AR takes what we are used to and supposedly "augments" it through its digitisation and manipulation. Little by little, swipe by swipe, this is becoming a new norm for many, something which has massive experiential, behavioural and moral implications because it goes way beyond any niche market or realm, embedded on our phones and thus in our lives in a manner that is already changing how we experience even basic objects and phenomena like our own faces, as well as the whole world of facts.

In summary, VR is transportation into a digital, virtual realm and thus is wholly imaginary - that is, irreal and digital. AR, when it takes place through some kind of digital screen or glasses, remains a medial, clearly digitised experience. Transparent natural screens like windscreens remain perceptual with some digital elements superimposed. However, it should also be coming clear that the lines are already starting to blur. The whole point of AR is to superimpose objects into our realities to the extent that, functionally at least, there is little or even no difference. This, along with AV, is already the whole of MR, which more generally blends all kinds of perceptual and digital objects. Here I may now ask: is there a world, like Flavián et al. (2019) suggest, where this mixture can become "pure", where one would no longer know the difference between real and virtual, physical and digitally produced and present? To answer this, I will look at one of MR's most promising and also potentially phenomenologically puzzling instantiations, tactile holograms.

 

"Pure" MR and the case of tactile holograms

A recent advertising campaign for the VR headset Oculus Rift summarises VR's position very well: "defy reality". VR brackets the real for a transcendent and immersive digital plane and experience. When it comes to AR, in many of the specific and usual instances, such as Pokémon Go and even a digital car windscreen, even though perceptual and digital elements are both present in the same frame, it remains very clear what is real and what is digitally superimposed. However, I have also argued for an extended notion of AR, namely any type of virtual technology that takes up real elements and augments them with digital and virtual elements and manipulations. This can be done in an obvious and explicit way (for example, many fun filters), but it can also be done in a manner where one no longer recognises which elements are real and genuine, and which are digitised and fake (examples: a very convincing filter; an airbrushed image; a "deepfake"; a piece of fake news). In these latter cases, we enter the issue of MR in a narrower and more phenomenologically thought-provoking sense: what Flavián et al. (2019) have dubbed pure MR. I have clarified that MR generally is just any kind of blend of real and virtual, and thus covers all of AR and AV, and basically anything that is not pure unmediated perception or pure filtered VR. "Pure" MR, for its part, is when the mixture is so blended and fused that one no longer is able to distinguish real from virtual elements. This can be problematic in itself, such as coming to overly rely on filters for one's online digital appearance, to being duped by a piece of fake news, to being financially scammed online, to even being groomed online and then lured into a dangerous and abusive situation in real life. Conceived in this way, MR generally is already a quite prevalent phenomenon that can be used for beneficial endeavours in domains such as entertainment, education, business and art; but it is also used to deceive and abuse.

If these contents stay within our digital screens, then at least we know where to be wary. With the right awareness and education, we can be continuously cautious of potentially damaging uses of (pure) MR on and through our screens (as well as other devices to come). Indeed, the mere fact that these contents appear through a screen, headset or otherwise still clearly delineates them from the real, perceptual world, and thus, structurally speaking at least (if not functionally, evaluatively and cognitively speaking), we can still tell a case of immediate perception from a mediated image or piece of virtual and digital information that might often need to be checked and taken with various pinches of salt.

However, there is already evidence that MR, and even pure MR, might eventually go beyond our neat little digital rectangles and vistas as we currently know them. The latest transparent AR technologies, not least the Microsoft HoloLens and the Magic Leap 2, look set to take the educational, medical, business, engineering and manufacturing worlds by storm, transporting digital phenomena and aids into one's perceptual environment. This is all well and good and could even be greatly beneficial for various reasons, and one could still remain phenomenologically and experientially aware of the difference between what is real and what is digitally imported and manipulated. A next possible step, though, seems phenomenologically and experientially more challenging and, even worrisome. This would be "pure" MR where the different elements can no longer be distinguished or identified.

It seems we are technologically still rather a long way from this. Nevertheless, it was surprising to find an article that is already investigating and showing, with technology that already exists, how to be able to see, hear and touch one's own virtual heart without any haptic gloves or noticeable physical screens. This technology (see Romanus et al., 2020) uses a Magic Leap AR headset, an Ultrahaptics pad and an Apple Watch all synchronised together so that one can see, hear and haptically engage with a "mid-air haptic bio-hologram", in this case a representation of one's own heart, right before one's eyes, ears and hands. The floating hologram is not only synchronised to beat as the user's own heart is beating; one can actually "touch" and handle it through "touchless ultrasonic haptic technology [that] employs electronically controlled phased arrays of ultrasound speakers (or transducers) to create high acoustic pressure points in mid-air that can be felt with bare hands" (ibid., p. 2). The authors are notably silent on how lifelike this touch might be; nevertheless, it is already a case of being able to physically interact with a floating digital object through three of the five senses, and arguably the three most important senses when it comes to covering the most basic aspects of perception.

For now, such interaction must of course be in a controlled, set-up environment and, although a hard, physical digital screen is not there, the phenomenon does use a good deal of virtuality-making hardware that is moreover quite conspicuous. Nevertheless, it is already quite a step to be able to situate a digital object in way that is much more conducive and natural to our ingrained perceptual instincts and capacities. What is more, holograms from a distance can already be visually and audially quite convincing, especially if one is not paying full attention (I had one such experience in an airport with a digital flight attendant). However, further scrutiny and the fact they are usually projected onto a flat two-dimensional surface then gives the game completely away. With this latest instance of the heart, however, one can see a pathway to full three-dimensional holograms that could be convincing in a visual, audial and tactile manner.

Touch is key and was Husserl's most valued and basic sense for the world of perception (compare Husserl [1952] 1989). Without a sense of touch, no physicality, self-awareness or even movement seems genuinely possible; it is unlikely one would be able to stand up. Without touch and the "distal attribution" (Riva et al., 2004, p. 405) it facilitates - namely the automatic referencing of our body to external space - it would be impossible to even basically navigate any kind of external world, real or imaginary. In the latter, beyond clearly touching and interacting with physical analogical materials (for example, a screen or keyboard), the current technological situation with regard to touching immaterial, digitally created images like holograms is extremely rare, staged, controlled and limited, meaning one still easily knows one is engaging with a digital structure, albeit quite elaborately. Nevertheless, it is not much of a stretch to imagine that technology could advance and become less obtrusive and bulky, and thereby so embedded and inconspicuous that holograms and other similar phenomena could come to be treated functionally and maybe even phenomenologically in the manner of real, non-digital objects. Indeed, if brain and body hacking also make serious actual strides, maybe holograms will be bypassed altogether, and pure MR will be attained in this way.

A hologram one would see, hear, touch and be able to converse with is already significantly on its way to a type of "pure" MR, with the other two senses (taste and smell) perhaps less important and, nevertheless, maybe possible in the long run too. By combining these virtual sensations with AI, it is conceivable to have a situation like in Blade Runner 2049 (2017) in which the protagonist Officer K (Ryan Gosling) can almost fully interact with an AI hologram Joi (Ana de Armas), even to the point of having sex with her (or it?). Given what has been explained here, this might not be as outlandish a prospect in thirty years as one might think. Thus, pure MR, not least as situated and realistic holograms, are a theoretical possibility as well as a live technological work in progress that could revolutionise a great deal of our experiences, and although it could probably never replace everyday natural perception completely, it is conceivable that it could rock basic phenomenological distinctions to their very core, if not completely collapse some altogether.

 

Final remarks

The rise of digital screens connected to the internet harbours a new, absolutely dominant age for virtual technology (Shields, 2003), which is separated from more classical media because its main function is to provide information, entertainment, professional and personal access and interactivity on an instantaneous global level that more isolated forms of image representation never even pretended or wanted to do. Indeed, fully digitised, computerised versions of media have all but taken over their analogue and paperback cousins, and thus it is this immense and ever-growing subcategory of image-consciousness that ultimately best fits the designation "virtual technology" in the fullest sense of the term.

Immersion and interactivity are taken to new heights in VR, AR and MR technologies which, although still in relative infancy in terms of widespread use and popularity, look set to grow and even explode in various crucial human domains, from education and health care to industry and engineering. Here, although games and VR usually keep one in a clearly demarcated digital fantasyland that is all about (serious) play, some of the latest AR and MR efforts are already beginning to decidedly blur the transcendent and fantastic with the real and practical.

The question then is, where is this heading? With the case of pure MR, we have seen it has the potential to not only blur but even collapse the boundaries between real and not, perceptual and digital. It is conceivable that advanced tactile and AI holograms, made possible perhaps through an increasingly pervasive pure MR system, could reach a level of technological acuity that makes even foundational distinctions up till now - like perception and image, real and irreal, present and absent - rather misplaced, or even obsolete. In this sense, if AR in the broad understanding outlined here has greatly facilitated our post-truth age online, the advent of prevalent pure MR could blur basic experiential categories beyond our screens as well.

Normally a complete collapse between reality and irreality would be the hallmark of a kind of pathology, especially an all-consuming psychosis. However, if holograms do ultimately manage to become indistinguishable from classical perceptual objects - and this is a colossal "if" - that would be a scenario where the real and irreal, as well as genuine and fake, natural and artificial, human and machine, and present and absent have all more or less collapsed, while also leaving the status of true and false teetering. These distinctions have already been blurred and sometimes inverted in the order of our values and use in our current virtual technology, screen-culture age. Experiential, phenomenological collapse, however, would be a new level or event, one that a prevalent and pure MR could in fact achieve. This might not even be the goal of MR technologies; however, it is at least a theoretical possibility we need to bear in mind and research further as we, and the technologies, continue to develop at quite a rapid speed and complexity, with consequences which could be very beneficial, entertaining and educational, but which could also be confusing and even damaging to some of the most basic category and experiential distinctions and dynamics that our current societies are built upon.

 

References

Barbalios, N., Ioannidou, I., Tzionas, P., & Paraskeuopoulos, S. (2013). A model supported interactive virtual environment for natural resource sharing in environmental education. Computers & Education, 62, 231-248. https://doi.org/10.1016/j.compedu.2012.10.029        [ Links ]

Baudisch, P. (2015). Virtual Reality in Your Living Room. Communications of the ACM, 58(6), 92. https://doi.org/10.1145/2754393        [ Links ]

Beisbart, C. (2019). Virtual Realism: Really Realism or only Virtually so? A Comment on D. J. Chalmers's Petrus Hispanus Lectures. Disputatio (online first), 1-35. https://doi.org/10.2478/disp-2019-0008        [ Links ]

Berger, V. (2020). Phenomenology of Online Spaces: Interpreting Late Modern Spatialities. [online first]. Human Studies, 43, 603-627. https://doi.org/10.1007/s10746-020-09545-4        [ Links ]

Bergson, H. (2005). Matter and Memory (N. M. Paul & W. S. Palmer, Trans.). Zone Books. (Original work published 1896)        [ Links ]

Boland, D., & McGill, M. (2015). Lost in the Rift: Engaging with Mixed Reality. XRDS, 22(1), 40-45. https://doi.org/10.1145/2810046        [ Links ]

Bujak, K. R., Radu, L., Catrambone, R., MacIntyre, B., Zheng, R., & Golubski, G. (2013). A psychological perspective on augmented reality in the mathematics classroom. Computers & Education, 68, 536-544. https://doi.org/10.1016/j.compedu.2013.02.017        [ Links ]

Chalmers, D. J. (2017). The Virtual and the Real. Disputatio, 9(46), 309-352. https://doi.org/10.1515/disp-2017-0009        [ Links ]

Carter, J. (2020). MR vs VR: why enter virtual reality when you can bring holograms into your world? https://www.scmp.com/lifestyle/article/1976097/mr-vs-vr-why-enter-virtual-reality-when-you-can-bring-holograms-your-world        [ Links ]

Cipresso, P., Giglioli, I. A. C., Raya, M. A. & Riva, G. (2018). The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Frontiers in Psychology, 9 (article 2086), 1-20. https://doi.org/10.3389/fpsyg.2018.02086        [ Links ]

Coles, C. (2020). Augmented and Mixed Reality: What is it, and Where is it going? https://www.idtechex.com/en/research-article/augmented-and-mixed-reality-what-is-it-and-where-is-it-going/20105        [ Links ]

De Warren, N. (2014). Towards a Phenomenological Analysis of Virtual Fictions. Metodo, 2(2), 90-112. https://doi.org/10.19079/metodo.2.2.90        [ Links ]

Deleuze, G. (2011). Bergsonism (H. Tomlinson & B. Habberham, Trans.). Zone Books. (Originally published in French 1966)        [ Links ]

Feiersinger, L., Friedrich, K., & Queisner, M. (2018). Editorial Image - Action - Space. Situating the Screen in Visual Practice. In L. Feiersinger, K. Friedrich, & M. Queisner (Eds), Image - Action - Space. Situating the Screen in Visual Practice (pp. 7-10). De Gruyter. https://doi.org/10.1515/9783110464979-001        [ Links ]

Fink, E. (1966). Vergegenwärtigung und Bild. In Studien zur Phänomenologie 1930-1939 (pp. 1-78). Martinus Nijhoff. (Original work published 1930) https://doi.org/10.1007/978-94-011-6422-1_1        [ Links ]

Flavián, C., Ibáñez-Sánchez, S., & Orús, C. (2019). The impact of virtual, augmented and mixed reality technologies on the customer experience. Journal of Business Research, 100, 547-560. https://doi.org/10.1016/j.jbusres.2018.10.050        [ Links ]

Friedman, M., Friedrich, K., Queisner, M., & Stein, C. (2016). Conceptualizing Screen Practices: How Head-Mounted Displays Transform Action and Perception. Media Tropes, 6(1), i-v.         [ Links ]

Husserl, E. (1989). Ideas Pertaining to a Pure Phenomenology and to a Phenomenological Philosophy. Second Book: Studies in the Phenomenology of Constitution (R. Rojcewicz & A. Schuwer, Trans.). Kluwer Academic Publishers. (Originally published in German 1952) https://doi.org/10.1007/978-94-009-2233-4        [ Links ]

Husserl, E. (2005). Collected Works, Volume XI: Phantasy, Image Consciousness, and Memory (J. B. Brough, Trans.). Springer. (Original lectures 1898-1925)        [ Links ]

Juul, J. (2019). Virtual Reality: Fictional all the Way Down (and that's OK). Diputatio (online first), 1-11. https://doi.org/10.2478/disp-2019-0010        [ Links ]

Katan-Schmid, E. (2020). Playing with Virtual Realities: Navigating Immersion within Diverse Environments (Artist-Led Perspective). Body. Space Technology (Oxford, England), 19(1), 224-238. https://doi.org/10.16995/bst.341        [ Links ]

Kleinsmith, A., Rivera-Gutierrez, D., Finney, G., Cendan, J., & Lok, B. (2015). Understanding empathy training with virtual patients. Computers in Human Behavior, 52, 151-158. https://doi.org/10.1016/j.chb.2015.05.033        [ Links ]

Laas, O. (2015). Contemporary Philosophical Theories of Virtuality: A Critical Examination and a Nominalist Alternative. Techné: Research in Philosophy and Technology, 19(3), 314-357. https://doi.org/10.5840/techne2015121441        [ Links ]

Lee, L. N., Kim, M. J. & Hwang, W. J. (2019). Potential of Augmented Reality and Virtual Reality Technologies to Promote Wellbeing in Older Adults. Applied Sciences, 9 (article 3556), 1-17. https://doi.org/10.3390/app9173556        [ Links ]

Liberati, N. (2012). Improving the Embodiment Relations by Means of Phenomenological Analysis on the "Reality" of ARs. IEEE International Symposium on Mixed and Augmented Reality 2012. Arts, Media, and Humanities Proceedings, 13-17. https://doi.org/10.1109/ISMAR-AMH.2012.6483983        [ Links ]

Liberati, N. (2018). Phenomenology, Pokémon Go, and Other Augmented Reality Games: A Study of a Life Among Digital Objects. Human Studies, 41, 211-232. https://doi.org/10.1007/s10746-017-9450-8        [ Links ]

Malik, A., Lhachemi, H., & Shorten, R. (2020). I-nteract: A cyber-physical system for real-time interaction with physical and virtual objects using mixed reality technologies for additive manufacturing. https://arxiv.org/abs/2002.06280        [ Links ]

McDonnell, N. & Wildman, N. (2019). Virtual Reality: Digital or Fictional? Disputatio (online first), 1-27. https://doi.org/10.2478/disp-2019-0004        [ Links ]

Metzinger, T. K. (2018). Why Is Virtual Reality Interesting for Philosophers? Frontiers in Robotics and AI, 5 (article 101), 1-19. https://doi.org/10.3389/frobt.2018.00101        [ Links ]

Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12), 1321-1329. https://doi.org/10.1007/978-3-642-87512-0_1        [ Links ]

Morie, J. F. (2007). Performing in (virtual) spaces: Embodiment and being in virtual environments. International Journal of Performance Arts and Digital Media, 3(2-3), 123-138. https://doi.org/10.1386/padm.3.2-3.123_1        [ Links ]

Noë, A. (2010). Out of Our Heads: Why You Are Not Your Brain, and Other Lessons from the Biology of Consciousness. Hill and Wang.         [ Links ]

Nunes de Vasconcelos, G., Malard, M. L., van Stralen, M., Campomori, M., Canavezzi de Abreu, S., Lobosco, T., Gomes, I. F. & Costa Lima, L. D. (2019). Do we still need CAVEs? Simulation - Virtual and Augmented Reality, 2 (3), 133-142. https://doi.org/10.5151/proceedings-ecaadesigradi2019_474        [ Links ]

Plasencia, D. M. (2015). One Step Beyond Virtual Reality: Connecting past and future developments. XRDS, 22(1), 18-23. https://doi.org/10.1145/2809921        [ Links ]

O'Shiel, D. (2019). Phenomenology and the Challenge of Virtuality. In J. Braga (Ed.), Conceiving Virtuality: From Art to Technology (pp. 21-43). Springer. https://doi.org/10.1007/978-3-030-24751-5_2        [ Links ]

Riva, G., Waterworth, J. A., & Waterworth, E. L. (2004). The Layers of Presence: A Bio-cultural Approach to Understanding Presence in Natural and Mediated Environments. Cyberpsychology & Behavior, 7(4), 402-419. https://doi.org/10.1089/cpb.2004.7.402        [ Links ]

Romanus, T., Frish, S., Maksymenko, M., Frier, W., Corenthy, L., & Georgiou, O. (2020). Mid-Air Haptic Bio-Holograms in Mixed Reality. https://arxiv.org/abs/2001.01441        [ Links ]

Sartre, J.-P. (2004). The Imaginary. A phenomenological psychology of the imagination (J. Webber, Trans.). Routledge. (Originally published in French 1940)        [ Links ]

Shields, R. (2003). The Virtual. Routledge.         [ Links ]

Smith, A. D. (2002). The Problem of Perception. Harvard University Press.         [ Links ]

Stein, C. (2016). Virtual Reality Design: How Upcoming Head-Mounted Displays Change Design Paradigms of Virtual Reality Worlds. Media Tropes, 6(1), 52-85. https://mediatropes.com/index.php/Mediatropes/article/view/27101        [ Links ]

Sun, D., Kiselev, A., Liao, Q., Stoyanov, T., & Loutfi, A. (2020). A New Mixed-Reality-Based Teleoperation System for Telepresence and Maneuverability Enhancement. IEEE Transactions on Human-Machine Systems, 50(1), 55-67. https://doi.org/10.1109/THMS.2019.2960676        [ Links ]

Van de Mosselaer, N. (2018). How Can We Be Moved to Shoot Zombies? A Paradox of Fictional Emotions and Actions in Interactive Fiction. Journal of Lie Theory, 12(2), 279-299. https://doi.org/10.1515/jlt-2018-0016        [ Links ]

West, R., Parola, M. J., Jaycen, A. R., & Lueg, C. P. (2015). Embodied Information Behavior, Mixed Reality and Big Data. The Engineering Reality of Virtual Reality, 9392, 93920E. https://doi.org/10.1117/12.2083519        [ Links ]

Wiesing, L. (2014). The Philosophy of Perception. Phenomenology and Image Theory (N. A. Roth, Trans.). Bloomsbury Academic. (Originally published in German in 2009 as Das Mich der Wahrnehmung: Eine Autopsie)        [ Links ]

Yilmaz, R. A. (2016). Educational magic toys developed with augmented reality technology for early childhood education. Computers in Human Behavior, 54, 240-248. https://doi.org/10.1016/j.chb.2015.07.040        [ Links ]

 

 

Correspondence:
daniel.oshiel@gmail.com

 

 

† This article is part of a collection of papers on Phenomenology and Virtuality, with guest editor Jean du Toit