The eXtended Realities (XR – VR, AR and MR) “Sector” – where we are and where we need to be

Introduction

In 1996, a “Competitive Study on the Virtual Reality Market” was prepared by the author for the Technology Foresight VR Sub-Committee, set up by the then Department of Trade and Industry.  One of the key conclusions of that report was:

“A classic business appraisal and SWOT analysis includes figures reflecting the health of an organisation or industry. Given the embryonic state of VR this is not possible. Sales volume would be disproportionately low by virtue of an imbalance between investment and return. VR has invested severally in innovations and is, generally, waiting for a return. There are no reliable indicators of performance beyond those in the public domain. As of mid-1996 not one British VR company is in a position to specify a market trend because there is, as yet, no market to demonstrate a trend. In essence, VR is a solution waiting for a problem and its business history over the past five years reflects this.

VR is classically an innovation-led pursuit. Each participant has a broadly similar idea of the “ideal” intuitive computer interface but has started from a unique innovative breakthrough and is addressing a different set of perceived problems. Thus the VR industry is best defined as a disparate group of companies, currently aiming at different markets and employing different means and technologies, but working within a loose framework because of a similarity in the techniques which they seek to promote.”

In many respects, the situation today is not that different from the 1990s.  Whilst one reads about extensive achievements and developments in XR online, the sector is still far from one where individuals and organisations can generate significant and sustainable ROIs, or can even guarantee that investments (of which, it has to be said, there are far too many being irresponsibly placed with immature start-ups, often led by technologists with little business acumen) will be sensibly deployed with the aim of building a sustainable business and XR product line or service.  The following then, is an attempt to compile a list of strengths, opportunities, weakness and threats currently facing the XR community.  The list is by no means exhaustive, and the components of each category are not in order of priority, but the list will, it is hoped, focus attention on prioritising some of the tasks of the Digital Board of the Centre for the New Midlands going forward.

For appropriate definitions of XR (VR, AR and MR), see: https://www.linkedin.com/pulse/virtual-augmented-mixed-reality-basic-definitions-bob-stone/

Strengths

• The key strength of XR technologies is that, when designed appropriately and correctly (in both hardware and software terms), with a priority focus on the needs of the end user and not the technology per se., they support intuitive, meaningful, efficient and enjoyable real-time interaction with computerised databases. Whitney Quesenbery’s “5 Es” of usability should apply:

• Effective: How completely and accurately the work or experience is completed or goals reached,
• Efficient: How quickly the work can be completed,
• Engaging: How well the interface draws the user into the interaction and how pleasant and satisfying it is to use,
• Error Tolerant: How well the product prevents errors and can help the user recover from mistakes that do occur,
• Easy to Learn: How well the product supports both the initial orientation and continued learning throughout the complete lifetime of use.

• Such computerised databases are typically (but not exclusively) visual, representing a range of synthetic environments, ranging from recognisable, real-world environments or even complex and abstract visualisations. Increasingly, the addition of other human-mediated sensation technologies is becoming mature (e.g. sound, haptics, smell).
• Potential exists for the manufacturing industry to adopt XR technologies moreso than exists at present – especially in relation to future “Metaverse” developments.
• XR also has the potential to provide excellent interface solutions for developers, users and exploiters of AI, machine learning, etc. This also applies to users of data sourced using specialist or remote systems – drones, surface and underwater vessels (e.g. sonar), LIDAR, CT/MRI devices, etc.
• The XR community has delivered and continues to deliver a variety of implementation options (HMDs, mobiles, screens, 360, etc.), all of which offer effective human-centred solutions to a variety of applications, as long as their implementation is (again) accompanied by unwavering attention to the needs of the end users.
• A small, but (hopefully) growing number of excellent XR case studies are becoming available (these need to be identified, catalogued and published widely).
• XR Hardware technologies are becoming highly affordable.
• Entry-level costs (using existing online or company-generated content/assets and real-time rendering toolkits) are becoming very affordable, in some cases free.
• Toolkits for modelling and real-time rendering are better and more usable than ever before (but, see weaknesses).
• CAD – the mainstay of design and prototyping across a wide range of engineering and manufacturing industries – now a reliable and, in the main, robust source of basic 3D data for import into XR toolkits. Some established CAD packages have their own “VR” add-ons.
• There is an excellent and huge online availability of 3D assets and real-time effects for different toolkits, many priced at very low levels (e.g. Unity Asset Store).
• XR plays to the creative interests and talents (and expectations) of the current generation.
• XR offers significant opportunities for the future education sector – schools, universities, colleges, CPD, etc. (but see threats). The technology is capable of introducing valuable experiential learning content, using technology recognised (and, increasingly expected) by the younger population, across the National Curriculum.
• Recent web-based platforms for networked VR have shown significant promise, especially in the delivery of conferences and other events during the COVID-19 pandemic (but it must be recognised that their current style and limitations make them not the choice of all). The networked XR community has a significant contribution to make, especially in the education and healthcare arenas.
• The XR field boasts over 35 years of history, including important lessons to be learned, even of relevance today (these lessons – positive and negative – need to be made explicit; often they are ignored).

Weaknesses

• Definitions of XR and its sub-classes of VR, AR and MR still differ wildly, helping to fuel market and end user confusion. In general, the management of adopter and end user expectations throughout the XR arena is poor and is only growing, fuelled by the mixed messages put out by numerous commercial and academic organisations on the international stage.
• As evidenced in the 1990s and regularly ever since, hype and false promises regarding the capabilities of XR technology are prevalent (compromising, as before, the management of end user expectations).
• The perception of XR as being accessible only via wearable technologies only (headsets, gloves, suits, etc.) is, as history confirms, an erroneous one, but the headset “purists” continue to push out the illusion that such technology is the only way to achieve “immersion”.
• Despite examples to the contrary, there is still a widespread perception of XR being a glorified game. This preoccupation has existed since the early 1990s.
• There is still a perception of XR as representing the “ultimate” in human-computer interfaces. This is not the case today and, even in the future, other interface technologies will persist.
• Recent commercial developers in XR have taken to using (abusing?) the term “enterprise” to detach their products from a potentially large consumer market, and to justify greater costs, which they expect commercial and industrial organisations to accept.
• Another weakness evident since the 1990s is the regular publication of expensive market surveys and 10 to 20-year (+) prediction documents put out by individuals and groups who are not “at the sharp end” of XR and have a questionable track record or minimum involvement in the field.
• Related to the above is the reliance on XR market reports by investees as evidence of the future “value” and penetration of their product or service and by investors as a means of shortening the due diligence process.
• It is a fact that there are far too many “self-proclaimed” XR “experts” and “expert groups (including start-ups and academic units) operating online, with very little experience or track record in the XR field. Unfortunately, these “talk-the-talkers” are often invited to participate in conferences and panels, as they are more likely to make positive and over-enthusiastic claims about what the technology is capable of delivering (often when it is blatantly not), as opposed to discussing pros and cons, and the negative as well as the positive.
• Related to the above, and contributing to the increasing market confusion that exists across the XR community, are the number of annual regional, national and international conferences that have become evident over the past decade or longer. Many of these take the form of panels of so-called “experts” (with the same faces appearing time and time again), putting out over-enthusiastic messages about the XR community.  As with the comments regarding XR experts above, down-to-earth, matter-of-fact deliveries by conference and panel participants are few and far between, painting a picture of the XR scene through rose-tinted glasses.
• Although not frequently evident in online sites, it is apparent that many of the technological “front-running” products evident, once delivered, demonstrate concerning reliability issues. This is indicative (as found in the 1990s and early 2000s) of rushing what is essentially a beta or prototype product to market.
• Whilst there are many very good introductory and advanced courses to XR – some even available free of charge – many are not geared towards the infrequent or “casual” user of XR. Rapid courses, introducing the basics of VR, for example, are not available to (again as an example) support school teachers in introducing VR into their courses.  The same might be said for University students who are not majoring in the computer sciences domain, but may wish to undertake an XR final year project as part of their degree in some other discipline.
• Despite the existence of capable toolkits and extensive online asset databases providing low-cost, sometimes even free assets and effects, there is little guidance on realistic pricing strategies for XR design, development and implementation effort.
• Undercutting the bids and offers by experienced XR development teams does occur, including by large organisations in certain sectors, often with no XR experience, but with the desire to become involved in XR activities (notably in defence).
• Although the accessibility and usability of XR toolkits have drastically improved over the past two decades, many still rely on specific programming skills and talents to produce the greatest quality output.
• Although there are definite “front runners” in XR, there are still many other toolkits available, a situation which does not help the levels of market confusion currently evident. The development of toolkits is unlikely to subside in the near-to-mid-term future.
• “Non-standard data” – survey data from drones, surface and underwater vessels (e.g. sonar), LIDAR, CT/MRI devices, etc. are still difficult and relatively costly (in terms of effort) to convert into real-time visual representations (this not yet a seamless process – some require specialist services).
• As in the 1990s and 2000s, academic/EU-funded centres are still being announced, all with similar “mission statements” and planned activities as demonstrated in the past. For XR to succeed, it is important that the technologies and applications examples (and the procedures/processes by which they are created and to be implemented) are taken out to real users, undertaking real tasks in the real world.  History demonstrates clearly that academic centre/hub/institutes and so on have not been successful in this respect and have not demonstrated as much real impact on end user organisations and groups as they should.  There needs to be more “policing” and holding to account in this area.
• Human Factors and Human-Centred Design processes are still significantly missing in the activities involved in the design, evaluation, delivery and evaluation of XR systems.
• Further to the above, it is also evident that lessons are not being learned from the history of XR (these, too, need to be identified, catalogued and published widely).
• Over-hyping of specific classes of XR technology are evident (and misleading), including AR and haptics. In addition, meaningless and erroneous examples of “superiority” claims, such as suggesting AR will surpass VR and vice versa indicate a serious lack of understanding of the potential of each “sub-class” of XR, again leading to market confusion.

Opportunities

• Stand-alone/tetherless XR technologies (such as current-generation headsets) with increasing sophistication (wireless, 5G, etc.) without doubt represent an important future opportunity – these devices will undoubtedly enhance the fortunes for wearable devices specifically and, XR adoption more widely.
• The publication of concise Human Factors/Human-Centred Design guidance, supported with strong academic/industrial evidence and case studies is essential to underpin the future development and risk-free adoption of XR (e.g. an update, at least, to author’s 2012 MoD-sponsored publication – shorter, more concise with references to other online sources?).
• The provision of independent and relevant Human Factors guidance and consultancy to third-party projects also presents significant opportunities for qualified, experienced practitioners.
• The development and publication of XR-relevant human performance metrics (objective and subjective) and metrics supporting the reporting of commercial ROI (financial and evidential) needs to be given far more serious attention today than ever before, if wider adoption, especially id adoption amongst technology-sceptical organisations is to improve.
• And, in addition to the above, the development of a widely-accepted “Gold Standard” for experimental designs and analyses is desperately needed – with evidence published on the international stage.
• Also required are high-quality (marketing rhetoric-free) publishable/published case studies across different XR-adopting sectors.
• There are still significant opportunities for delivering enhanced and minimally intrusive XR perceptual and motor delivery wearables and associated and sensing technologies (e.g. olfaction, locomotion, motion capture and haptics).
• Significant opportunities exist to introduce XR technologies and applications into education – schools, universities, colleges, CPD. Opportunities exist for the development of very basic and short XR courses – for pupils, students and teachers alike.  This could have a significant impact on issues such as the “Digital Divide” and strengthening the Digital Supply Chain.
• Retail and tourism applications are often mentioned online – there is definitely potential in those sectors (especially if “merging” tourism with cultural heritage), but more case studies and evidence are required.
• Healthcare still presents the XR field with its greatest challenges and opportunities. In many respects, the technologies being used today in this field, especially with regard to surgical training and/or in-theatre support, are inadequate.  However, applications in pain control, medication support, geriatrics, healthcare personnel stress and anxiety are well developed.
• Significant opportunities for XR developments exist in the space and subsea sectors (training, visualisation, telepresence, etc.).
• Related to the above, many XR technologies can be effectively adopted in other sectors – remote operations (space, subsea, battlefield, mining, supervisory control and “seamless” reversion to manual control for autonomous systems, etc.).
• Significant opportunities also exist to support remote, collaborative working techniques (beyond those stimulated by the COVID-19 pandemic). This is also a selling point for those current proponents of the “Metaverse”.
• Brain-Computer Interface (BCI) implementations, many embedded within XR wearables, offer huge potential to researchers and training authorities wishing to quantify human performance in complex systems. But see comments in threats.
• As mentioned above, opportunities exist courtesy of the many online platforms currently available or being developed to support the “Metaverse”. Education and manufacturing are but two sectors capable of benefiting.
• Opportunities exist in medicine, defence, space, subsea and other applications domains for true room-scale holograms (volumetric displays in the shorter term), supporting true 3D perception.
• Opportunities are definitely available for more effective collation and dissemination of XR case studies, presented realistically in the form of case studies, “piggy-backing” via parallel sessions onto mainstream, peer reviewed conference events in healthcare, defence, education, heritage and others. In this case, the presenters are not, as is often the case “preaching to the converted” and have to justify their developments to critical audiences. Similarly, the “same-old, same-old” situation with presenters and presentations, as witnessed all-too-often in XR conferences can be avoided.

Threats

• Continuation of potential adopters believing all they see and read regarding XR online is a persistent threat – how can we manage expectations when trying to sell or support?
• XR company sales teams/individuals continue to “over-egg” what the technology is capable of. Sales representatives are well known for selling the hype of XR to potential customers, especially new sales representatives who may have come to the XR sector for the first time.  This is still evident today (and was modelled as part of a business-related VR project in the late 1990s).
• The performance of so-called “mentors” set up by academic and nationally-/internationally-funded initiatives is concerning, especially when institutions front those who are not experienced enough to deliver truly informed and independent XR advice/support for technology adoption, start-up management and so on.
• A particular threat exists in the medical sector relating to the use of unlicensed, unproven, non-medically-approved technologies for training and in-theatre XR application. A related concern is that approval for use of such technologies may be given by established medical regulatory bodies with personnel who are likely to have only a limited appreciation and experience of the XR arena and the risks posed by the current immaturity of candidate technologies.
• Related to the above is the potential impact on the XR community – including high-profile litigation – of errors (and especially injuries or fatalities) caused by the procurement of, or the inappropriate adoption and/or use of immature and unproven medical XR technology.
• Large-scale failure of LBEs is also a threat (again – caused in the past by the evolution of home computer gaming and the inability to refresh the XR experiences in a timely fashion to attract return visitors). Hygiene issues, especially post-COVID, are an area of concern as well.
• A threat seen in the past also exists today, namely investor lack of confidence/tolerance, brought about by poorly-placed or squandered investments, or mythical exit strategies.
• XR Start-up company directors with little or no business acumen pose a big threat to the XR community, which can ill afford another “dot com”-level failure, as seen between 1995 and 2000.
• In addition to the above, a similar threat to the XR community is that posed in the event of increasing evidence of start-up failures.
• Reinvention of the wheel is a persistent issue in today’s XR arena, with developments and even patents being exposed with zero reference to prior art and historical XR achievements.
• Post-investment diversification is also a threat – vividly witnessed in the 1990s and early 2000s (i.e. companies losing focus on developing the technology for which investment was first provided and spending investments on risky developments outside of their initial product line or service that are very much the province of other established and capable groups).
• A threat of concern for now and especially the future is misuse of personal data captured using various techniques – posture, eye-tracking, speech, BCI. Putting to one side the attitudes of certain companies to the capture, via XR technologies, of personal data, AI techniques are already being used to capture “semantic and behavioural” aspects of human performance, for selecting candidates for certain jobs.  Where are the controls for this?
• Further to the above cases of “Metaverse” hacking and cyber-attacks pose a threat to widespread adoption and exploitation (see author’s recent position paper on this topic).
• Education (school and university) in VR tools and techniques is weak at the present time and threatens the digital supply chain for future XR developers and adopters. There is potential resistance to adoption at teacher/school/college level. Training costs are also an issue.
• Broad standards across the XR community are, at the present time, unnecessary, especially given the rapid way in which technologies and process are evolving. Immature standards now, typically written by committees (often not well versed in the XR field) could pose a threat to innovation and creativity and result in costly third-party “midware”-type solutions.
• Home XR systems could, once again, threaten the attraction/existence of LBEs.
• Cyber/VR sickness – online postings tend to over-exaggerate the extent of the threat, but could compromise adoption. Technology alone is unlikely to rid the XR community of such problems, but due diligence when developing applications, and exposing them to end users, is definitely a necessity (and guidelines exist).
• A threat that has existed for many years is that national and international funding organisations are not demanding enough concrete evidence of real-world impact – certainly not before the final contract payment has been made. This could threaten the XR adoption process by allowing poor-quality R&D effort to be perceived as the “norm”.  This is especially true of academic XR centres.
• XR associations and the rhetoric they put out, plus the events they hold, could similarly compromise the adoption process. XR associations need to be controlled in terms of who they are, what they offer, especially in the case of subscription-based set-ups, and what tangible evidence they can supply to demonstrate to support the community.  At the moment there are too many such associations and braches, with little to report in terms of achievements and impact.  Similar comments apply to the XR award “circuit”, many XR association awards being meaningless, value-less transcripts.
• A final threat, although nowhere near as significant as some of those listed above, is the inappropriate and erroneous use of definitions and terminologies (and, again, the potentially negative impact on managing adopter expectations) – hologram and “Holodeck” are typical examples.

Conclusions

Unlike the exercise undertaken for the DTI in the mid-1990s, the situation for XR today, after a 3-decade rollercoaster ride of successes and failures, is a complex and involved one.  The international scene can be characterised as a confusing market of products, techniques, definitions and commercial groups, the greater majority of which are constantly vying for “supremacy”, often using whatever over-hyped messages they can concoct.  Newcomers and early adopters regularly report that, whilst they are keen to become involved in XR, they find the community hard to navigate when it comes to receiving genuinely independent and informed advice regarding how to move forward with their application.  Others report dissatisfaction with the solution they have been sold, including poor after-sales support and service.  This is very reminiscent of the mid-1990s and, despite the fact that the international marketplace was much, much smaller, these and other negative experiences were a major factor in the demise of VR towards the end of that decade and early on in the next.

Again in the 1990s, and following the publication of the “Competitive Study on the Virtual Reality Market” referred to earlier, it was decided that the best course of action at that time was to set up what was loosely referred to as a “VR Trade Body”, underpinned by the DTI and open to membership from organisations active in VR (academic institutions were invited much later). The “VR Forum”, as it was known, was proposed to be a body of representation of Virtual Reality practitioners and users in the UK whose main aims were:

(a) to promote national and international business opportunities through a coordinated campaign of awareness,

(b) to present itself as a body of competence in the field of VR, acting as a focal point for independent advice and guidance,

(c) to foster consistent technical and commercial excellence through the coordination of research, development and educational programmes, and

(d) to provide advice and guidance in the exploitation of the results of such programmes throughout the VR industry.

For a variety of reasons, the VR Forum, which was launched in 1998 (at a time when the VR community was experiencing a national (indeed global) downturn), was unsuccessful in bringing the UK VR community together.  A number of reasons were cited for this outcome, including a lack of interest on the part of the membership to share experiences, contacts, IP and so on, and an unwillingness to expend time travelling to London or other regional settings for regular meetings.  Setting up something similar today on a regional level, even with today’s more efficient methods of group communication would, more than likely, result in a similar failure.

The two most popular outcomes from the launch of the VR Forum was (a) the collation and distribution of commercial, academic, conference and other relevant material, and (b) a widely-distributed CD, entitled “Getting Started in Virtual Reality”, containing definitions, case studies, technology overviews, references/links and “What Next?” advice.  This was designed to help groups structure their thoughts about adopting VR for their own commercial advantage and providing them with recommendations on how best to approach VR companies and groups for consultancy or developmental support.  Another popular aspect was the VR Forum’s activities in delivering regional “awareness raising” events.  However, given the logistics in moving demonstrable VR technology around the country, or relying on local businesses to provide free technical support (thus removing said technology from their day-to-day process of delivering solutions to customers), this too was short-lived.

So what can the Midlands do (a) to support the longevity and growth of its rich, if somewhat disparate XR community (b) to stimulate innovation and creativity by supporting the uptake of XR across the region, and (c) to demonstrate leadership in providing independent, experienced advice and guidance to potential adopters of XR primarily (but not necessarily exclusively) throughout the region?  The region has an impressive story to tell, with many groups being highly active in the XR domain over many years, with a number of high-profile projects conducted well beyond the boundaries of the region.  But, historically, the region has not voiced its pedigree loudly enough, resulting in other regions of the UK, such as the North West and South West, gaining greater exposure, not necessarily, it has to be said, as a result of high-impact achievements.  Setting up a single body, as was attempted with the UK VR Forum in the 1990s, is, as was stressed earlier, likely to meet with limited success, even failure.  However, in order for the region to grow a significant capability in this field, it will be necessary to form a small group of committed and proactive individuals to help coordinate the activities proposed below.  Furthermore, in order to make an initiative of this kind and scope stand any chance of being successful, there must be explicit support from the West Midlands Combined Authority (at least) – most likely at Mayoral level – to help endow the XR capabilities and aspirations of the region with an appropriate level of gravitas and, thus, a highly visible reason for organisations to commit support.  Just some of the activities (and this list will undoubtedly be refined in the weeks to come) are:

• Develop a comprehensive “catalogue” of regional expertise and resources – individual, group, commercial and academic – within the region, focusing not just on technology availability, but on specific skills, knowledge and experiences – general and sector-specific – that could, under the right circumstances, be brought together and tailored to support potential; adopters in the future.
• Identify sector-specific organisations (healthcare, education, transport, automotive, defence, heritage, tourism, retail, etc.) where XR could be, but has not yet been implemented or exploited.
• Organise a small number of within-region seminars (possibly sector-specific), based on the findings of (a) and (b) to establish whether or not there is an appetite for support the adoption of XR and what the barriers to said adoption might be. The seminars could feature speakers from new and established XR groups, but must paint a realistic picture of the pros and cons of XR, the possible costs, sources of support and so on.  Marketing talks must be avoided at all costs.
• Engage with schools, colleges and universities to investigate how best to develop and deliver learning material suitable for all ages and for a range of courses and elements of the National Curriculum. Consider a bi-annual XR competition, soliciting sponsorship and training support from regional companies, associations and national funding bodies, for educational establishments to develop their own XR concepts addressing key societal issues (e.g. from policing to climate change, healthcare to the impact of natural spaces).
• Engage with regional trade and professional associations and with key national institutes (e.g. the IET, BCS, CIEHF) to support the collation and distribution of relevant material nationally and internationally.
• Ensure an explicit presence for the Midlands’ XR capability at key national and international events.
• Generate an online library of relevant XR material and resources, updated regularly, structured in such a way as to be of benefit to those early adopters seeking further information and guidance (along the lines of the successful “Getting Started” CD mentioned above.

About Professor Bob Stone, C.Psychol., FCIEHF

Bob Stone is a Human Factors specialist and the UK’s foremost immersive technologies specialist – a 34-year “veteran” of the international Virtual, Augmented and Mixed Reality community.  In 1993, whilst researching VR and robotics at the UK’s National Advanced Robotics Centre, Bob established the world’s first industrial VR team, launching a countrywide collaborative VR initiative, wholly funded by industry.

Bob’s research has taken him from Royal Navy vessels conducting close-range weapons and missile trials to underwater operations onboard submarines and rescue submersibles; and from search-and-rescue helicopter missions to operating theatres and medical units throughout the UK, US and South Africa.

Today, as well as being an Emeritus Professor at the University of Birmingham, Bob works closely with the Royal Centre for Defence Medicine and various UK hospitals, researching the use of VR and MR for physical and mental health restoration and rehabilitation, and for the training of future military Medical Emergency Response Teams. He also has a passion for the Virtual Heritage arena, exploiting VR and AR to help make invisible rural and oceanic historic sites visible once again, particularly around the Plymouth area where he was born and bred.

In 2011, Bob was awarded the Ministry of Defence Chief Scientific Advisor’s Commendation for his contributions to Defence Science & Technology, and, more recently, his team received the Chartered Institute of Ergonomics & Human Factors 2020 Innovation Award for their defence medical work.

Connect with Bob on LinkedIn

Follow Bob on Twitter