Increasing Collaboration in Workplaces Projection with Interactive Capture

Bridging the physical and digital worlds Back in 1975, these are the words used to describe ‘The office of the future’ in the pages of the Business Week: “There will be a TV-display terminal with a keyboard sitting on my desk. I don’t know how much hard copy [printed paper] I’ll want in this world.”

Almost fifty years later, despite the push towards digitization of documents and workflows, in many workplaces physical paper documents are still commonly used. There are statistics suggesting that total office printing volume is even slightly increasing1, 2 and our own insight studies identified the main reasons why office workers still print and work with physical documents.

Even if workflows will become fully digitised, and even if new technologies such as blockchain3 and increased levels of security will guarantee the trust required within legal and regulatory contexts, there are good reasons to believe that paper documents and notes will not disappear from the office environment in the immediate future. However, the current workplace provides minimal support on transforming information between these physical and digital forms.

As part of Cognitive Hub - the platform of services under development by Konica Minolta that uses artificial intelligence (AI) to interconnect and orchestrate formerly disconnected and unstructured data, applications, and services - we have designed the Projection with Interactive Capture (PIC): a cyber-physical system that bridges digital and physical content4. While traditional printers and scanners typically only make a one-to-one exact copy without understanding the information content, PIC is a system that can turn any standard surface (e.g., a table or a wall) into a smart surface able to comprehend, organise and augment the available content. It can recognize physical objects positioned on the surface, giving visual feedback and enabling augmented reality (AR) workflows on these objects without the need for any other AR device. The user can work comfortably in their physical

Very often, a workflow is only partially digitized and some steps still require paper documents. This happens, for example, to obtain an original handwritten signature, or for expense reporting.

Many people prefer to read on paper than on screen5. Paper documents, books and learning materials have advantages that their electronic counterparts are currently not able to offer6. People seem to concentrate more, read faster, comprehend better and navigate better on paper.

In some verticals, regulation requires all records to be archived in paper form. For example, financial audits often require all information on paper. In the legal industry, case files are often archived in paper form.

Many office workers still carry around paper diaries and paper notebooks to meetings because they feel more comfortable in capturing notes using pen and paper rather than directly in digital format.Post-it® is another popular office supply that many people use to keep to-do items, reminders and quick notes.

1 Neville W., Office Printing Statistics 2017. https://www.lasersresource.com/blog/office-printing-statistics, Lasers Resource, last accessed 2018/12/17.

2 West M., Is Office Printing Increasing or Declining? https://www.printaudit.com/printaudit-blog/premier/is-office-printing-increasing-or-declining-answer-yes, Print Audit, last accessed 2018/12/17

3 https://www.theguardian.com/legal-horizons/2018/oct/17/can-blockchain-shake-up-the-legal-sector-for-the-better

4 What is presented in this whitepaper may or may not be taken forward depending on our customers’ feedback.

5 Anne M., Bente R.W., Kolbjørn B., Reading linear texts on paper versus computer screen: Effects on reading comprehension. International Journal of Educational Research 58, pp. 61-68, (2013)

6 Myrberg, C., Ninna W., Screen vs. paper: what is the difference for reading and learning? Insights 28 (2) (2015)

workspace, with physical documents and objects, yet reap the benefits of digital functionality via PIC.

Conceived in collaboration with Konica Minolta Laboratory USA, the Projection with Interactive Capture system easily enables smart digitization of documents, and by integrating with the Semantic Platform, the Distributed Intelligence and other features of Cognitive Hub, provides access to advanced functionalities and AI services.

Increasing Collaboration in Workplaces | March 2019

Kirk Tecu, Yujia Cao and Jiri Vasek | Konica Minolta Inc.

The benefits of PIC for individuals and team working

Make handwritten notes digital

Digitise your hardcopy, on your desk, in a comfortable workspace next to your PC workstation

Randomly access information, analogous to digital search, by querying for information and getting information highlighted on physical hardcopy

User problem The ‘old way’ The ‘new way’ Solution

Make hardcopy information digital (documents, business card)

Projection with Interactive Capture (PIC) constitutes the entry to Cognitive Hub functionalities: with Cognitive Hub’s microservice design it is possible to enable the deployment of loosely coupled containerised applications. As an Edge device, PIC bridges the physical and digital worlds in a comfortable work style:

• it provides functionality normally associated with digital documents, but for printed material • it improves possibilities for collaboration even for remote workers to effectively co-create

new content together, for example, brainstorming on a paper or whiteboard • it enables easy access to digital information with your hands even in contexts that are different from an office desk

Bridging physical documents with digital convenience through Digitization & AR

Search for information in physical documents

Use piece of paper and pen to draw your ideas. Other person’s ideas are projected on your paper. You now naturally share your ideas with each other

Customers can place a produce on the PIC surface and receive product information. Customers can place an item they already own on the surface and PIC will determine the model and size, directing them to the in store location for a match

PIC recognises ingredients and packaging and delivers instructions on food preparation

PIC recognises parts to make the assembly, giving guidance (instructions, assembly videos) and real-time feedback

Display complementary information on/near specific textbook page or object in lesson plan

Use physical sticky notes on a projected grid. Other person sees your notes, and you see their notes in physical/digital collaborative space

User problem

User problem

The ‘old way’ The ‘new way’ and solution

The ‘new way’ Solution

Brainstorm on Whiteboard with colleagues

Retail OwnerProvide customers with product information even when salesperson is not available

Food Service OwnerProvide associates with information and preparation guidance when not onsite or available

Manufacturing OwnerGive workers information and assembly guidance when not onsite or available

EducatorProvide students with complimentary information to their textbooks and study material

Brainstorm with sticky notes with colleageus

Collaboration by Connecting Analogue to Digital through AR Technology

Augmented Expert Systems for interaction with physical objects

Interacting with the user for more comfortable workplaces

7 Understanding the workplace of the future, Konica Minolta Labs Europe (2018).8 Cloud Edge: Optimal Orchestration of Intelligence in a Hyperconnected Era, Konica Minolta Labs Europe (2019).

3D Depth Imager and High Resolution 2D Imager: PIC uses a combination of visual sensors to interact with physical documents and objects. Using computer vision, intelligent character recognition, machine learning, and semantic technologies, the system is able to recognize the content on the work surface viewed by the PIC.

Digital Projection System: by projecting information on and around objects, PIC augments any work surface, providing different kinds of information upon the requests of the users and in connection with the Semantic Platform of Cognitive Hub. PIC allows for presenting any type of visual material, which could be displayed on a standard computer screen, and can overlay this information on physical objects, while not requiring AR glasses.

Cognitive Hub: PIC enables the connection to the rich ecosystem of Artificial Intelligence applications for digital workplaces provided by Cognitive Hub. With a seamless interface, PIC will guarantee the access to different services of the Semantic Platform, with Natural Language Processing (NLP) and dialogue systems enabling the smart search and the Semantic Enrichment and Linking Framework (SELF). It will enable easier understanding of documents, of the users and their operations in the physical and digital working environments7. The Projector with Interactive Capture represents the entry to the augmented intelligence systems of Cognitive Hub.

Edge Processing: By making PIC part of the Light Edge infrastructure8, an Edge computing solution devised by Konica Minolta Laboratory Europe, an improved quality of the experience when interacting with different applications on PIC is assured. Through microservices orchestration between Cloud, Core, and Edge devices, LightEdge can create a fabric that runs secure low latency and high bandwidth tailored services on users’ premises, guaranteeing quick access to information and compelling, timely and relevant interactive experiences for remote collaboration or interaction with physical objects.

Multimodal Interaction: As part of the Cognitive Hub, PIC supports novel interaction modalities including voice, touch and gaze interaction. Users can communicate with a virtual assistant by voice, asking to provide information, to scan a document or to create a summary of the information captured with the PIC’s field of view.

Increasing Collaboration in Workplaces | March 2019

AI at your fingertips

Laboratory Europe

© Konica Minolta Inc. Konica Minolta Laboratory Europe - Konica Minolta Inc.90 Chancery Lane, London,WC2A 1EU,United Kingdom All rights reserved.

At Konica Minolta we think Artificial Intelligence represents the ultimate User Interface. A layer through which users may access a variety of semantic-based and smart data services that offer the benefits of improved, and more efficient ways for working and collaborating. Projection with Interactive Capture is one of the elements in the ecosystem of our research activities about Cognitive Hub that is capable to bridge physical and digital worlds. It enriches our lives by enabling us to handle physical objects in a digital way and to manage digital information within physical contexts. Join us to take part in the challenge to break the boundaries between digital and physical worlds to empower you with AI at your fingertips.

What is presented in this whitepaper may or may not be taken forward depending on our customers’ feedback.

research.konicaminolta.eu/research.konicaminolta.us/research@konicaminolta.eu