801

1 INTRODUCTION

Unmanned harbour tugs are being commercialized to

reduce risks of the human factor in towing operation

and labour costs caused by long idling time. With the

fast-growing state-of-the-art technologies such as 5G,

Virtual Reality/Augmented Reality technologies, and

advanced sensors, a high degree of stability of remote

control and situation awareness is more feasible than

before. Rolls-Royce and Svitzer [11], Kotug [8],

Wärtsilä [16], NYK [10], and Samsung Heavy

Industries [12] have successfully demonstrated remote

tug operations with different kinds of remote control

centre demonstrators.

However, there are still major challenges that

compensate for sensory loss since important sensory

channels like kinetic and tactile ones are eliminated

during remote operations. Furthermore, for a remote

operator, sufficient information must be transferred

from the ship to the shore control centre in a timely

manner to gain adequate situational awareness.

It is essential for tug operators to have sufficient

situational awareness to analyse the situations, plan

actions, and execute remote control. However,

achieving a high-quality visual presentation with less

sensor loss is very challenging with conventional 2-D

camera-based approaches and additional information

in the form of text or sensor data. Furthermore,

substantial investment for the equipment in the shore

control centre is not avoidable.

Hence, the goal of the FernSAMS Human-Machine

interface is to improve the situation awareness with

VR/AR technologies and reduce the cost for setting up

the shore control center. This paper is organized as

follows. In Chapter 2, we present the basic concept of

FernSAMS and its shore-based remote control center.

After the overall introduction of FernSAMS, in

Chapter 3, we describe the Human-Machine-Interface

within the FernSAMS project. After a short result

Remote-Controlled Tug Operation via VR/AR: Results

of an In-Situ Model Test

S. Byeon, R. Grundmann & H.C. Burmeister

Fraunhofer Center for Maritime Logistics and Services, Hamburg, Germany

ABSTRACT: The German-funded FernSAMS project aimed at the development of an unmanned, remote-

controlled tug operation with AR/VR technology. After an extensive simulation test with ship-handling

simulators, the developed FernSAMS AR/VR system has now been in-situ tested with a scale model of the tug.

The model test results showed very robust stability in remote operations with improved situational awareness

with the VR/AR system and sensors. After a short introduction of the FernSAMS concept as well as some first

insights into FernSAMS Human-Machine-Interface tests within the simulator, this paper introduces the

technical setup of the scale-model tests being conducted with the FernSAMS concept to test the operational and

technical feasibility of AR/VR-based remote control. This includes an overview of the systems and sensors

integration and an analysis of the effectiveness of AR/VR system combined with 360-degree video streaming.

http://www.transnav.eu

the International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 15

Number 4

December 2021

DOI: 10.12716/1001.15.04.12

802

analysis from simulation run in chapter 4, chapter 5

presents the In-Situ Model Test's technical setup.

Finally, we summarize the results from In-Situ-test

and discuss the future outlook in Chapter 6.

2 FERNSAMS CONCEPT

The typical harbour tug operation consists of several

stages, from unberthing, transit, and towage operation

to berthing. The FernSAMS project is aiming for an

unmanned, remote-controlled tug based on Voith

Schneider propulsion (VSP), Macgregor Maritime

Data Engine (MDE), Media Mobile communication

system, and FernSAMS assistance system (Fraunhofer

CML) during this harbour operations.

Figure 1. Flow chart for tug operation with remote

FernSAMS assistance system [15]

The development of a remote-controlled harbour

tug including all components required for its

operation necessitates the previous analysis and

specification of operations, which is presented in

detail by [Figure 1]. The considered operations

between the tug being ordered and moored again

range from leaving berth, berthing through transiting,

waiting, convoying and taking position to pulling or

pushing operations with or without establishing a line

connection respectively. This analysis not only

provides the basis to determine the degree of

automation or remote control, but also to specify

features of the tug and its assistance system required

for remote-controlled operation assisted by

autonomous functionalities [15].

2.1 Overview of the FernSAMS

The main concept of the FernSAMS assistance system

is to develop an enhanced HMI with the help of

VR/AR technology and sensors, which enables remote

operation during the assistant jobs and autonomous

operations in manoeuvring mode from RCC (Shore

Control Centre). FernSAMS assistance system is

designed to fulfil the level 3 automation, according to

the International Maritime Organization's (IMO)

definition [7], which means "remotely controlled ship

without seafarers on board."

2.2 Remote control centre

The main purpose of the RCC is to provide the ability

to take control of autonomous vessels from a remote

location, especially as means to avoid critical

situations, collisions, and allisions that are outside the

capability of the automatic navigation algorithms [4].

Typical RCC infrastructures consist of large screen

displays and working stations to provide a clear

overview of surroundings and navigational

information such as Electronic Chart Display and

Information Systems (ECDIS) and AIS/radar, which

shows resemblance to modern ship's bridge.

However, as the importance of RCC increases,

innovative RCC technology and design is needed to

enhance human-machine interactions and safety of

operations.

The hardware requirements for the landing station

were defined during the planning of the onboard tests

[Figure 2]. The landing station will be set up in a room

with a table and a seat. There must be at least 1x1m,

and preferably 2x2m of space around the seat. To

receive ship and sensor data, the PC is connected to

the network via Ethernet LAN, and a 220V connection

is required for power supply. Ideally, the shore

station is located near the in-situ test environment.

Figure 2. Shore Control Station Set Up

3 HUMAN-MACHINE-INTERFACE WITH VR/AR

Situational awareness (SA) describes the mental state

of a person to be aware of the elements in his

environment and their meaning [5]. Situational

awareness plays a particularly important role during

vehicle control, a situation in which the environment

may be subject to high dynamics. The studies [6]

found that 71% of all human errors are mainly caused

by insufficient situational awareness. Therefore,

during the development of the FernSAMS assistance

system, ensuring the operator's high SA was one of

the main criteria for evaluating the human-machine

interface.

3.1 Virtual and Augmented reality

Virtual Reality allows the users to experience the real

world [9], which is one of the key characteristics for

compensating sensory losses like kinetic and tactile

perception during remote control from RCC. VR/AR-

based HMI design in FernSAMS [Figure 3]aimed to

improve situation awareness by creating a

803

virtual/augmented environment that resembles the

actual bridge system with a human-centric design.

3.2 Virtual bridge system

The VR/AR system was implemented with the Unity

3D engine and used the ship's data to generate 3D

visualization. The virtual bridge's current design were

modified and improved based on the feedback and

surveys during simulation runs [15]. The main

information displays and virtual representations are

as follow:

− Navigational data display

− ECDIS / radar

− Top-down display for LiDAR sensor

− Environmental data (wind) display

− Joysticks and touch buttons for VSP command

− 3D space with the virtual ocean, ships, and line

Figure 3. Virtual bridge in simulation

4 SIMULATION RUN

Within FernSAMS, extensive simulation runs and

usability tests have been carried out with a state-of-

the-art RME ANS 6000 simulator with two bridges

from project partner MTC Hamburg GmbH. The

knowledge gained in the simulation test regarding the

usability test with different setups was transferred to

In-Situ Model Test. In the simulations, three different

setups were tested as below and compared to a zero

alternative, where the simulated vessel was directly

controlled. With regards to the test set-up and

detailed results it is referred to [2].

4.1 VR with Oculus Touch Controllers / Oculus Touch

Hand rotation

The new setup of an innovative approach not only for

controlling a tug, but also for inputting and

displaying information initially caused difficulties for

the test person. The unfamiliar controls and the

initially obstructive visual system (due to the image

quality in the VR view) made the manoeuvres

difficult.

Only after a certain training phase and certainly

also with the support of the additional setup solution

mentioned in section 3 was it possible for the test

person to understand the visual system in a

differentiated way. After a certain training phase,

manoeuvres with this setup were possible without

problems. However, it can be assumed that an

adequate period of acclimatisation to the VR goggles

is necessary to enable the helmsman to work under

VR for a longer period of time.

4.2 VR with Voith handles

As mentioned earlier, this setup was included in the

evaluation of the runs based on initial feedback from

users and served to help the test subject become

familiar with the VR system. After initial problems,

this setup proved to be the best among the VR setups

[Figure 4]. The desired functionalities visible through

the goggles and the familiar control via the haptic

feedback on the position of the levers made it easy for

the test person to safely steer the ship as usual. The

lack of freedom in VR due to the missing controls was

not perceived as annoying. In general, the

functionalities were based on templates and menus

and arranged according to subject areas.

The results from this simulation run showed that

the operator preferred to use the No.3 setup to feel

direct haptic feedback of the commanded values to

limit sensory loss [2]. And another important finding

from the simulation test is that one operator can

conduct harbour tug manoeuvring and towing runs

with the developed AR/VR system.

Figure 4. Simulation run at MTC

5 TECHNICAL SET-UP FOR IN-SITU MODEL TEST

After the simulation runs with other industrial

partners, the actual system integration and

communication test have been carried out for the In-

Situ Model test. FernSAMS's assistance system has

been integrated with the Voith Schneider propulsion

(VSP), Macgregor Maritime Data Engine (MDE),

Media Mobile communication system.

5.1 Overview of the system architecture

Figure 5 shows the system architecture for the In-Situ

Model test. The system integration's main focus was

to minimize the latency in the communication and

achieve steady reliability during remote operation.

The distinctive difference with the simulation run

setup is that 360 degree-camera and LiDAR sensors

were integrated with the VR/AR system. The LiDAR

sensor was chosen as 2nd visualization sensor to

detect adjacent objects efficiently with low bandwidth.

804

Figure 5. System architecture

5.2 System integrations

Macgregor's Maritime Data Engine is a data

normalizer that collects and standardizes the data

from the ship's system. MDE provides standardized

Open Platform Communication United Architecture

(OPC UA), a data exchange standard for industrial

communication [14]. The MDE has been integrated

with the voyage sensors and Voith Schneider

propulsion system for the model ship's standardized

data collection. The data collected from the MDE is

converted into XML format and transferred to the

AMQP server in the RCC with the fixed interval via

Virtual Private Network (VPN).

The latency and limitations of the networks, such

as the bandwidth, are the main challenges of remote

rendering [13] from the 360-degree camera into the

VR/AR system. The study [1] shows that low latency

(below 1 second) is achievable with Realtime

Streaming Protocol (RTSP) and VLC plugin

integration within VR/AR system. In the FernSAMS

project, the 360-degree camera was integrated into the

ship’s side local area network, and 360-degree image

rendering can be done via VR/AR system over the

VPN.

For the Point cloud data streaming from the

LiDAR sensor, Robot Operating System (ROS) is

integrated with the LiDAR sensor for the point cloud

data pre-processing. A voxlgrid filter was used to

down sample the point cloud data. The VR/AR system

can subscribe the pre-processed data over ROS bridge

server, which provides a WebSocket transport layer.

In the RCC, all the information is shared via the

AMQP server, which is a message broker allowing

two parties to communicate. The VR/AR system

interacts with Tug Assistance System (TAS), a desktop

application developed by the Qt C++ framework. TAS

subscribes relevant XML data from the AQMP server

and transfers created sea-chart back to the AMQP

server. The MDE interface communicates with the

AMQP server in the RCC bidirectionally.

5.3 Communication

Communication plays a crucial role in remote control

[3]. The communication link should be robust and

secure for the real-time data transmission between the

ship and RCC. In the FernSAMS project, the

communication link consists of three independent

communication channels as below.

− Point-to-Multipoint (PMP)

− Mesh Networks

− Very Small Aperture Terminal (VSAT)

The maximum bandwidth, packet losses, and

latency of each communication channel must be taken

into account to cope with worst-case scenarios that

limit the system's full functionality. Increased packet

losses and latency in the network causes serious

artifacts during the steaming from 360- degree camera

in the VR/AR environment. In the VSAT

communication channel, the video streaming from

360- degree camera is set to be disabled for secure

data transmission in limited bandwidth and longer

latency. Besides, the communication between the ship

and RCC should be monitored. For this purpose, the

information about the connection quality ("0"

undefined, "1" very poor to "5" very good) for each

connection type (PMP, Mesh, VSAT) is transferred to

the MDE server. Therefore, the operator is able to

confirm the network condition and diagnose the

problems in the virtual bridge.

6 RESULTS FROM IN-SITU MODEL TEST

The findings gained in the simulation with regard to

the usability of the various test setups were

transferred to the model tests of the system. An

important outcome of these tests was the need for

direct haptic feedback of commanded values through

joysticks as opposed to gesture-only control to limit

sensory loss. Furthermore, the pass-through AR

interface was highlighted as an intuitive technology to

minimize sensory loss for vision.

Thus, two setups were tested in the simulations:

1. operation from the VR with Voith handles

2. operation from the VR with Oculus Touch hand

rotation

Additionally, a direct remote control with direct

visual contact from ashore was used as a back-up and

zero alternative.

Figure 6. Model Ship

805

6.1 Data tracking

During the test runs [Figure 6] it was possible to

record 23 parameters. The recording was carried out

at 1Hz and stored in the database with a time stamp.

− Id: the sequence of the data

− InstanceID: Instance ID represents different

sessions

− Timestamp

− Longitude (± 0 - 90 Degree)

− Latitude (± 0 - 180 Degree)

− Rate of turn (degree / minute)

− Heading (0-360 degree)

− COG (0-360 degree)

− SOG (kn)

Subdivided between the sent or commanded

values and the received or controlled values.

1. Commanded values

− EOT 1

− EOT 2

− VSP Steering 1

− VSP Steering 2

− VSP Driving 1

− VSP Driving 2

2. controlled values (feedback)

− EOT 1

− EOT 2

− VSP Steering 1

− VSP Steering 2

− VSP Driving 1

− VSP Driving 2

The type and quality of communication was also

investigated and recorded.

− Communication type (1-3): each number represents

a different communication channel (1: Mesh 2:

PMP 3: VSAT).

− Communication quality (1-5): index numbers on

communication quality (5 is the best quality and 1

is the worst quality).

Figure 7. Plot from trial run

For the evaluation, different parameters were

compared to show how the transmission of the data

and the execution of the manoeuvres worked. For this

purpose, in addition to comparing the data, the

positions of the ship were plotted on a map at

intervals of 10 seconds (Figure 7).

Both together give a good impression of the

feasibility of a remote-controlled system. For

explanation, here is one of these plots and the

corresponding data comparison (Figure 8). For

simplicity, the descriptions of the values have been

abbreviated.

In the first graph, the heading and the COG

(course over ground) were compared. In order to

make statements about the quality (CommQuality)

and the type of communication used (CommType),

this was also implemented in the graph and

multiplied by 100 for better representation.

The values from the commanded values and from

the control were compared in separate graphs. The

designations were changed slightly. Position 1 is

always the eighth system and position 2 the front.

This resulted in the following abbreviations:

− VSP Steering VSPS Aft

− VSP Steering VSPS Bow

− VSP Driving VSPD Aft

− VSP Driving VSPD Bow

The suffix com or ste stands for the commanded or

steered values.

Figure 8 Graphical representation of the parameters of a

trial run

6.2 Lessons Learned from In-Situ-Test

In the measurement runs, possibilities for the remote

control of tugs were successfully demonstrated under

the thematically similar running topics as in the

simulation runs. As in the simulation runs, similar

performance indicators were used for the

measurement runs. These were defined in advance

with the corresponding benchmarks in order to be

able to carry out a good evaluation. The Key

Performance Indicators used for the evaluation of the

VR Remote Tug Assistance are basically:

− Stress on the subject,

− Comparison of receptivity between the different

setups,

− Orientation in the different systems,

− Efficiency in manoeuvring the tug

806

Through the runs in the simulator and the

previous assessment of possible control systems, an

ideal set-up for the test person could be established

for the in-situ test. Even with the new perspective

through the VR glasses, which was still unfamiliar in

the simulation, a familiarity with the system and the

operation was now clearly recognisable. The operator

experienced significantly less sensory loss during the

360-degree visualisation in the VR environment

[Figure 9] than in the simulation. The interaction

between the command interface in the VR/AR system

and the model vessel was very responsive in terms of

latency in communication, and the operator does not

feel any significant delays.

Figure 9. Virtual bridge with 360 degree-camera streaming

The graphical representation of the in-situ test runs

provides us with insights into latency times in

connection with different transmission qualities. The

basic finding in addition to the graphical evaluation in

connection with the visual representation of the run

and the underlying objective shows that a tug can be

controlled in a controlled manner with the aid of a

remote control within the framework of the tested set-

up.

A final determination as to whether and to what

extent this concept can be used on tugs in the future

can only be made with the help of a real test on a real

tug.

7 CONCLUSIONS

Based on the executed technical demonstration tests,

the principal technical feasibility of a remote

controlled tug according the the FernSAMS-concept

can not be denied. The legal feasibility has not been

assessed by this project, however, the ongoing work

on international and national levels with regards to

MASS are expected to allow for a legal perspective on

remote tug operation’s feasibility as well. With

regards to the further development, a more in-depth

feasibility study with a full-scale demonstrator vessel

is recommended. Further, additional technical

stabilization measure as well as defined fail-to-safe-

procedures are necessary for a commercial realization.

With regards to the AR scope, the visualization of the

HMI during pure satellite connectivity must also be

further investigated.

ACKNOWLEDGEMENTS

This work is based on the FernSAMS project, which

received funding from the German Federal Ministry

for Economic Affairs and Energy under the grant

agreement number 03S443E.

REFERENCES

1. Börner, K.M., Fuhrmann, A., Bösinger, M.A.:

Performance of Augmented Reality Remote Rendering

via Mobile Network. In: Weyers, B., Lürig, C., and

Zielasko, D. (eds.) GI VR / AR Workshop. Gesellschaft

für Informatik e.V. (2020).

https://doi.org/10.18420/vrar2020_14.

2. Burmeister, H.C., Grundmann, R., Schulte, B.:

Situational Awareness in AR/VR during remote

maneuvering with MASS: The tug case. In: Global

Oceans 2020: Singapore – U.S. Gulf Coast. pp. 1–6

(2020).

https://doi.org/10.1109/IEEECONF38699.2020.9389455.

3. DNVGL: Class guidance:Autonomous and remotely

operated ships. (2018).

4. Dybvik, H., Veitch, E., Steinert, M.: Exploring

Challenges with Designing and Developing Shore

Control Centers (SCC) for Autonomous Ships.

Proceedings of the Design Society: DESIGN Conference.

1, 847–856 (2020). https://doi.org/10.1017/dsd.2020.131.

5. Endsley, M.R.: Toward a Theory of Situation Awareness

in Dynamic Systems. Hum Factors. 37, 1, 32–64 (1995).

https://doi.org/10.1518/001872095779049543.

6. Grech, M.R., Horberry, T., Smith, A.: Human Error in

Maritime Operations: Analyses of Accident Reports

Using the Leximancer Tool. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting. 46, 19,

1718–1721 (2002).

https://doi.org/10.1177/154193120204601906.

7. International Maritime Organisation: MSC 99 Report on

the Maritime Safety Committee on its ninety-ninth

session. , London, UK (2018).

8. KOTUG: KOTUG demonstrates remote controlled

tugboat sailing over a long distance. (2018).

9. Mazuryk, T., Gervautz, M.: Virtual Reality - History,

Applications, Technology and Future. (1999).

10. NYK: NYK completes remote tugboat navigation tests.

(2020).

11. Rolls-Royce: Rolls-Royce demonstrates world’s first

remotely operated commercial vessel. (2017).

12. Samsung Heavy Industries: Demonstration of

Autonomous and Remote-Controlled Ship Operations.

(2020).

13. Shi, S., Hsu, C.-H.: A Survey of Interactive Remote

Rendering Systems. ACM Compututing Surveys. 47, 4,

(2015). https://doi.org/10.1145/2719921.

14. Steinkrauss, U.: Overview: OPC United Architecture,

http://www.ascolab.com/images/stories/ascolab/doc/ua_

whitepaper_techni caloverview_e.pdf, last accessed

2021/02/01.

15. Walther, L., Hartmann, A., Burmeister, H.C., Jahn, C.:

Mariners in the Context of Remote-controlled Tugs.

European Journal of Navigation. 19, 1, 4–10 (2019).

16. Wärtsilä: Wärtsilä successfully tests remote control ship

operating capability. (2017).