789

1 INTRODUCTION

Autonomy and artificial intelligence are disrupting

many sectors, including the marine industry. Many

companies and academia are researching to evolve

the field. Some companies have even started testing

autonomy in real commercial routes (with safety

drivers on board to meet current regulations). In late

2018 a ferry, developed by Finferries and Rolls Royce,

went between two cities in Finland, first navigating

autonomously and then remotely operated when

returning [1]. In Norway, also in 2018, Kongsberg

started testing autonomy on an autonomous ferry

with passengers and cars on board, mainly to reduce

the workload and to increase the safety [2]. To

convince authorities to change regulations to permit

using ships without a crew on board, it is of utmost

importance to guarantee safety. A human onboard a

ship is very flexible, and will in many situations

discover if the ship is behaving strangely or if an

unexpected event arises. When removing the crew,

the vessel will need to incorporate this extra safety

feature into the system instead.

When it comes to safe navigation, to have a correct

position is vital. Nowadays, crew members rely

heavily on the Global Positioning System (GPS) for

this. A loss of the GPS signal, or a jammed or spoofed

GPS, can for a crew-less ship result in hazardous

situations. The global quality assurance and risk

management company DNV GL believes unmanned

ships may need alternative positioning methods to

convince authorities that their safety is satisfactory

[3]. Furthermore, they believe autonomous ships will

not be fully autonomous for many years, but instead

rely on autonomy and remote control in combination.

Rolls Royce also believes this, as they see the

teleoperation of ships as a key technology in the

transferring process towards autonomous ships [4].

Moreover, they claim that the teleoperation of an

autonomous vessel will increase reliability and

performance. The communication link for the

teleoperation system is vulnerable to downtime,

VR Teleoperation to support a GPS-free Positioning

System in a Marine Environment

M. Lager, E.A. Topp & J.Malec

Lund University, Lund, Sweden

ABSTRACT: Small autonomous surface vehicles (ASV) will need both teleoperation support and redundant

positioning technology to comply with expected future regulations. When at sea, they are limited by a satellite

communication link with low throughput. We have designed and implemented a graphical user interface (GUI)

for teleoperation using a communication link with low throughput, and one positioning system, independent of

the Global Positioning System (GPS), supported by the teleoperation tool. We conducted a user study (N=16),

using real-world data from a field trial, to validate our approach, and to compare two variants of the graphical

user interface (GUI). The users experienced that the tool gives a good overview, and despite the connection with

the low throughput, they managed through the GUI to significantly improve the positioning accuracy.

http://www.transnav.eu

the International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 14

Number 4

December 2020

DOI: 10.12716/1001.14.04.01

790

though, and during this time, the system must solve

the actions autonomously.

The work described by this paper has focused on

how to use remote operation to improve positioning

accuracy for small affordable vessels. Unmanned

ground vehicles (UGV) have, for many years, been

teleoperated to master harsh environments during,

e.g., military or search and rescue (SAR) missions [5]–

[7]. Small autonomous vessels at sea are also

essential, and Murphy believes they will play an

important role during future SAR operations [8]. The

challenges with remote control and positioning are

similar for small and large ships. However, the

communication link’s throughput sets a limitation on

smaller, more affordable vessels, as they can not have

a large satellite antenna due to the size, weight, and

cost constraints. This limitation makes the streaming

of video and transmission of high-resolution images

infeasible. For the positioning problem, we have, for

the same reason, confined ourselves only to use

affordable navigation sensors.

Figure 1. A participant of the user study taking a bearing

by pointing towards an augmented landmark.

The positioning system is built upon our previous

implementation with terrain-aided navigation (TAN),

presented in [9]. This paper estimated the position

from a real-world field trial by comparing the bottom

depth and magnetic intensity with available maps. To

enhance the position accuracy even further, we

manually measured bearings to landmarks from the

recorded 360 image, making it possible for the

positioning tool to adjust the position estimation

accordingly. This is not possible to do manually on an

unmanned ship. In this new work, a user instead

measures these bearings from a teleoperation system

in virtual reality (VR), see Figure 1.

The teleoperation system also builds on our

previous work, presented in [10], [11]. This work

focused on developing a teleoperation tool with a

low-cognitive load that could provide a good

situational awareness (SA), leading to better safety for

the vessel. In the work described in the latter paper,

we developed a specific GUI to compare the

performance when using VR, 3D visualization on a

laptop, and 2D visualization on a laptop. In this

earlier study, we observed that the longer available

time for decisions at sea, measured in seconds or

minutes, makes it ideal for teleoperation. This

contrasts with the fast dynamics of the traffic

situations for cars and airplanes, often measured in

milliseconds, reported as challenging teleoperation

areas due to the vulnerability from mainly long

latency [12], [13]. Several research papers propose

methods to compensate or predict the teleoperated

vehicle’s pose to mitigate the latency problem [14]–

[16]. We use this knowledge to predict our current

position based on heading, speed, and the received

estimated position from the remote vessel. We

concluded in our previous study that 3D, and

especially VR, gave the best performance. VR can

strengthen the visualization, and thereby the total

communication between the machine and the human

[17]. It has also been shown that VR can enhance SA

when driving a remote car [18], [19]. Because of our

good results for VR in our previous work, we use

only VR in this current work. Here we have re-built

the GUI to evaluate how teleoperation can support

navigation, and more specifically, the TAN

application. We base the user evaluation on

recordings from a field trial to make the user

experience as realistic as possible.

One of our main objectives has been to provide the

user with an immersive experience that provides

good SA. To gain trust in the system’s ability to

navigate, it is essential that the user gets a good

overview and instantly can determine whether the

position is estimated correctly or not. When

navigating onboard a manned vessel, the usual way

of doing this is to try to match the real-world terrain

with objects on the sea chart or radar and try to judge

if the directions and ranges coincide. The mental

rotations needed for this task are difficult for a

human to perform [20], [21], and we believe it is even

more challenging to do remotely, i.e., by comparing

what is seen on a video screen with what is seen on

the sea chart. Porathe concluded it is better on

manned vessels to guide the operators by visualizing

a 3D map oriented to match the user’s view of the

surrounding world [20]. Figure 1 shows that we have

built our GUI corresponding with this research, as the

user will see if the real-world corresponds to the 3D-

world, and thereby the position, easily and instantly.

Moreover, if the system’s position is not entirely

accurate, the user can enhance the position accuracy

by providing new bearing updates to the positioning

system.

Our main contribution is to provide a GUI design

for ship teleoperation providing good situation

awareness, which meets the limitation of ships with a

low throughput connection. We have shown that the

users experienced the GUI to be simple to use while

having a good overview of the situation. When the

positioning system estimated an inaccurate position,

the users could react upon this instantly.

Furthermore, we have shown that our TAN

application can be supported remotely by an operator

taking bearings to landmarks.

This paper is organized as follows: Section II

describes the Implementation and Method of the

project, including the design of the applications in

Subsection II-A, the Field Trial in Subsection II-B, and

the User Study in Subsection II-C. The results are

given in Section III, followed by Discussion and

Conclusion in Section IV and V.

2 IMPLEMENTATION AND METHOD

This section describes how the software for the

teleoperation tool and the positioning tool, called

791

TAN application, have been designed and

implemented, followed by a description of the field

trial and user study.

2.1 Design

In our study, we focus on a GUI for teleoperation of a

small ASV via a connection with limited throughput,

which inhibits the transfer of video or high-resolution

images. The ASV is expected to be semi-autonomous

to handle a SAR mission but is still assumed to need

some human supervision to take measures if

something unexpected happens.

We have developed the GUI to suit a small ASV with

a computer capacity and sensor suite comparable to

an autonomous car. The postulated sensors and

capabilities are:

− Global Positioning System (GPS), (only used for

ground truth in the study, as we want to test the

system using the redundant navigation system).

− A redundant navigation system, which can

estimate the global position. In our case, this has

been accomplished by fusion of compass and

speed log data with data from a particle filter (PF)

comparing available maps with bottom depth and

magnetic intensity [9].

− Camera with 360 coverage.

− Satellite communication system with a small

antenna, providing a bandwidth of 256kbps.

− Application for cropping and compressing

images, so that the ship can transmit panoramic

images with a frequency of 0.1Hz, as well as an

image with enhanced quality in the operator’s

direction, with a frequency of 1Hz.

2.1.1 System as a whole – Architectural Overview

An autonomous ship contains multiple sub-

systems, all interacting with each other to create a

smart system that can perceive its environment and

act upon it. In this project, we focus on two sub-

systems, the teleoperation tool to remote control an

ASV, and the TAN application, which is used as a

redundant positioning source to complement the

GPS. The two subsystems are important on their own,

but we evaluate how they can interact and benefit

from each other in this project. How does the position

estimation influence the user experience of the

teleoperation tool, and how can the teleoperation tool

strengthen the TAN application’s performance?

Figure 2. An architectural overview of the system.

Figure 2 shows the information flows between the

subsystems. The TAN application will run on a

computer onboard the ASV, making it possible to

receive all ship data in real-time. The teleoperation

tool receives heading and speed together with

cropped and compressed images. The TAN

application sends the estimated position to the

teleoperation tool, which transfers bearings to

landmarks in return to the TAN application. The

TAN application uses these bearings together with

the heading, speed, depth, magnetic field, and the

pre-loaded sea chart and magnetic anomaly map. All

the interfaces between the TAN application and the

teleoperation tool will be transmitted through a

satellite link.

Figure 3. The GUI of the TAN application. The upper right

corner shows an enlargement of the current operation area.

The ship is going in the east direction. The particles are

visualized as grey and pink dots, where the pink dots have

just been discarded due to being outside of the bearing’s

cone towards the lighthouse in bearing 107. The blue dot

in the middle of the enlarged image indicate ground truth,

which comes from the GPS. The large grey dot indicate the

mean of the particles, and the pentagon the estimated

position from the Kalman filter.

2.1.2 Terrain-Aided Navigation

The project described in this paper builds upon our

previous work with a TAN application, which

estimates the position by using a particle filter (PF) to

compare known maps to depth and magnetic

intensity measurements [9]. We concluded that the

position estimation gains accuracy when using

multiple information sources instead of only using

either depth or magnetic intensity separately.

Figure 3 shows a screenshot of the GUI. For a better

understanding, a video recording from the user study

can be found on YouTube

(https://youtu.be/zu40PEsk5cQ). The upper part of

the figure shows the sea chart with an enlargement of

the image showing the particles estimating the ship’s

position. The lower part shows the 360 image. In the

bearing 107 in the 360 image, a bearing to a

landmark has just been detected. This is shown in the

sea chart as a long blue cone originating from the

landmark in the middle of the figure and stretching

towards the ship. To satisfy the bearing measurement,

the particles outside the cone are discarded, indicated

as pink dots. In the user study, this GUI has been

used for evaluating the TAN application’s

performance. The participants have not used it.

792

Tests showed that the positioning gained in

performance from using the bearings to landmarks.

These bearings were measured offline manually from

the high-resolution images, which will not be possible

in an unmanned vessel. A more realistic scenario is to

use either image recognition software to detect

landmarks, or that a human marks the landmarks

from a remote location using low-quality images. In

this project, we use the latter approach, where the

user detects the landmarks in VR from a remote

location.

2.1.3 Graphical User Interface for teleoperation

We have implemented the GUI for the

teleoperation tool in Unity 3D [22], which game

developers usually use for creating 2D and 3D games.

We have used a 3D replica of the real world as a

foundation for implementing the GUI. This 3D

environment has been developed from maps and sea

charts by the shipyard Saab Kockums AB [23].

The operator teleoperating the ship is virtually

placed on board the virtual ship, positioned in the 3D

environment according to where the TAN application

is estimating the position. The tool receives speed and

heading from the remote ship, which are used to

move the vessel between each GUI frame. The 360

image is updated only six times every minute to

minimize the bandwidth usage. The zoom image,

which is an image with better quality in the pointing

direction, is updated every second. The GUI presents

the images with some latency to simulate the slow

satellite communication link.

Figure 4. A frame of the high resolution (16384x8192) 360

video, recorded during the field trial.

The GUI uses the panoramic video from the field

trial both to create the zoom image and the 360

image. The video quality is high, with 16384x8192

pixels of resolution. Figure 4 illustrates this with an

example image. The problem is that the size of the

images that build up the video is large and can not be

transmitted in real-time over a satellite connection

with low throughput. To meet the limitations, we

have cropped and compressed the images. Figure 5

shows both the zoom image and the 360 image in the

teleoperation GUI, where the 360 image surrounds

the user and the zoom image is in the direction of the

pointing device. The zoom image is presented in front

of the 360 image so that the better quality image

covers the lower quality image. It slowly moves away

from the user and vanishes behind the 360 after a

few seconds. If, e.g., holding the pointer steadily

towards a specific object, one new image in that

direction will appear every second.

To minimize the throughput, the 360 image is

sent with 0.1Hz and the zoom image is sent with 1Hz.

The compressed panoramic image has a size of

around 30kB, and the compressed zoom image with

higher quality has a size of 3kB. This results in a

throughput of 6kB/s, or 48kbps, which is a lot less

than the 256kbps capacity of the communication link,

leaving room for more user interface features.

Figure 5. The GUI version called GUI

without

, without

augmented landmarks. In the image the user is to take a

bearing towards the shore.

2.1.4 Two variants of the Graphical User Interface

There are two variants of the GUI, GUI

without

and

GUI

with

, each variant tested by half of the user study

group. Figure 5 shows the variant without augmented

landmarks called GUI

without

, where the user shall try to

match objects between the upper 360 image with

objects in the lower 3D environment, without any

augmented landmarks that guide the user. In the

figure, the user is pointing towards the shore, which

is also found below in the 3D environment. By first

shooting the laser towards the specific point of the

shore in the 3D environment and then towards the

360 image, the application knows the ship is located

in the opposite direction from the shore’s specific

position. This information is sent to the TAN

application, which adjust the PF’s position estimation

accordingly.

This benefit of the GUI

without

’s design is that the

user is free to use all landmarks that can be found.

The disadvantage is that it is quite difficult to point

the laser pointer to the exact location in the vertical

direction. If pointing a little bit over the intended

direction, the user is pointing towards a position

further away, which will result in the wrong position

estimation. Another disadvantage with GUI

without

that the users must be more creative and find the

landmarks themselves.

Figure 6. The GUI version called GUIwith, with augmented

landmarks. (The user is overlaid over the image.)

Figure 6 presents the other variant of the GUI with

augmented landmarks, called GUI

with

. The difference

is that GUI

with

shows proposed landmarks as large

pink markers, often with arrows, that turn blue when

the user point towards them. The user in the image

first shoots the laser towards the arrow (that points

towards the right part of the house), and then directly

at the right part of the house in the 360 image. The

tool then knows the landmark’s exact position, in

793

contrast to GUI

without

, where the user could slightly

miss the target. A disadvantage is that the user is

limited to the usage of only the proposed landmarks.

Another screenshot from GUI

with

is shown in Figure 1.

2.2 Field Trial

Figure 7. The route of the field trial overlaid on the sea

chart. The boat went in the south-east direction.

We conducted a field trial in Västervik

archipelago in Sweden to validate our approach, see

Figure 7. We have used this field trial to collect data,

which we have used for simulations and

teleoperation tests. By using simulations for the user

study instead of running the user study on the

realworld ship, we have had the exact same scenario

for all participants, making the results comparable.

The boat used in the field trial is of type CB90, see

Figure 8. It has been complemented with additional

sensors to support the Swedish Universities via the

WASP program [24] with a research platform for

developing autonomous ships. We believe that the

sensors onboard, see Table I, correspond to a sensor

suite of a typical future affordable autonomous ship.

We collected data from the digital compass onboard,

but the ship was not equipped with a speed log.

Instead, a virtual speed log was created using data

from the GPS and adding an error of 0.2 knots (i.e.,

NM/h) to simulate a worst-case scenario. In the last

step, to study the robustness of the algorithm’s ability

to navigate, we added a drift of a constant 0.5 knots to

mimic the drift from the wind and current that can

not be detected by the compass and speed log. The

drift speed can, in general, be estimated with quite a

good accuracy, and these 0.5 knots should be seen as

the error between the estimated drift and the correct

drift. If we can show that the TAN application can

manage an inaccuracy of the drift speed of 0.5 knots,

we believe the system is very robust. As the particle

filter is estimating the position and not velocity nor

the drift speed, the algorithm’s results are not helped

by a constant drift. The boat traveled a 9.2NM (17km)

long route in 54 min, but to make the user-study more

manageable, we only used the first 20 minutes for the

study.

Figure 8. The boat of type CB90 used in the field trial.

Table 1. Sensors used during field trial

_______________________________________________

Sensor Description

_______________________________________________

Compass* Heading (Accuracy 0.5) - 1Hz

Speed Log* Speed Through Water (STW) (Accuracy

1% + 0.1knots) - 1Hz

Echosounder Depth from surface to sea bed (Accuracy

0.1m) - 1Hz

Magnetometer Magnetic Intensity measured as a vector –

100Hz

360 camera Provides visual image of the horizon

around most of the ship. Can alternatively

be multiple cameras. Images from 6

cameras were compiled into an image

with a resolution of 16384x8192 - 15Hz

_______________________________________________

* The digital compass and speed log could be exchanged to

an INS.

More information regarding the data collected

from the field trial can be found in our earlier work

[9], where we used the data to evaluate the

performance of the TAN application when using

various fusion methods.

We conducted the whole study in an office by

using the collected data. No ship was teleoperated for

real, but the teleoperation interfaces were restricted to

accommodate the low throughput connection.

2.3 C. User Study

We evaluated our implementations with a user study

of 16 participants, recruited mainly from Lund

University and the shipyard Saab Kockums AB, in 20-

minute long trial sessions with the task and scenarios

described below. We recorded what the users saw in

VR, as well as the TAN application GUI and its

performance data. We informed the participants of

the possibility of withdrawing at any time, and they

agreed upon the use of screen recordings and other

data for research purposes. Lund University ethics

council did not require reviews of this kind of study

since no personal data was studied. Four of the user

study videos can be seen on YouTube

(https://youtu.be/HwnIPuX-Azg, https://youtu.be/zu4

0PEsk5cQ, https://youtu.be/PCkAQhyAC6Q, and http

s://youtu.be/HTm2GEZsxh0). The videos show both

the teleoperation GUI and the TAN application GUI

for two of the users. One of the users used GUI

without

(without augmented landmarks), and the other used

GUI

with

(with augmented landmarks).

After an introduction phase based on written

instructions and a quick oral summary, the user used

the GUI in VR to remotely supervise the ship. The

main task was to point towards the same objects in

both the virtual 3D environment and the 360

panoramic image, resulting in a bearing to a

landmark. The teleoperation tool sent these bearings

794

to the TAN application, which updated and increased

the position’s accuracy. Half of the subjects were

randomly assigned using the GUI

with

, and the other

half were assigned the GUI

without

where they had to

find the landmarks by themselves. Our main

questions to evaluate in the study were:

− Do the bearings from the operators increase the

position accuracy for the TAN application, despite

the low quality of the 360 images?

− Do users experience they gain trust in the system’s

ability to navigate?

− Did users gain or lose performance when

augmenting specific landmarks that the operators

were to take bearing measurements from, instead

of letting the operators freely pick landmarks that

they thought would be good?

We compared the mean position error for each

participant’s recorded data from the user study

(objective results).

After the experiments, the participants evaluated

the GUI subjectively by answering the following four

questions on a scale of 1-10 (1 was best on the first,

and ten was best on question two to four):

1 It was difficult to handle the tool.

2 I experienced that I had a good overview of the

situation.

3 If the tool further evolves, I believe that a real ship

can be teleoperated using this technique.

4 If I practice 100 hours, my ability to use the tool

would enhance further.

3 RESULTS

We have summarized the collected data from the user

experiments in the objective and subjective results

below, followed by some observations. We have

interpreted the results in Section IV.

3.1 Objective Results

It is possible to estimate a ship’s position by dead

reckoning (DR) the position by using the compass

and speed log. The problem with this method is that

the error increases with time, as each measurement is

based on the previous measurement, leading to a

position error being accumulated over time. The TAN

application uses a PF to compare the bottom depth

and magnetic intensity with available maps to

estimate the position more accurately. With this

approach, the position error is not supposed to

increase with time but, instead, holds its position

relatively close to the correct position. We have

Kalman filtered (KF) the mean of the PF’s particle

cloud. The KF provides a smoother and more

accurate position estimation compared to only using

the mean of the PF’s cloud as a position estimation.

By using bearings to landmarks, it is possible to reset

the DR or PF’s position estimation in the bearing

direction by moving the particles or the DR

estimation to the closest point in the bearing’s

direction, see Figure 3 or any of the YouTube videos

for an example.

Figure 9. Position accuracy when not receiving any

bearings to landmarks at all. In this particular example, the

PF performance is worse than DR the first 15 minutes. This

can happen as the PF use the DR as basis for its calculation,

spreads the particles randomly, and then corrects the

particle positions by comparing the maps with sensor

measurements. In this particular situation, the maps

matches the measurements quite well for the first 15min,

despite the position being 150m off. After 15min, the

bottom depth do not match any more, which results in an

adjustment of the particle cloud’s mean position.

In the following, we present graphs and statistical

analysis of the position error performance. The mean

values are summarized in Table II. As we based the

user study on a recording from a field trial, the DR

without any bearing updates is the same for all user

tests.

Table 2. Mean position error without bearings or with

bearings from GUI

without

or GUI

with

_______________________________________________

Algorithm Without Bearings from Bearings from

bearings GUI

without

GUI

with

_______________________________________________

DR 111.6m 71.8m 57.6m

PF 123.5m 51.4m 42.8m

PF with KF 109.1m 36.5m 34.9m

_______________________________________________

Figure 9 shows how the position error varies when

not using any bearings to landmarks at all. As seen in

the plot, the DR position error (red line) increased

with time to around 200m after 20min. The purple

line, showing the position error when using PF, peaks

after 13min on about 200m. The accuracy is enhanced

by Kalman filtering the PF (blue line), which peaks on

about 175m. The mean error for DR was 111.6m, and

the mean error for PF with KF was 109.1m. We can

now use the value from PF with KF and see how the

performance increases when the application receives

bearings to landmarks.

Figure 10 shows a graph constructed from the

eight GUI

without

user trials. The graph presents the

averaged position error over the 20-minute test. The

DR position error increases with time (red line), but

as the user takes bearings, the DR error is reset in the

direction from the bearing (yellow line). As shown by

the blue line, the KF corrected PF’s mean error peaks

at about 65m after 7min. The mean error of the

bearing-updated DR was 71.8m (yellow line), and the

bearing-updated PF with KF was 36.5m (blue line).

795

Figure 10. Position accuracy averaged from eight user trials

using GUI

without

, compared to DR when not using any

bearings at all (red line).

Figure 11. Position accuracy averaged from eight user trials

using GUI

with

Figure 11 shows a graph constructed from the

eight GUI

with

user trials. As shown by the blue line, the

bearing-updated PF with KF mean error is relatively

stable around 35- 40m. The mean error of the bearing-

updated DR was 57.6m (yellow line). The bearing-

updated PF with KF was 34.9m (blue line), and

peaked after about 17min on about 60m.

We found that the tests with both the GUI

without

and

GUI

with

significantly improved position accuracy by

updating the TAN application with the bearings

towards the landmarks. It was significantly better to

use the bearing-updated PF with KF than just the

bearing-updated DR, but also significantly better with

bearing-updated DR compared to only DR (without

bearings).

Figure 12 summarizes the objective results, along

with the p-values showing the significant difference,

computed in a series of one-tailed t-tests. The mean

values are presented in Table II. Even though it was

quite a small user study, power tests (alfa=0.05,

power>0.80) have shown that there were enough

participants to support the significant results.

However, there was no significant difference between

the two user groups.

3.2 Subjective Results

Table 3. Subjective results summary

_______________________________________________

Question GUI

without

GUI

with

_______________________________________________

Difficult to Manage* 3.3 2.9

User Overview 6.3 6.9

Applicability for real-world usage 8.5 8.3

Gain from Training 8.4 8.9

_______________________________________________

*Lower score is better.

For the user evaluation’s subjective results, the

users using GUI

with

gave better scores to all four

questions except the question regarding the

applicability for real-world usage. In general, though,

the results were quite similar, and there have not

been any significant differences. The scores, with a

scale of 1-10 (1 was best on the first, and ten was best

on question two to four), are presented in Table III.

The users answered that the tool was not challenging

to use (3.3 and 2.9) (lower score is better). They also

had a good overview of the situation (6.3 and 6.9).

Even more importantly, they believed a real-world

ship would be possible to teleoperate in this way if

the tool was further developed (8.5 and 8.3). They

thought they would be even better handling the

teleoperation tool after 100h of usage (8.4 and 8.9).

Figure 13 summarizes the subjective results.

Figure 12. The objective results summarizing the position accuracy when using the different methods. (x) and (*) denote

comparisons with p ¡ 0.01 and p ¡ 0.001 respectively. Because we based the user study on a recording from a field trial, DR

(to the left) is the same for all tests.

796

Figure 13. Subjective results from the questionnaire. Users answered the following questions: (a) It was difficult to handle

the tool. (Lower score is better.) (b) I experienced that I had a good overview of the situation. (User Overview Experience) (c)

If the tool further evolves, I believe that a real ship can be teleoperated using this technique. (Applicability for Real-World

Usage) (d) If I practice 100 hours, my ability to use the tool would enhance further. (Performance Boost after Training)

We also asked the users to elaborate on good and

bad aspects about the tool and how it felt using it.

Starting with the point that users wanted to be

enhanced, most users (11 of 16) wanted either better

resolution, higher frame-rate, or better lightning of

the 360 image. Two persons lacked support for

glasses, as the head-mounted display HTCVive is of

the older type where glasses do not fit. One

participant suggested adding support for taking a

bearing during the turning of the ship, which we have

not implemented yet. When the shore was close to the

vessel, the 3D environment could cover the 360

image, which one person pointed out to aggravate the

usability. There were also suggestions to enhance the

3D visualizations of buildings. Two persons found it

a fun experience, while one person got bored after a

while.

We asked the users about their VR experience,

where the value 10 meant they were very

experienced, and 1 meant they had no experience,

resulting in the mean value of only 3.1. Still, 12 of the

participants wrote that it was easy to use, easy to

understand, or intuitive. Some had a problem

understanding the tool during the first minutes but

then said that they quickly learned. Three persons

said they had either a good overview of the situation

or that it was easy to orient, and another said that it

was easy to find out if the estimated position did not

match the 360 image. One person reflected that he

did not get any motion sickness, which surprised

him, as he usually gets motion sickness from VR.

3.3 Observations

By observing the user’s behavior during the study, it

became clear that the users of GUI

without

, in general,

had more difficulties at the beginning of the study, as

they needed to orient themselves and learn quickly

what type of landmarks would be appropriate to

pick. Some of them got stressed, especially initially,

and some participants took very few bearings during

the first 5 minutes. After this initial period, these

users seemed to have learned the teleoperation tool

and were very creative in finding new landmarks to

improve the position accuracy.

The users of GUI

with

, in general, figured out quite

early what to do. Most of them acted quite skilled and

took many bearings towards these fixed landmarks.

We believe this task was not as challenging as with

GUI

without

, and some of the participants looked a bit

bored after a while. Being bored indicate low

cognitive load, leaving room for conducting other

tasks simultaneously. For not having even higher

positioning performance, we consider a large reason

to be the difficulty of hitting the GUI

without

’s landmark

accurately in the vertical direction with the laser

pointer. We discovered from the saved video files

from the user study that the laser pointer has many

times pointed at an object behind the intended object,

797

resulting in lower performance. To get better

accuracy from using the GUI

with

, it should have been

beneficial with more augmented landmarks, as the

users could have handled many more.

4 DISCUSSION

What we mainly wanted to see in this study was if the

users could get good situational awareness and feel

that they were positioned in the correct location by

being able to easily compare the 360 image with the

3D environment. The questionnaire results, together

with what the users wrote they experienced, have

confirmed both these hypotheses. This is important,

as it is difficult for a remote user to know if the

position is accurate in a typical teleoperation system.

Furthermore, the users have been given a good

overview of the situation by using the 360 image.

Relatively small buoys more than 500m away have

been easily discovered.

We also wanted to confirm that the TAN

application could gain in performance using bearings

that the user took remotely. We did not know this

beforehand, as we knew that the images would have

relatively low quality, given the communication link

with poor throughput.

We also wanted to know if there were any

considerable differences between the two GUI types,

which the user study did not imply. Still, the different

GUI versions have given us some insights, as there

were different advantages and disadvantages of the

versions.

To enhance the implementation further, we have

learned that:

− The 360 image already has a low quality; hence it

is helpful to increase the visibility as much as

possible by, e.g., increasing the brightness.

− By using an augmented landmark to point

towards, it is possible to get a more accurate

position of the landmark, which is beneficial for

the positioning system.

− Users are creative and can keep track of many

objects. Do not limit the number of available

augmented landmarks too much.

− The 360 image should not be covered by the 3D

environment, even when the ship is very close to

shore.

Some users wanted a higher resolution and frame-

rate for the 360 image, which can be achieved when

there is a better communication connection available

with higher throughput. In this study, the throughput

was very limited, though, as we wanted to see if it

worked in the worst-case scenario.

The study provides knowledge about multiple

aspects about how to create a teleoperation tool for an

autonomous vessel, but the user-study has not

intended to evaluate a complete system design. We

believe more research is needed for this. We still do

not know if VR is a good solution for multiple hours

of operations, and we believe a final design for expert

users should be designed in a different way,

optimized for the intended usage and scenario.

5 CONCLUSION AND FUTURE WORK

We have developed and tested a GUI for the

teleoperation of an affordable Autonomous Surface

Vehicle using a low throughput connection. Our

findings show that users have had a good overview

despite the low-quality images. The users have

experienced the position as correctly estimated by

easily matching the 3D environment with the 360

image. When it did not match, they quickly have

reacted and tried to solve the problem by updating

the positioning system with new bearings. We can

conclude that the positioning system has increased its

accuracy by using these bearings, despite the low-

quality link connection.

Together with the results from our previous work

[10], [11], which focused on features to provide good

situational awareness and safety, while maintaining a

low cognitive load, we now believe we have all

functions needed to combine all building blocks into

a more comprehensive GUI with more complexity,

tailored for trained expert users. Building upon this,

we aim to conduct a new user study with expert users

teleoperating the ship while having safety drivers on

board to meet current regulations.

ACKNOWLEDGMENT

This work was partially supported by the Wallenberg AI,

Autonomous Systems and Software Program (WASP) [24]

funded by Knut and Alice Wallenberg Foundation. Saab

Kockums AB [23] provided the ship used for the field trial.

REFERENCES

[1] Kongsberg. (2018) Rolls-royce and finferries

demonstrate world’s first fully autonomous ferry. Last

accessed 10 October 2020. [Online]. Available:

https://www.rolls-royce.com/media/press-

releases/2018/03- 12-2018-rr-and-finferries-demonstrate-

worlds-first-fully-autonomousferry. aspx

[2] Kongsberg. (2018) Technology for the ferries of the

future. Last accessed 10 October 2020. [Online].

Available: https://www.kongsberg.com/maritime/about-

us/news-andmedia/ news-archive/2018/technology-for-

the-ferries-of-the-future/

[3] (2018) Remote-controlled and autonomous ships in the

maritime industry. Last accessed 9 September 2020.

[Online]. Available:

https://www.dnvgl.com/maritime/publications/remote-

controlledautonomous- ships-paper-download.html

[4] O. Levander, “Autonomous ships on the high seas,”

IEEE Spectrum, vol. 54, no. 2, pp. 26–31, 2017.

[5] C. Lundberg, H. I. Christensen, and A. Hedstrom, “The

use of robots in harsh and unstructured field

applications,” in ROMAN 2005. IEEE International

Workshop on Robot and Human Interactive

Communication, 2005. IEEE, 2005, pp. 143–150.

[6] Y. Zheng, M. J. Brudnak, P. Jayakumar, J. L. Stein, and

T. Ersal, “Evaluation of a predictor-based framework in

high-speed teleoperated military UGVs,” IEEE

Transactions on Human-Machine Systems, 2020.

[7] R. Luz, J. Corujeira, L. Grisoni, F. Giraud, J. L. Silva, and

R. Ventura, “On the use of haptic tablets for UGV

teleoperation in unstructured environments: System

design and evaluation,” IEEE Access, vol. 7, pp. 95 431–

95 442, 2019.

798

[8] R. Murphy, “Disaster Robotics. Intelligent robotics and

autonomous agents series,” The MIT Press, 2014.

[9] M. Lager, E. A. Topp, and J. Malec, “Robust terrain-

aided navigation through sensor fusion,” in 2020 IEEE

23rd International Conference on Information Fusion

(FUSION). IEEE, 2020, pp. 1–8.

[10] M. Lager, E. A. Topp, and J. Malec, “Remote operation

of unmanned surface vessel through virtual reality-a

low cognitive load approach,” in Proceedings of the 1st

International Workshop on Virtual, Augmented, and

Mixed Reality for HRI (VAM-HRI), 2018.

[11] M. Lager and E. A. Topp, “Remote supervision of an

autonomous surface vehicle using virtual reality,”

IFAC-PapersOnLine, vol. 52, no. 8, pp. 387–392, 2019.

[12] S. Neumeier, N. Gay, C. Dannheim, and C. Facchi, “On

the way to autonomous vehicles teleoperated driving,”

in AmE 2018-Automotive meets Electronics; 9th GMM-

Symposium. VDE, 2018, pp. 1–6.

[13] S. Neumeier, P. Wintersberger, A.-K. Frison, A. Becher,

C. Facchi, and A. Riener, “Teleoperation: The holy grail

to solve problems of automated driving? Sure, but

latency matters,” in Proceedings of the 11th

International Conference on Automotive User Interfaces

and Interactive Vehicular Applications, 2019, pp. 186–

197.

[14] S. Lu, M. Y. Zhang, T. Ersal, and X. J. Yang, “Workload

management in teleoperation of unmanned ground

vehicles: Effects of a delay compensation aid on human

operators’ workload and teleoperation performance,”

International Journal of Human–Computer Interaction,

vol. 35, no. 19, pp. 1820–1830, 2019.

[15] A. Hosseini, F. Richthammer, and M. Lienkamp,

“Predictive haptic feedback for safe lateral control of

teleoperated road vehicles in urban areas,” in 2016 IEEE

83rd Vehicular Technology Conference (VTC Spring).

IEEE, 2016, pp. 1–7.

[16] F. Chucholowski, M. Sauer, and M. Lienkamp,

“Evaluation of display methods for teleoperation of

road vehicles,” Journal of Unmanned System

Technology, vol. 3, no. 3, pp. 80–85, 2016.

[17] T. Williams, N. Tran, J. Rands, and N. T. Dantam,

“Augmented, mixed, and virtual reality enabling of

robot deixis,” in the International Conference on

Virtual, Augmented and Mixed Reality, 2018.

[18] A. Hosseini and M. Lienkamp, “Enhancing

telepresence during the teleoperation of road vehicles

using hmd-based mixed reality,” in Intelligent Vehicles

Symposium (IV), 2016 IEEE. IEEE, 2016, pp. 1366–1373.

[19] X. Shen, Z. J. Chong, S. Pendleton, G. M. J. Fu, B. Qin,

E. Frazzoli, and M. H. Ang, “Teleoperation of on-road

vehicles via immersive telepresence using off-the-shelf

components,” in Intelligent Autonomous Systems 13.

Springer, 2016, pp. 1419–1433.

[20] T. Porathe, “User-centered map design,” in Usability

Professionals’ Association Conference, 2007.

[21] S. Witt, “Technologies used for navigation,” Novel

Interaction Techniques for Oceangoings, University of

Passau, 2017.

[22] (2018) Unity 3d. Last accessed 29 September 2020.

[Online]. Available: https://unity.com/

[23] (2020) Saab Kockums AB. Last accessed 29 September

2020. [Online]. Available:

https://www.saab.com/products/naval

[24] (2020) WASP. Last accessed 29 January 2020. [Online].

Available: http://wasp-sweden.org/