511

1 INTRODUCTION

At the Norwegian University of Science and

technology (NTNU) in Trondheim a new centre for

research driven innovation has recently received 8

year-long funding by the Norwegian Research

Council. The name of the centre is SFI AutoShip and

the research focus will be on autonomous ships with

four different use cases: a) deep sea bulk shipping, b)

short sea container shipping, c) ferries and d) offshore

support operations. The centre is divided into several

work packages focusing on problem areas within this

new technology: 1) automation and remote sensing, 2)

communication and digital infrastructure, 3) human

factors and remote control centers, 4) safety and

assurance, 4) sustainable operations and 6) innovation

and commercialization.

After a resent presentation of the scope of this

autonomous ship center a member of the audience

asked a question about how we would approach

navigation and ship handling in waters where local

knowledge of how conditions and weather will affect

a vessel are generally used? How will the

“automation” know about local peculiarities? For

instance, bathymetry which may create wave patterns

during different weather conditions?

1.1 Knowledge repositories in the past

The question is very much to the point and reminded

me of a story my grandfather told me in my

childhood. Along the Swedish west coast where I

grew up, there are some places where the archipelago

opens, and the inner fairway is exposed to the rage of

the open sea. Islandsberg is one of these places and I

was always on my toes when we passed there in my

grandfather’s small cutter, even if the weather never

was really bad. Many times, he told me of his father,

Human-Automation Interaction for Autonomous Ships:

Decision Support for Remote Operators

T. Porathe

Norwegian University of Science and Technology, Trondheim, Norway

ABSTRACT: At NTNU in Norway an 8-year research project has been established to (among other things)

research the interaction between humans and unmanned, autonomous ships. The human will become even

more important when ship operator will be located remotely in shore control centers ashore. This concept paper

will take a closer look on remote decision-making by operators monitoring several ships. How can interface

design help them to get quickly into-the-loop when something unexpected suddenly happens? I will in this

paper suggest keeping a copy of the AI expert-system controlling the ship, updated and running in parallel in

the control center to keep the operator’s situation awareness during short communication glitches. Also, to

design a “quickly-getting-into-the-loop-display” which automatically will appear in an alarm situation,

allowing the operator just-in-time and simple-to-understand information. I will also stress the important of the

concept automation transparency.

http://www.transnav.eu

the

International Journal

on Marine Navigation

and Safety of Sea

Transportation

Volume 15

Number 3

September 2021

DOI: 10.12716/1001.15.03.03

512

my great grandfather, who had been captain on a

coastal express steamer trafficking the area in the

1930’s. He said that passing Islandsberg in a westerly

storm took some nerves, because of the fearsome

chaos of braking waves rolling in over shoals from the

open sea and reflecting back from the steep cliff. But,

if you had these nerves you could find smooth water

some 40-50 meters out from the cliff where the waves

cancelled each other in a valley of “dead seas”.

Where do this knowledge go when those who

remember are gone? Some of it is rightly lost because

new, larger ships are not affected by weather and

wind in the same way, and, as in this case, goods and

passengers are not transported along the coast in the

same way anymore. But some knowledge is collected

in “expert-systems”. It could be old analogue expert-

systems like for instance sailing directions. The sailing

direction is a textual description of a voyage. It

mentioned landmarks, courses and warned for

specific local weather and sea conditions. So, for

instance the British Admiralty sailing direction for the

Norwegian coast puts it this way for an area not fare

from Trondheim:

“Area 11, Hustadvika (63°00.00’N 7°00.00’E) is a

notoriously dangerous area; the coast is completely

exposed to the weather and extensive shoals lie

offshore. Strong winds from SW to NW raise a large

steep swell with hollow breaking seas, especially

during the out-going tidal stream. These conditions

are likely to be particularly severe in the area of

Budadjupet between Bjørnsund (62°53.75’N 6°48.96’E)

and Kolbeinsflua, 5 miles NNE. Breaking surf is

reported to occur throughout the whole area.” [11]

The sailing direction was the major medium for

communicating navigational information to the

mariner until the end of the eighteenth century, when

its function was partly overtaken by the nautical chart.

A nautical chart is a container for geographic

knowledge. A nautical chart “represent the

accumulation of more observations than any one

person could make in a lifetime. It is an artifact that

embodies generations of experience and

measurements” [6].

The experience of past generations of seafarers are

also promulgated in maritime academies where active

or ex seagoing officers teach young apprentices based

on their experience. Assuming future autonomous,

unmanned ships are successful, where will then the

experience come from? At the millennium change

1900, when the era of sail transitioned to the era of

steam, cadets still had to have experience from

commercial oceangoing sail ships to get their master’s

certificates. As the sailing ships were getting scarce

this became a business opportunity for the ship owner

Gustav Eriksson from Mariehamn in Finland which

manned the last fleet of three and four masted iron

windjammers on the wheat rout between Australia

and the UK with paying apprentices (as well as a

small safety crew). Maybe we will see the same again,

if the number of manned merchant ships start to

decrease? Or maybe the question is if we can gater

previous generations of local and seagoing experience

in a computerized “expert-system”?

1.2 The expert-system

The question of using expert knowledge collected in

huge databases came in focus with the development

of computers in the 1960’s and 70’s. At Stanford the

Heuristic Programming Project started to investigate

if expert-systems could be useful in analyzing

chemicals and for medical diagnoses. [9]

Using expert-systems as decision-analyst and

decision-maker is part of the Artificial Intelligence

domain. I will in this paper use the term “expert-

system” instead of the more general “AI” in order to

emphasize its knowledge-based repository of

maritime experience, which I think will become a

commodity in short supply in the Remote Operation

Centers (ROC) of future autonomous shipping.

2 THE REMOTE OPERATIONS CENTRE

In the MUNIN project I too, where I took part in

drawing up the general framework for what was then

called the Shore Operations Centre (SOC). Remote

operator were to actively monitor up to six ships

during relaxed conditions from the safety of a

computer console ashore. Sensors would give them

the necessary information about the state of the ships

out there on the oceans, about winds and waves,

engine performance and radar should detect traffic in

the surroundings. Will it mean that remote operators

no longer need the experience of how a hull performs

in different wave patterns? Risks of broaching in

following seas, or slamming if you head too fast into

breaking waves? Can this experience be contained in

computerized expert-systems? Can it be derived from

experience collected through machine learning and

harbored in “Artificial Intelligence”? Well, we don’t

have that answer to that yet. Still, we got to continue

designing as if these problems can and will be solved.

Some of that has already been done. The Electronic

Chart and Information Display System (ECDIS) with

the digitalized Electronic Navigational Charts (ENC)

are an example. Sailing directions are also digitalized,

and attempts to make them smarter is underway, e.g.

the embryo of a new interactive version of The

Norwegian Pilot [7]. One wants to foresee future

expert-system that can produce contextualized

knowledge and experience form generations of

mariners in an easy to use manner when the situation

demands decisions to be made.

Lisanne Bainbridge in 1983 talked about the Ironies

of Automation. We automatize what we can (the

simpler tasks), but when it gets really complicated

automation cannot cope and we are up for a surprise

when it suddenly hands us the full responsibility. I

am thinking about an example from the aviation

domain: the Air France 477 accident in 2009.

2.1 Effects of automation

The night of the 1st of June 2009 Air France fight 447

from Rio to Paris, disappeared in the middle of the

Atlantic Ocean. And with it, 228 passengers [2].

513

The plane was in mid air over the South Atlantic

approaching the Inter-Tropical Convergence Zone, the

area where air masses coming from the different

hemispheres converge at the humid equatorial

latitude, with electrical storms as a result. Suddenly

the automatic flight management system lost speed

input from the triple redundant pitot tubes (which

were all clogged by ice) and handed, what had up to

then been a fully automatic airplane, into the hands of

the relatively inexperienced junior pilot flying.

The accident investigation tells a horrifying story

of mismatch between the automation interface and the

pilots trying to make sense of what they saw on the

screens. The highly automated flight system, which

interfaced the human and the machine through

sophisticated computer programs that in this case was

difficult to understand and handle correctly. The

automation can prevent human mistakes when

everything works as planned by the engineers, but

became incomprehensible for the same operators

when computers did not receive proper inputs and

went blind [4].

After the accident, accusations were made

regarding loss of basic skills of manual flight (e.g.

[10]). Was it that liner pilots had lost their skills and

their capability to manually fly a plane, because of the

use of autopilot and of the overwhelming technology?

Bainbridge [1] remarked that skills deteriorate when

not used and a formerly experienced operator

monitoring an automated process may well have

turned into an inexperienced one. Then when manual

take-over is needed there is likely something wrong,

so that unusual actions will be needed to control it,

and one can argue that the operator needs to be more

rather than less skilled.

When designing a new workplace for marine

watch officers moving into a shore-based Remote

Operation Centre, one might envision such a loss of

skills. How long will it take for an experienced deck

officer to become unexperienced? Having forgotten

the inertia and dynamics of maneuvering a big ship in

heavy seas or birthing in an intricate dockland? Can

this be avoided? Can we replace individual experience

with expert-systems? The challenge is enormous. Let

us do some design sketching of such systems to

support decision-making in a ROC.

I will sketch two concept solutions: how to keep

the operator in the loop during communication

glitches and how to quickly bring the operator into

the loop if he or she has been “absent” and there is a

sudden alarm situation.

3 THE DECISION SUPPORT SYSTEM AND ITS

“DIGITAL TWIN”

Let us assume that we can collect this nautical

experience in an expert-system capable of making

decisions and transparently showing the monitoring

operator its current status of knowledge (the input

from sensors on the remote ship) and what it is

planning to do (its intentions, plan A, B and C…).

Let us assume that all necessary data from the ship

is communicated to the ROC through some kind of

communication system (satellite, 5G, Maritime

Broadband, etc.). Our experience tells us that we will

have communication glitches or outages from time to

time. In the MUNIN project we said that a MASS that

has lost communication with its control centre should

stop in a “fail-to-safe” mode. Now, stopping in the

middle of whatever situation the vessel might be in

might not always be the smartest action. Smarter

would be to go to a “minimum risk conditions” [5],

meaning that the ship will do what is best given the

current circumstances: in the middle of the ocean,

with no other ships within 30 nautical miles, the ship

might just as well carry on its course for some time

waiting for communication to come back. But in a

densely trafficked shipping lane in the English

Channel, the ship could check for oncoming vessels

on its starboard quarter and then proceed out of the

Traffic Separation Scheme and stop and hover on its

DP (or anchor) while sending relevant pre-recorded

messages on VTH channel 16 and hoisting relevant

signals. And most important of all is that we realize

that, having lost communication, decisions must made

by the expert-system on the ship alone. The

automation will have the final say.

What if we see it from the remote operator point of

view: suddenly the communication link with the

vessel is lost. Camera and radar images go blanc,

own-ships-symbol disappear from the ECDIS. The

operator is now in the back. But for some time on he

or she could extrapolate ships motions into the future.

This is where the “digital twin” comes into the

picture.

3.1 The expert-system digital twin

As mentioned above the expert-system on the

autonomous ship is the one deciding on how to move

into minimum risk condition when the vessel has lost

communication with the ROC. The expert-system

onboard is normally constantly updated by sensors

onboard, as well as with information from the ROC

and from others (e.g. AIS messages from surrounding

ships, information from traffic centers like a VTS, etc.).

If an exact copy of the expert-system onboard are also

present in the ROC and is simultaneously updated

with the same information as the one onboard, the

system ashore should make the same decisions as the

one onboard. Now, if communication is lost and the

operator loses his eyes and ears of what is going on

onboard, the digital twin in the control room will for

some time continue to extrapolate the situation into

the future. It might even be able to catch AIS

information through other vessels and coastal radio.

The operator could then with probability see a

simulated reality, where the digital twin demonstrates

the same actions for some period of time in the ROC.

Such a simulation will soon become obsolete, but a

might keep the ROC operator in the loop over a short

communication glitch.

Another problem is when an operator that has

been attending other vessels suddenly is summoned

by an unexpected alarm from one of his ships he has

not been attending for some time.

514

4 QUICKLY GETTING INTO THE LOOP

Endsley [3] talks about Out-of-the-loop-syndrome. We

know that humans are bad at monitoring well

function automation. Then the human mind drifts off

to more entertaining or rewarding tasks. And when

then suddenly the alarm goes off, it will take some

time to “get into the loop”. In some situations (like in

the AF477 accident above) this time may be very

short. And this will happen in the ROC as well. When

it happens, the operator must quickly search for the

information needed for the particular emergency the

alarm is about. For such situations a Quickly -getting-

into-the-loop display is needed. This “QGILD” is a

screen or window that opens, following an alarm

activated by the expert-system requesting help from

the human operator in the ROC.

In the QGILD the expert-system must collect all

relevant information and display it in an uncluttered

and pedagogical way so that the operator is will be

informed in the shortest timespan possible. The screen

must also display a timer counting down to when the

operator needs to make the decision, else the expert

system will take a spelled-out action.

This is of course easier said than done. The expert -

system must understand the nature of the problem at

each instance. It must present enough, but not too

much information (information overload) to the

operator. It should display possible actions in order of

preference and allow the operator an easy way to

select among suggested actions. Can this be done?

Let us elaborate a little on this.

4.1 The QGILD

The QGILD must fulfill the following three objectives:

1. It must give a tactical update, an at-a-glance

understanding of the present situation. Showing

all necessary information - but not more!

2. It should offer automation transparency, giving the

operator an at-a-glance understanding of how the

expert-system sees the situation, and its intentions

for solving the problem.

3. Finally, it must supply tools for intervention,

simple and intuitive ways for the operator to

intervene and override the expert-system.

4.1.1 Tactical update

The QGUILD should provide all necessary

information for tactical decision-making, i.e. decision-

making in the short time span 1-10 minutes into the

future. The operator should not be asked to make split

second decisions – then the expert-system must take

the full responsibility. Even so, this will be a

challenging task as the QGILD must timely present

the right information in an uncluttered and simple-to-

understand way. Information overload must be

avoided. This means that the expert-system must have

understood that there is a problem, and it must have

diagnosed the problem in the correct way in order to

give the right information. If it has not realized that

there is a problem, there will be no warning and no

QGILD will be brought up. If the diagnose is wrong

the information displayed will not be relevant,

leading to lost time.

4.1.2 Automation transparency

This objective should give the operator a quick

peek into the expert-systems mind, an at-a-glance

understanding of how the expert-system sees the

situation, and its intentions for solving it, thus being

able to judge if the system has got it right. Important

to note here is that, to avoid losing time due to

communication outages or an operator that is slow in

responding, the expert-system could already be

underway applying its preferred decision. Not

waiting to the human input, but instead offer the

human operator to overrule. This way we avoid that

the expert-system hands the vessel over to the human

who might not be in the loop - as did the flight

management system on AF447, mentioned above.

Important for the technical developers of the expert-

system is to know that the system always has a

responsibility to find a solution to any problem! It can

ask the human for advice, and offer tools for

intervention to override the system, but the system

must always have a plan. (I acknowledge that this will

never be fully possible, because of the “unknown

unknowns”, the black swans, that will always be

there. But it should be the goal to strive for.)

4.1.3 Tools for interventions

Finally, the QGILD must offer simple-to-use tools

for intervention to override the expert-systems

actions. This can in a simple case be a list of possible

actions, with the one the expert-system has chosen

highlighted but allowing the operator to select

another action. In Figure 1 is a simple example from a

suggested control interface for the AutoFerry in

Trondheim [8].

Figure 1. Example of automation transparency from

a suggested control board of a small autonomous and

unmanned urban passenger ferry. For details, see the text.

In the “Routine procedures” column to the left, we

can peek into the expert-systems “mind” and see the

different steps the automation is conducting and the

time that is estimated for each step (they can also be

opened to see sub-procedures). The operator can

interact with each routine by clicking. We see that the

ferry is underway and has 39 seconds left until the

docking procedures start.

515

To the right is the panel for “Emergency

procedures” where the operator can intervene and

override the automatic transit, e.g. by “stopping” (and

hoovering) or stopping (and drifting with the water

current) to assist a MOB (Man Over Board).

If another boat is approaches and a close quarters

situation is expected, the expert-system will switch

into “autonomous control” mode, meaning that it is

now acting and making decisions based on its own

“intelligence”, acquired experience and the rules of

the road. It is illustrated in Figure 2 by a trivial

example of a ship coming from starboard and thus

having right of way (COLREGS Rule 15). The expert-

systems decision to turn starboard and go behind the

other vessel is illustrated by the green solid line (and

some kind of certainty index of “98”). However, other

COLREG compliant actions are shown by green

dashed lines, and not-COLREG-compliant actions by

yellow and red lines. The operator can override the

expert-system by clicking on an action. (Or as a final

resort grab the joystick and press the “manual”

button.)

Figure 2. An example of automation transparency in a

COLREG situation. For details see the text.

5 SHOULD THE COMPUTER OR THE HUMAN

OPERATOR HAVE THE FINAL SAY?

A consequence of what I have stated above is that the

computer always has a plan that it is executing, but

the human operator is invited to override that plan.

Now, we know that a lot of accident in complex

transport systems are to some extent attributed to

what is called “human error”. And it is true that

humans make mistakes, become distracted, take short-

cuts, and even fall asleep when not intending to. This

is all part of the human condition. Especially for an

automated future, humans are very bad at monitoring

well function automation, as mentioned above.

Some of these “human error” attributed accidents

involve watch officers asleep, being drunk or simply

not monitoring what is going on. At the same time

“the system” – the GNSS position, the ECDIS, etc. –

could be very aware of that the ship is underway to

beach on an island or a shoal. Should our design then

allow the expert-system to prevent the human from

making such a serious mistake? Who has the final say,

the system or the human? The operator monitoring

the automation, and the automation monitoring the

human. How should the teamwork be designed? That

is a very important question to think about for the

design of decision support for Remote Control

Operators.

6 CONCLUCIONS

In this paper I have presented a few concepts and

ideas regarding decision-support and Human-

Machine Interface in the Remote Operation Centre of

autonomous ships. Due to pandemic restriction

during the passed year this is all desktop research and

no focus groups or interviews with end users have

been conducted. The concepts discussed are collection

of mariners experience in the expert-system (the

Artificial Intelligence), the prevalence of an updated

“digital twin” of the ships expert-system in the ROC

to allow as much as possible seamless situation

awareness during short communication glitches. And

finally the launching of a Quickly-getting-into-the-

loop display to help the operator achieve situation

awareness during alarm situations.

ACKNOWLEDGEMENTS

This research is conducted within the LOAS (Land-based

monitoring of autonomous ships) and the SFI AutoShip

projects, which are both funded by the Norwegian Research

Council, which is hereby gratefully acknowledged.

REFERENCES

1. Bainbridge, L.: Ironies of automation. Automatica. 19, 6,

775–779 (1983). https://doi.org/10.1016/0005-

1098(83)90046-8.

2. BEA: Bureau d’Enquêtes et d’Analyses pour la sécurité

de l’aviation civile 2012. Final Report on the accident on

1st June 2009 to the Airbus A330-203, registered F-

GZCP, operated by Air France, flight AF 447, Rio de

Janeiro – Paris.

3. Endsley, M.R.: Designing for Situation Awareness: An

Approach to User-Centered Design. CRC Press (2012).

4. Faccini, B.: Four Minutes, 23 Seconds: Flight AF447,

http://understandingaf447.com/extras/18-

4_minutes__23_seconds_EN.pdf.

5. Fjørtoft, K.E., Rødseth, Ø.J.: Using the Operational

Envelope to Make Autonomous Ships Safer. Presented at

the The 30th European Safety and Reliability Conference

, Venice, Italy (2020).

6. Hutchins, E.: Cognition in the Wild | The MIT Press. A

Bradford Book (1995).

7. Kartverket: Den norske los, https://kartverket.no/til-

sjos/nautiske-publikasjoner/den-norske-los, last accessed

2021/05/19.

8. Porathe, T.: Human-Automation Interaction for a small

autonomous urban ferry: a concept sketch. (In prep.)

(2021).

9. Russell, S., Norvig, P.: Artificial Intelligence: A Modern

Approach. Pearson (2016).

10. Shaw, A.: Pilot error to blame for Air France plane crash

that killed five Brits, https://www.mirror.co.uk/news/uk-

news/pilot-error-to-blame-for-air-france-144671, last

accessed 2021/05/19.

11. UK Hydrographic Office: , British Admiralty Sailing

Directions, Area 57B: Norway Pilot Vol 2B (2017).