International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 6
Number 4
December 2012
Decision-making processes, especially those
occurring in the transport, held responsibility for the
safety of people, equipment and the environment.
Such important decisions should be taken with a
minimum of uncertainty of the decision maker. This
uncertainty may be due to the existence of a number
of factors, such as: the level of training of decision-
makers, the lack of information regarding the
situation of the surrounding area and sometimes an
excess of information provided to decision-makers
simultaneously from multiple sources.
IT systems become more efficient over time
regarding constant increase of computing power
available to standard users. That allows developing
advanced systems which collect and analyze
relevant data to support decisions that minimize the
risk of collision.
These advanced systems may be used in support
decisions on the real ships maneuvering and ship
models and simulators used during the training of
future officers at training centers.
Currently still being developed computational
methods are meuroevolutionary methods which,
thanks to its efficiency are becoming widely used in
many fields of science and technology, such as:
automation and robotics systems, e.g. control of a
robot arm (Siebel and Sommer 2007);
designing and diagnostic systems, e.g. mobile
hardware acceleration (Larkin, Kinane and
O'Connor 2006), search hull damage (Kappatos,
Georgoulas, Stylios and Dermatas 2009),
processors design (Ratuszniak 2012), the
detection and evaluation of the risk of breast
cancer (Janghel, Tiwari, Kala and Shukla 2012)
control systems: for example, a helicopter flight
stabilization (De Nardi, Togelius, Holland and
Lucas 2006);
decision support systems, e.g. systems applied
artificial intelligence in computer games
(Kenneth, Bryant and Risto 2005).
A large number of positive results of the
implementation of neuroevolutionary methods
obtained in many areas of science encouraged the
author of this paper to undertake research and
Neuroevolutionary Ship Handling System in a
Windy Environment
M. Łącki
Gdynia Maritime University, Gdynia, Poland
ABSTRACT: This paper presents the advanced intelligent ship handling system able to simulate and
demonstrate learning behavior of artificial helmsman which controls model of ship in a windy environment of
restricted water area. Simulated helmsmen are treated as
individuals in population, which through
environmental sensing and evolutionary algorithms learns to perform given task efficiently. The task is: safe
navigation through heavy wind channels. Neuroevolutionary algorithms, which develop artificial neural
networks through evolutionary operations, have been applied in this system.
develop his own algorithms intended for use in
maritime transport (Łącki 2010b, a).
In this paper, the extension of the functionality of
navigational decision support systems is proposed.
This solution generates specifications of
maneuvering decisions (rudder angle and propeller
thrust) that maintain a safe ship trajectory computed
in the available water channel. In addition to rudder
angle and propeller thrust this system also includes
information about time of their execution. It is
possible in this system that all maneuvering
decisions may be calculated and presented in real
time for a given ship dynamics in the presence of
certain external disturbances.
1.1 Reinforcement Learning Algorithms
One of the main tasks in machine learning is to
create the advanced systems that can effectively find
a solution of given problem and improve it over
time. Reinforcement learning is a kind of machine
learning, in which an autonomous unit, called a
robot or agent, performs actions in a given
environment. Through interaction and the
observation of the environment (by input signals)
and performing an action he affects this environment
and receives an immediate score called
reinforcement or reward (Figure 1).
Figure 1. Interaction of helmsman with an environment.
The main task of the agent is to take such actions
to adjust the value R which is the sum of the partial
reinforcement as much as possible (1).
Such abilities are very important for simulating
helmsman behavior in ship maneuvering on
restricted waters.
For simpler layouts learning process can be
performed using classic approach, i.e. Temporal
Difference Reinforcement Learning (Tesauro 1995;
Kaelbling, Littman and Moore 1996) or Artificial
Neural Networks with fixed structures. Dealing with
high-dimensional spaces is a known challenge in
Reinforcement Learning approach (Łącki 2007)
which predicts the long-term reward for taking
actions in different states (Sutton and Barto 1998).
Evolving neural networks with genetic algorithms
has been highly effective in advanced tasks,
particularly those with continuous hidden states
(Kenneth, Bryant and Risto 2005). Neuroevolution
gives an advantage from evolving neural network
topologies along with weights which can effectively
store action values in machine learning tasks. The
main idea of using evolutionary neural networks in
ship handling is based on evolving population of
The artificial neural network is the helmsman's
brain making him capable of observing actual
navigational situation by input signals and choosing
an appropriate action. These input signals are
calculated and encoded from current situation of the
In every time step the network calculates its
output from signals received on the input layer.
Output signal is then transformed into one of the
available actions influencing helmsman’s
environment. In this case the vessel on route within
the restricted waters is part of the helmsman’s
environment. Main goal of the helmsmen is to
maximize their fitness values. These values are
calculated from helmsmen behavior during
simulation. The best-fitted individuals, which react
properly to wind effect, become parents for next
External disturbances acting on a seagoing ship are
mainly wind, wave and ocean current. In this paper
the author assumes that the waves in the harbor area
have little effect on ship maneuvering, and this type
of interference is omitted in the system.
As the ship moves forward on the straight path
(assuming there aren’t any significant distorting
external forces) there are the two major forces acting
on her - the force from the propellers and the force
of water resistance. At a constant speed, these forces
are equal, but with the opposite direction.
When the ship is turning the additional forces act
on the rudder and lateral forces from water pressure
appear. During this maneuver, the ship loses a little
of her velocity and the pivot point moves back
toward amidships.
2.1 Effect of wind in the ship handling
Under pressure of wind force, depending of the
ships’ design (location of the superstructure, the