565

1 INTRODUCTION

Watchkeeping officers are evaluated according to

exams by state authorities and company evaluation

procedures during recruitment, promotion and being

certificated. However, these theoretical exams and

procedures don’t cover evaluation in terms of

practical application. The evaluation bases such as

reactions to compelling navigation circumstances of

watchkeeping, the way they show their navigation

skills and experiences at bridge could just be seen

with the simulation which is real-like and has

different difficulty levels. In this regard, to perform

evaluation phase more efficiently, it has been

surmised that using bridge simulator would be an

effective method.

By compiling comprehensively from international

conventions, guidebooks and company ISMs

(International Safety Management), requirements for

officers and procedures during navigation in different

circumstances should be well considered. At this

point, competences that STCW (Standards of Training

and Certification of Watchkeepers) asks to officers

and which competences could be evaluated with

education based on simulator are stated in this study.

In addition, watchkeeping principles, how to

maintain a proper look-out, the principles of

performing navigation watch are stated. Moreover,

information about rules that are obligatory to obey by

COLREG (Convention on the International

Regulations for Preventing the Collisions at Sea) and

two important subjects in BPG (Bridge Procedure

Guide), situational awareness and risk of collision, are

detailed.

In the light of this information, the study aims to

create a bridge simulator application that could be

Development of Evaluation Procedures for

Watchkeeping Officers Using Bridge Simulator

M.S. Solmaz, B. Özsever, A. Güllü & C. Meşe

Piri Reis University, Istanbul

, Turkey

ABSTRACT: A comprehensive assessment of watchkeeping officers should be considered as a preventive

measure in terms of safety of navigation. Watchkeeping officers are assigned after passing exams of state

authorities and various evaluation procedures of companies. However, they are not subjected to comprehensive

evaluations with practical implementations. Therefore, this study aims to develop effective use of simulators in

terms of comprehensive evaluation of watchkeeping officers. In this study, simulation was designed with Goal-

Directed Task Analysis (GDTA) technique to indicate important points on the situations watchkeeping officers

face in navigation. 3 parameters were determined for difficulty adjustment; visibility, traffic density,

geography. At the last phase of this study, performance evaluation method was prepared for performance

evaluation of watchkeeping officer and scenario was evaluated for comprehensiveness by oceangoing masters.

Hereby, the developed performance evaluation method for navigation can be used to generate more reliable

method to evaluate the officers’ competence of technical skills.

http://www.transnav.eu

the

International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 14

Number 3

September 2020

DOI:

10.12716/1001.14.03.07

566

used for evaluating the watchkeeping officers

effectively and comprehensively. It was planned that

writing a scenario consists of four steps and difficulty

levels increase at each step, then these scenarios were

rendered as playable in simulator.

In the study, firstly the following questions were

answered:

− What are the requirements that are asked to

watchkeeping officers by international

conventions, guidebooks and companies for safe

navigation?

− What should be considered to write a scenario that

could supply best efficiency?

− What are the required parameters for evaluating

watchkeeping officers most effectively in

simulator?

1 GENERAL REQUIREMENTS AND RECENT

STUDIES

1.1 General requirements for watchkeeping officers

With STCW convention many rules came into force

about standards of competence. These competences

can be demonstrated with four different methods and

one of these methods is simulator training. Also this

convention provided watchkeeping principles in

general as well as explanation of watchkeeping under

different conditions and in different areas. How to

maintain a look-out and how to perform the

navigational watch are also its significant topics about

watchkeeping issue. All these topics under STCW

serve to improvement of watchkeeping skills and are

guide for watchkeeping officers.

For safety of navigation all vessels must comply

with rules of COLREG. Especially on high traffic

density areas COLREG plays an important role in

collision avoidance. A safe watchkeeping depends on

following the rules of COLREG. Therefore

watchkeeping officers must know, understand and

apply the rules.

Apart from STCW and COLREG, other significant

informations are gathered from Bridge Procedures

Guide and ISM system. Situational awareness and

risk of collision are stated in this study according to

BPG. On the other hand the requirements for

watchkeeping under restricted visibility and during

coastal navigation as well as CPA (Closest Point of

Approach) /TCPA (Time to Closest Point of

Approach) limits are investigated.

1.1.1 STCW

An officer in charge shall be required to plan and

conduct a passage and determine position as a

competence. Officer must have the ability to

determine the ship’s position by use of landmarks,

aids to navigation including lighthouses, beacons and

buoys, dead reckoning, taking into account winds,

tides, currents and estimated speed. He/she must also

have the ability to perform those by use of electronic

aids. STCW also requires competence for manoeuvre.

Depending on this competence officers must have the

knowledge of the effects of deadweight, draught,

trim, speed and under-keel clearance on turning

circles and stopping distances and the effects of wind

and current on ship handling. For any man over

board situation all officers shall know the manoeuvres

and procedures for the rescue of person overboard.

Also squat, shallow-water and similar effects must be

known as well as proper procedures for anchoring

and mooring (STCW, 2011). All abilities mentioned

above, can be demonstrated in bridge simulator.

1.1.2 COLREG

The following rules of COLREG can be evaluated

in simulator. Rule 5 (Look-out) which is laid emphasis

on STCW and BPG, Rule 6 (Safe speed), Rule 7 (Risk

of collision). A watchkeeping officer should be aware

of the collision risk according to the related conditions

under the Rule 7 (Deseck, 1983). Rule 8 (Action to

avoid collision), Rule 9 (Narrow channels), Rule 10

(Traffic separation schemes), Rule 13 (Overtaking),

Rule 14 (Head-on situation), Rule 15 (Crossing

situation), Rule 16 (Action by give-way vessel), Rule

18 (Responsibilities between vessels), Rule 19

(Conduct of vessel in restricted visibility) and light,

shape and sound signals can be also evaluated in

bridge simulator.

1.1.3 Bridge Procedure Guide

Bridge Procedures Guide of International

Chamber of Shipping is a well-known guide book for

safe bridge procedures. It is generally used by

Masters, watchkeeping officers, companies and

training institutions. BPG emphasises the importance

of situational awareness of watchkeeping officers for

safe conduct of vessels. BPG also gives suggestions

about risk of collision. These suggestions must be

taken into account to avoid any risk of collision

situation.

According to BPG, a qualified watchkeeping

officers should develop and maintain situational

awareness of the area around the ship, the ship’s

activities and the possible impact of external

influences on the safety of the ship. And this

awareness must include following issues (ICS, 1998).

− A clear understanding of the passage plan;

− An effectively managed Bridge Team;

− A proper and continuous look-out by all available

means;

− Familiarity with and understanding of bridge

equipment and the information available from

radar, AIS, ARPA and ECDIS;

− Using look-outs, ECDIS, radar and visual

monitoring techniques to confirm the navigation

safety of the ship;

− Using look-outs, radar and ARPA to monitor

traffic; and

− Cross-checking information from different sources.

1.1.4 Company ISM

ISM systems of shipping companies have

directives on different navigation conditions like

restricted visibility, coastal navigation for

watchkeeping officers. These include efficient radar

567

practices, position fix methods, monitoring of traffic

etc. Some company ISM can also define the minimum

CPA and TCPA; “if the circumstances permit, the CPA

should be at least 1.0 nm and the TCPA should be at least

15.0 minutes, if not; the turning circles of the vessel should

be taken into account to define the minimum CPA/TCPA.

Minimum CPA should not be set at radars less than the

diameter of the ship specific turning circle from 0° to 180º

and minimum TCPA should not be set less than the time

which vessel completes its turn to a contrary heading with

maximum rudder angle. If the vessel navigating at Pilotage

waters, canals or straits CPA and TCPA values should be

set as safe as possible.”

1.2 Recent studies

In earlier studies, Cook et al. (1981) performed a

simulator study to assess cognitive performances of

marine officers. They used the criteria which are mean

track, cross track variability (XTE), mean speed and

rpm, mean frequency of engine, rudder and course

orders, mean CPA to each vessel and lowest CPA to

each vessel for performance measurement.

In other studies, the complicated scenarios those

difficulty levels vary from easy to difficult, were used

for performance measurement. Robert et al. (2003)

constituted 6 scenarios including routine or

emergency collision threat, alterable or fixed target

behaviour and traffic density. They took the

variabilities of collision risk, deviation from track,

course changes, rule following, target acquisition, test

manoeuvre, bearings taken for ship control measures.

Grabowski and Sanborn (2003) determined better

performance parameters of operators as smaller XTE,

fewer manoeuvring order command, fewer

communication and sufficient CPA in three levels of

navigation scenario. While lower level of scenario

contains clear visibility and low traffic, medium level

has high traffic and equipment failure and high level

of scenario contains tidal currents, speed restrictions,

restricted waterway, traffic congestion, bad weather

and heavy traffic. Similarly, Gould et al. (2009)

developed the difficulty levels of the scenarios for

performance evaluation using geography, visibility

and traffic density variables.

Kim et al. (2010) constituted the scoring index

including collision avoidance ability, decision making

time and degree of deviation based on only COLREG

rules. Maurier et al. (2011) in the same way as in

previous studies, have performed performance

measurement in routine and unplanned events by

making the traffic multilevel. Kircher and Lutzhoft

(2011) used criteria such as position taking, rule

following (COLREG), and detection range of targets,

keeping a safe CPA, communication and attention

variabilities for performance evaluation with the

similarity of previous studies.

2 METHOD

2.1 Goal-Directed Task Analysis (GDTA) for scenario

design

In the performance evaluation, the GDTA method

which is based on the Situation Awareness (SA)

model of Endsley (1993) (Figure 1) was used for the

outputs requested from the operators. According to

model, situation awareness is the perception of the

elements in the environment within a volume of time

and space, the comprehension of their meaning, and the

projection of their status in the near future (Endsley,

1993).

Figure 1. Goal-Directed Task Analysis model

(Endsley,1993).

The outputs requested from the operators has

integrated to GDTA in maritime domain as in Figure

2. These are collected and harmonized from previous

studies.

Figure 2. Integration of GDTA to safe navigation

parameters.

While ‘Safe Navigation’ is main goal of the task,

‘Collision avoidance’, ‘Identify and communicate

navigation landmarks’, ‘Determine position’ and

‘Identify hazards’ are sub-goals of the task. The items

1.1, 1.2 and 1.3 are respectively perception,

comprehension and projection process items of SA

model. Scoring index of performance will include the

items such as 1.2.1, 1.2.2 etc.

Figure 3 presents the integration of GDTA to better

performance parameters in similar with the

integration of GDTA to safe navigation parameters.

568

Figure 3. Integration of GDTA to better performance

parameters.

2.2 Ship, geographical area and voyage particulars for

created scenario

A LNG vessel was chosen for scenario (Figure 4). She

has 297,5 m length overall and 10,8 m maximum

draft. Dover Strait was chosen to ensure having

realistic conditions of strait passage in terms of traffic

density (Figure 5).

Figure 4. The LNG vessel used in scenario

Figure 5. Dover Strait

More than 400 commercial vessels use the Dover

Strait every single day. Geographically, Dover Strait

has strong tides, sandbanks, shoals. Weather

condition can change rapidly. Also restricted visibility

is possible and this situation makes navigation

difficult. Cross channel traffic density is high because

of ferries. (UK., 2018)

The scenario was divided to 4 steps those have

different difficulty levels. 3 parameters were

determined for difficulty adjustment:

− Visibility

− Traffic density

− Geography

The outputs requested from operators vary with

regard to the difficulty level of step.

2.2.1 Step 1

As difficulty levels of first step, operator perform

the scenario in high visibility, low traffic density and

easy geography. The operator is requested to alter

course to starboard to avoid collision and overtake

another vessel as seen in Figure 6.

Operator should consider the related COLREG

rules about collision avoidance and overtaking

situations in this step. Moreover he/she is evaluated

according to the better performance parameters for

navigation (Figure 3).

Figure 6. The interactions of Step 1.

2.2.2 Step 2

In this step of the scenario, visibility is high, traffic

density is moderate and geography is easy. Operator

is requested to perform the necessities of vessel being

overtaken and do necessary actions for two risks of

collision as seen in Figure 7.

Figure 7. The interactions of Step 2.

569

Operator should consider the related COLREG

rules as previous step and try to perform scenario in

area where the traffic density is higher than first step.

2.2.3 Step 3

Visibility is moderate, traffic density is high and

geography is moderate in third step of the scenario.

At the beginning of the step, the vessel which is at the

other side of the separation, is altering her course and

making a short cut. This causes collision risk for

operator later. Besides, operator is requested to

handle two more collision situations during the step

(Figure 8).

Figure 8. The interactions of Step 3.

After waypoint, operator is faced with damaged

tanker and leakage. Operator should proceed safely at

that area. With the decreasing of visibility to 4 miles,

operator should reduce the speed and make the

proper arrangements on radar to use it effectively.

Acceptable CPA limits, identification of buoys, rule

following for separation are other parameters which

operator should consider in addition to parameters of

previous steps.

2.2.4 Step 4

In the last step of the scenario, visibility is low,

traffic density is high and geography is hard. At the

beginning of the step, the operator is requested to

make a sharp turn to proceed opposite side of the

separation. There are 3 risk of collision situations in 15

minutes and operator is overtaken by another vessel

(Figure 9). At 20

minute of the step there is strong

current causes the vessel drift to a wreck. Moreover,

there are nets of fishing vessels at the port side of the

course.

Operator should reduce the speed due to decrease

of visibility to 1.5 mile, high traffic and short distance

to pilot station. Current affects operator in terms of

manoeuvre of ship as well as the visibility in this step.

Safe manoeuvre of ship becomes crucial due to fact

that there is wreck, fishing nets and current. Hence,

operator is evaluated according to especially ship

control measures.

Figure 9. The interactions of Step 4.

3 RESULTS

3.1 Expert comments

The questions were asked to ocean-going masters to

evaluate the whole scenario in terms of consistency of

scenario to be used for evaluation of watchkeeping

officers. The questions and related answers are stated

below;

− How much the interactions, which are specified in

steps, reflect the truth?

1st reviewer:

“The extreme occurrences in your scenario could

not happen simultaneously. However the

possibility of happening simultaneously should

taking into consideration too. Also watchkeeping

officers should always be well prepared against

this kind of situations because the regions as

Dover Strait have high traffic density. So in my

opinion it is close to real-like.”

2nd reviewer:

“Scenario generally summarize complications at

strait area. However beside our own decisions,

communicating with VTS and port authority and

listening directions are very important. If this

point would improve, scenario could be more

affective for evaluation.”

3rd reviewer:

“This complications and even more complicated

situations and events are possible to happen in the

regions have high traffic density and separation

areas. Especially in the Singapore Strait and region

of continent the traffic density is even higher.”

− Are performance measures proper and adequate

for evaluation of watchkeeping officers? Do you

think is there any criterion to add or take out?

1st reviewer:

“In my opinion situational awareness is most

important evaluation criterion. If watchkeeping

officer is calm and confident, situational awareness

will expand correspondingly. And this helps to

570

notice of parameters in sight and by electronic

equipments. Most significant thing is usage of

these information during watchkeeping. As is seen

this study emphasised on these.”

2nd reviewer:

“Accordance with passage plan and knowledge of

watchkeeping officers about each COLREG rule

are the two criterion which can be add to

performance measures.”

3rd reviewer:

“Most of collisions arise from reason that intention

of ship could not be figured out before. Evaluation

in the simulator also should involve the criterion:

the start time of avoiding manoeuvre from

collision. It is really important to communicate by

VHF directly, verbally and actively with the faced

ship. Under the favour of asking “what is your

intention please” question in the right time and

receiving the answer, the officer could manoeuvre

in the right time and right direction. The officer

should manoeuvre properly in advance by

avoiding from the risk of collision or should not be

late for VHF communication. In this context

performance measures are enough and I advise to

add the starting time of avoiding manoeuvre from

collision and time of communicating with other

ships as criteria.”

− Are the steps in a linear form?

1st reviewer:

“Steps are linear and becoming difficult in proper

proportion. Parameters are changing properly. The

only deficiency that I noticed is bad weather

conditions are important too apart from restricted

visibility. And if it is possible weather conditions

should be added to further studies.”

2nd reviewer:

“Distribution of parameters to steps are excellent

and proportional.”

3rd reviewer:

“Yes, difficulties are rising at each step and it is in

correct form.”

3.2 Performance evaluation

Performance evaluation was generated based on

Goal-Directed Task Analysis (GDTA) structure by

ocean-going masters. Table 1 presents the sample

coefficients of the necessary evaluation parameters for

each step of the scenario.

Table 1. The coefficients (α) of each criteria for each step.

____________________________________________

Step 1 Step 2 Step 3 Step 4

____________________________________________

Safe γ1

γ11 0.1 0.15 0.1 0.1

Navigation γ

12 γ121 0.1 0.1 0.1 0.05

122 0.1 0.15 0.1 0.05

123 0.2 0.15 0.1 0.05

13 γ131 0.15 0.15 0.15 0.15

132 0.15 0.2 0.1 0.1

133 0.05 0.05 0.05 0.05

2 0.05 0 0.05 0.05

3 0.1 0.05 0.1 0.15

4 0 0 0.15 0.25

Total 1 1 1 1

____________________________________________

Better η1

0.2 0.2 0.2 0.2

Perfor- η

2 0.2 0.2 0.2 0.2

mance for η

3 η31 0.2 0.2 0.2 0.2

Navigation

η32 0.1 0.1 0.1 0.1

η33 0.1 0.1 0.1 0.1

η34 0.2 0.2 0.2 0.2

Total 1 1 1 1

____________________________________________

* γw represents the symbol of score for ‘safe navigation’

parameters, η

ν represents the symbol of score for ‘better

performance parameters for navigation (See Figure 2-3 for

other symbols).

The operator's performance is equal to the

weighted sum of the scores evaluated for the relevant

parameters. Following equation presents the

performance score for ‘safe navigation’:

Psafe αw. w

∑

(1)

where α

w = coefficient of criteria; γw = performance

score on related parameter of ‘safe navigation’.

Similarly, following equation presents the

performance score for ‘better performance for

navigation’:

Pbetter perf. αν. ν

∑

(2)

where α

ν = coefficient of criteria; ην = performance

score on related parameter of ‘better performance for

navigation’.

4 CONCLUSION

State authorities and shipping companies evaluate

watchkeeping officers according to written theoretical

exams during the process of recruitment and

promotion. However, as the theoretical exams and

similar procedures do not evaluate watchkeeping

officers in terms of practical implementation,

evaluation of navigation performance on bridge

during navigation cannot be done properly. This

makes difficult the decision-making process on

selection of watchkeeping officers. Therefore, the

simulator-based assessment is thought to improve the

quality of the watchkeeping officer selection.

In this study, it was tried to be evaluated the

applications in ISM documents of companies and

571

requirements of BPG and STCW. It was seen that

some items such as celestial navigation, pollution

prevention, passage planning cannot be evaluated in

simulator environment. However, most of technical

skills of navigation can be evaluated and simulated in

the scenario.

Experts mainly argued the scenario that is mostly

real-like, performance measures of that is sufficient to

evaluate the officers and the steps are progressively

designed, the difficulty levels are distributed

properly. One of the experts stated that the detection

time and range of targets is important issue to avoid

any collision. The novel performance evaluation

method involves this parameter. Another one

suggested that VHF communications should be in

steps to carry out the ship-to-ship and ship-to-shore

interactions. It is important to reflect the truth for

navigation scenario.

Goal-Directed Task Analysis (GDTA) method is

seen to be used for a performance evaluation method

in navigational watch. The coefficients, stated in Table

1, can be re-evaluated for different steps of different

scenarios. In this study, for example, the coefficient of

criteria ‘Identify hazards (γ

4)’ is 0 for step 1, 0.25 for

step 4. The coefficient may be increased for another

step where hazards are more common. Therefore, this

performance evaluation method can be upgraded for

other simulator scenario designs.

As a result, the developed performance evaluation

method for navigation can be used to generate more

reliable method to evaluate the officers’ competence

of technical skills.

REFERENCES

Cook, R. C., Marino, K. L., & Cooper, R. B. (1981). A

Simulator Study of Deepwater Port Shiphandling and

Navigation Problems in Poor Visibility (No. EA-80-U-

099). ECLECTECH ASSOCIATES INC NORTH

STONINGTON CT.

Deseck, P. 1983. International Regulations for Preventing

Collisions at Sea: Barker & Howard Ltd.

Endsley, M. R. (1993). A survey of situation awareness

requirements in air-to-air combat fighters. The

International Journal of Aviation Psychology, 3(2), 157-168.

Gould, K. S., Røed, B. K., Saus, E.-R., Koefoed, V. F.,

Bridger, R. S., & Moen, B. E. (2009). Effects of navigation

method on workload and performance in simulated

high-speed ship navigation. Applied ergonomics, 40(1),

103-114.

Grabowski, M., & Sanborn, S. D. (2003). Human

performance and embedded intelligent technology in

safety-critical systems. International journal of human-

computer studies, 58(6), 637-670.

ICS 1998. Bridge Procedures Guide: Marisec Publications.

Kim, H., Kim, H. J., & Hong, S. (2010). Collision Scenario-

based Cognitive Performance Assessment for Marine

Officers.

Kircher, A., & Lutzhoft, M. (2011). Performance of seafarers

during extended simulation runs. In International

Conference on Human Factors in Ship Design and Operation

(pp. 53-59).

Maurier, P., Barnett, M., Pekcan, C., Gatfield, D., Corrignan,

P., & Clarke, G. (2011). Fatigue and Performance in

Bridge and Engine Control Room Watchkeeping on a

6on/6off Watch Regime”.

Robert, G., Hockey, J., Healey, A., Crawshaw, M., Wastell,

D. G., & Sauer, J. (2003). Cognitive demands of collision

avoidance in simulated ship control. Human factors,

45(2), 252-265.

STCW, International Convention on Standards of Training,

Certification and Watchkeeping for Seafarers (STCW)

1978, as amended in 1995/2010, 2011.

UK. (2018). Dover Strait crossings: channel navigation

information service (CNIS). Available:

https://www.gov.uk/government/publications/dover-

strait-crossings-channel-navigation-information-

service/dover-strait-crossings-channel-navigation-

information-service-cnis, Access Date: 23.12.2018