255
1 INTRODUCTION
Over the last decades, the maritime industry has
witnessed some rewarding outcomes from the
adoption of the Crew Resource Management
frameworkandtheuseofsimulators inthetrainingof
bridgeofficers.Still,therearedoubtsamongmaritime
professionalsabouttheeffectivenessofsuchtraining
in the improvement of ma
ritime safety (Barnett,
Gatfield,&Pekcan, 2006). In aneducational context,
the situation is more problematic since, among the
requirementsofdevelopingtechnicalskills,thereisa
needtobuildandevaluatethenontechnicalskillsof
cadets with little experience in bridge team
management. For some time, the Portuguese Naval
Academy have been using the ship’s navigation
simulator (NAVSIM) to conduct tea
m leadership
model courses (Bué, Lopes, & Semedo, 2015). More
recently, namely after the latest NAVSIM’s upgrade
in 2004, there was an overall perception among
students and instructors that the complexity of the
trainingneededanewapproachforthedevelopment
andassessmentofnontechnicalskills.Thus,astudy
tofurt
herassesshowthesimulatorisusedandhow
to improve the evaluation and assessment of non
technical skills during navigation training was
initiated.The adopted approach aimed to develop a
framework supported by previous research results
andcont
extualizingtheneedsofthePortuguesenavy,
notonlythrough lecturerobservationsand
assessment of the students, but also by analysing
Navy accidents reports over the last two decades.
Thisresearchisanimportantcontributiontoincrease
the effectiveness of NAVSIM training sessions, by
Development of a Behavioural Marker System for
Rating Cadet’s Non-Technical Skills
V.P.daConceição
ChalmersUniversityofTechnology,Göteborg,Sweden
UniversityofLisbon,Lisbon,Portugal
PortugueseNavalSchool,Lisbon,Portugal
J
.C.Basso&F.C.Lopes
PortugueseNavalSchool,Lisbon,Portugal
J
.Dahlman
ChalmersUniversityofTechnology,Göteborg,Sweden
NavalPostgraduateSchool,Monterey,CA,USA
ABSTRACT: Despite the adoption of crew resource management training for bridge teams over the last
decades,thetrainingisstillheavilyfocussedontechnicalachievements.Inaneducationalcontext,thesituation
ismoreproblematical,sincewithrequirementofdevelopingthetechnicalskills,thereisa needtobuildand
evaluatethenontechnicalskillsofcadetswithl
ittleexperienceinbridgeteammanagement.Inparallelwith
theapplicationofteamleadershipmodels,thePortugueseNavalAcademyconductedaresearchtoimprove
thedevelopmentandassessmentofnontechnicalskillsinbridgesimulators.Thispa
perdescribesthemethod
usedto identify thekey nontechnical skillsrequired for navalcadetsand todevelopa behavioural marker
systemfortheirmeasurement.Aliteraturereviewofbehaviouralmarkersystemswassupplementedwithan
analysis of interviews conducted with students and simulator instructors. Additionally, furt
her analysis of
PortugueseNavyaccidentsreportswasmade,applyingtheHFACSframeworktoidentifythe relevantnon
technicalskillsinvolvedintheaccidents.Theresultingratingsystemcovers five skill categories (leadership,
situationalawareness,communication,teamworkanddecisionmaking),eachonewiththreeratingelements.
Theframeworkiscurrentlyunderevaluationtestsinbridgesimulatorssessions,withinaneducationalcontext.
http://www.transnav.eu
the International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 11
Number 2
June 2017
DOI:10.12716/1001.11.02.07
256
addinganobjectiveframeworkfortheassessmentof
nontechnical skills to the present technical skill
developmentprogram.
1.1 NAVSIMtrainingandeducation
Regardless of the general consensus over the
advantagesofnavigationsimulatorsineducationand
training,namelyinrelationtoincreasedsafety(Cross,
2012; Magdy Ali Elashkar, 2016),
their effectiveness
has been questioned. McCallum and Smith (2000)
evaluated simulators and provided a framework for
their assessment. From another perspective, Cross
(2003, 2007), Emad and Roth (2008) researched the
development of competencebased training and
assessment in simulators. Other researchers have
shown that training programs display some
weaknesses in
the development of decision making
skills,whenaddressing themanagement ofcomplex
situations (Chauvin, Clostermann, & Hoc, 2009).
Other studies emphasize issues associated with the
effectiveness of simulation training, such as the
advantagesofexhaustivetrainingofunsafesituations
balancedwiththeriskofimprintingriskybehaviours
(Forsman,Hill,Dobbins,Brand,
&MacKinnon,2012).
Newinsightsonpedagogicalstrategiesarecalling
for a more innovative role of the lecturer. Students
need to learn by experiencing their working
environment and performing tasks with experts.
Thus, instructor shall gradually leave space for the
studentsandcoachthemintheirunderstandingofthe
undergoing
processes (Emad, 2010; Magdy Ali
Elashkar,2016).Arecentstudy,whichwasbasedona
survey to assess training in simulators at the
MerchantMarineAcademyofAthens,Tsoukalasetal.
(2015) concluded that adjustments in the course
programs and methodologies were necessary. They
specificallysuggested modificationtomethodologies
used
in the evaluation of trainees and the teaching
designforproblemsolving.
1.2 Nontechnicalskills(NTS)
Thisterm,commonlyusedinsafetycriticaldomains,
hasbeendefinedbyFlin,O’Connor&Crichton(2008)
as the cognitive, social and personal resource skills
that complement technical skills, and contribute to
safeand
efficienttaskperformance.Still,thereareno
straight guidelines to classify NTS, since their
relevanceandappearancearestronglycoupledtothe
workingcontext.Yet,NTShavetwodissimilareffects,
since good skills reduce the possibility of human
errorsand,ontheotherside,poorskillincreasesthe
chance of
error occurrence or extend their
consequences. Regardless of the research made to
addressthedevelopmentofNTSinthemaritimefield
(Barnett et al., 2006; Devitt & Holford, 2010;
Hetherington, Flin, & Mearns, 2006; Long, 2011;
MagdyAliElashkar,2016;Saeed, 2010),Hayward&
Lowe (2010) identified significant shortfalls in
maritime
training,whencomparedtootherdomains.
Basedontheexistingfindings,wemaycommonly
agree with the following list of nontechnical skills,
frequentlyrecognisedwhenaddressingsafetyissues:
Situationalawareness;
Assertiveness, ability to criticize and to be
criticized;
Decisionmaking;
ProblemSolving;
Leadership,ability
touseauthority
Managingstress;
Workload management, Multitasking and
selectiveattention;
Teamwork,teambuildingandcooperation;
Communication,Interteamcommunication;
Copingwithfatigue;
Timemanagement,planningandpreparing
Situationmonitoring
Anticipationoffuturestates;
Settingpriorities;
CrisisManagement.
1.3
Behaviouralmarkerssystems
Followingthefirstbeha viouralmarkingsystemmade
bytheUniversityofTexas(UTBehaviouralMarker),
created for pilots, others have been developed for
severalworkingdomains,suchasthe:
Pilots’nontechnicalskills(NOTECHS)inaviation;
NonTechnicalSkillsforSurgeons(NOTSS);
Anaesthetists’Non
TechnicalSkills(ANTS);
TraumaNonTechnicalSkills(TNOTECHS);
Well Operations Crew Resource Management
(WOCRM)inoffshorewellcontrol.
In the maritime industry, few developments are
foundwithafirmemploymentofamarkingscheme
for the Bridge Resource Management (BRM)
framework. After undertaking a study for the
development
ofresourcemanagementandleadership
behavioural markers for shipping, Devitt & Holford
(2010) pointed out some factors that have to be
considered when establishing behavioural markers,
namely:
Culturalimplications;
Shipboardandshoresidecontextualdifferences;
Specificorganisationalneedsandrequirements;
TrainingofBehaviouralmarkerassessors.
Their
behavioural markers were yet to be
validated in simulated environments. In another
research,Long(2011) developeda marking
framework (Nontechnical Skills for Officers of the
Deck‐NTSOD) to be used for the evaluation of
Officers of deck in the US Navy. It comprises four
categories (leadership, decisionmaking,
communications and situation
awareness), with ten
rating elements, on a scale of 1 to 4. This prototype
wasinitiallydesignedtobeusedonboard,however,
the author considered the need for further work,
namelyitsimplementationinsimulationtraining for
moretestsandrefinementsoftheratingsystem.
1.4 TheHFACSFramework
Grounded in Reason’s “Swiss cheese” model of
accident causation, the Human Factor Analysis and
Classification System (HFACS) was designed to
provide a classification scheme and used for the
systematicidentificationofaccidentcausationfactors
(Shappell&Wiegmann,2000).TheHFACScomprises
fourhierarchicallevelsof failures, from UnsafeActs
atthe
frontenduptotheOrganizationalInfluenceat
257
the backend, passing by Preconditions for Unsafe
Acts and Unsafe Supervision. After its initial
application in military aviation accidents analysis,
severalderivedmodelshavebeendesignedforother
domains such as the rail, mining and medicine
industry. Adaptation to the maritime industry, has
also been undertaken by several authors
(Celik &
Cebi,2009;Chauvin,Lardjane,Morel,Clostermann,&
Langard, 2013; Chen et al., 2013; Rothblum et al.,
2002).TheHFACSMAmodeldesignedbyChenetal
(2013)includesanadditionallevel(Otherfactors),set
above the organizational level, to address the
legislation and political factors. To perform the
analysisofcollisionsatsea,Chauvinetal(2013)also
made some adaptions to consider factors such as
communicationandBRM.
Additionally, it was found that, despite the
positive results on inter‐ and intrarater reliability
tests of the HFACS framework, it still needs
improvements in its implementation (Ergai et al.,
2016).
2 METHODOLOGY
TheproposedmethodologyaimedtodevelopanNTS
behaviourmarkingsystemtobeusedintheNAVSIM
of the Portuguese Naval Academy. Technical skills
were not consideredin this research,since there are
alreadyobjectivelyevaluatedthru thecomplianceof
proceduresandtheeffectivenessofthedecisions
and
actions. Thus, we considered a fourstep process to
design a framework that should be coupled to the
navy needs and context. The first step was a
comprehensive literature review to create a list of
NTSthathavebeeneffectivelyusedinsafetycritical
domain,whichwouldbeusefulfor
theOfficerofthe
Watch (OOW). This was followed by focus group
questionnaires to NAVSIM instructors, aiming at a
bettercategorizationoftheNTS.Thethirdstepaimed
at the contextualization of the navy needs, thus we
performed an analysis of accident reports involving
navy vessels, complemented by a second
set of
questionnaires involving NAVSIM trainees. The last
step consisted of the qualitative analysis of the
collected data and design of the NTS behaviour
markingsystem.
2.1 Questionnaires
All questionnaires were completed over the first
semesterof2016.Thefirstquestionnairewasgivento
lecturers and instructors who use or have
recently
usedbothNAVSIMfromtheNavalAcademyandthe
Tactical Training Centre, in training and teaching
sessions, making a total of 10 participants. The
questionnairewascomposedbythreeparts.Thefirst
onecoveredthedemographicdataincludinggender,
age, years of experience and attended courses. The
secondpart,
aimedtheevaluationoftheeducational
programinrelationtotheuseoftheNAVSIMandthe
characterization of the simulated sessions (9
questions).Thelastpartfocusedontheassessmentof
howtheNTSaredevelopedandwhichareconsidered
themostrelevant(24questions).
The second questionnaire was directed
towards
student perceptions over the educational program
around the use of the simulator. The questionnaire
waspresented to all students, except those from the
1stacademicyear,fromallgraduate degreeprogram
that use the NAVSIM, totalling 139 participants,
representing90%ofthepopulation(seeTable1).The
navygraduate
degreeprogramme(63%ofthecadets)
has more courses with modules conducted in the
NAVSIM. This questionnaire also comprises three
parts. The demographic part collected age, gender,
academicyearandcourseprogram. The second part
aimed the evaluation of the educational program in
regardstotheirtrainingasOOW(10questions).
The
third part pointed the student perception over the
development of their technical and nontechnical
skillsinsimulatedtraining(22questions).
Table1.Studentparticipants
_______________________________________________
Academicyear Total 
Graduateprog. 1
st
 2
nd
3
rd
 4
th
 5
th
 N %
_______________________________________________
Navaladministration 4 4 5 4 17 17100%
Weapons,elect.Eng. 5 2 3 3 13 13100%
MechanicalEng. 7 6 3 5 21 21100%
Marine0 0 2 1 3 0 0%
Navy28 28 24 18 98 8890%
_______________________________________________
Total44 40 37 33 154 13990%
_______________________________________________
Participants(N) 42 33 35 29 139
95% 83% 95% 88% 90%
_______________________________________________
Both questionnaires involved mostly closeend
questions of multiplechoice and ordering. Seven
openended questions were included, three for the
students and four for the instructors. Before the
implementation, a pretest was performed on 3
individuals, to validate the adequacy of the
questionnaires.Theanalysiswasperformedwiththe
SPSSv.20software.
2.2 Accidentsanalysis
The analysis of the accidents was made with the
HFACS‐ Coll framework proposed by Chauvin et al
(2013), with minor changes to adapt it to the cases
under analysis, namely to account the groundings.
The accident reports used in this analysis were
obtained from
investigation reports set up by
Portuguese Navy, on accidents involving navy
vessels. All the navigation accidents cases were
considered, totalling 20 cases. They involve 8
collisions, 5 groundings and 7 collision in mooring
manoeuvres, from 1995 to 2016. The accident events
occurredwithinthefollowingcontext:
Oceannavigation2
Coastalnavigation4
Restrictedwaternavigation14
Closemanoeuvres6
Trainingprograms4
Thecodingprocesswasseparatelycarriedoutby
two analysts, one of them an experienced mariner,
usingqualitativeanalysissoftware(NVIVO).
Two iteration were made and the interrater
reliability was assessed by measuring
percentage
agreementsandKappacoefficients.Asitisshownin
Figure 1, the second iterationsignificantly increased
258
the degree of agreement. The adjustments of the
HFACS framework were performed during the
codingprocess,afterdiscussionsbetweenthecoders.
Figure1.Kappadistributiondiagramofthecodingprocess
3 RESULTS
3.1 Nontechnicalskills
The NTS were mostly retrieved from existing
behavioural marker systems, the UT Behavioural
Marker(Klampferetal.,2001),NOTECHS(Flinetal.,
2003),ANTS(Fletcheretal.,2004),NOTSS(Flin,Yule,
PatersonBrown, Rowley, & Maran, 2006), and
NTSOD (Long, 2011), we also considered skills
identifiedby Flin etal. (2008)and Devitt & Holford
(2010). The literature review resulted in a list of 13
categories(seetable3),whereagreateremphasiswas
found in skills like communication, leadership;
situational awareness, decisionmaking and team
work. The criteria used for choice and descriptions
were
guided by the necessity to: 1) cover all the
critical behaviour; 2) avoid overlap between
categories;3)useterminologyeasilyperceivedwithin
the organization; 4) be easily accessible and
observable;and5)bemeasurableinsomedimensions,
evenqualitatively.
3.2 Lecturersurvey
The participants, were all navy officers, with an
average
of 6,5 years of experience in NAVSIM
training,90%male,twoyoungerthan35yearsand4
olderthan 45,none withany documentedsimulator
training education. The majority (80%) considered
thatthetrainingintheNAVSIMcouldbeimproved.
The participants use the NAVSIM for four different
coursesprograms,
asitcanbeseeninthetable2.
Table3presentstheNTSthatwereconsideredas
themostrelevant,byaskingthe participanttoselect
fiveofthe13categoriesofNTS.
Twoquestionsweredirectedfortheperceptionof
thebestsimulatedsession,onetoknowwhichtype
of
session is usually used and a second to identify the
recommended type for the development of non
technicalskills.LookingattheresultsinTable4,we
mayseethattherecommendedtypeofsessionisthe
one that flavour a greater integration of the lecturer
withintheteam
activity.
Table2.CourseprogramsconductedintheNAVSIM
_______________________________________________
Whichcourseprogramdoyoumanage?Y N
_______________________________________________
Tacticalnavigation/navaloperations 6 2
Seamanship3 5
Leadership/organizationalbehaviour 1 7
Navigation7 1
_______________________________________________
Table3.IdentificationofthemostrelevantNTS
_______________________________________________
NTSN  Order
_______________________________________________
Decisionmaking9 1
Situationalawareness8 2
Leadership6 3
Taskplanningandmanagement6 3
Monitoring,vigilance6 3
Teamwork5 4
Communication5 4
Assertiveness3 5
Managingstress1 6
Perception,intuition1 6
Copingwithfatigue1 6
Energy,mentalalertness0 12
Workloadmanagement
0 12
_______________________________________________
Table5presentsthesomeresultsofthelecturers’
survey. While the first set of questions attempts to
characterize the context of the training session, the
secondtriestogivesussomeunderstandingoverthe
participant perception in the development of non
technicalskills.
3.3 Studentssurvey
The participants of this
survey were all students of
the Naval Academy, being between 18 and27 years
old, 41.7% being between 21 and 22, 79% male.
Bearing in mind that 97,1% have already performed
tasks in a bridge team, 92.8% considered that
simulation training could be improved and almost
halfofthemconsidered
longtermsessionsasagood
waytoincreasetheir skills.Threedifferenttestswere
performed with the collected data, Kolmogorov
Smirnov, KruskalWallis and MannWhitney.
KolmogorovSmirnovtestshownthatthedatadoesn’t
have a normal distribution, therefor we choose to
proceedwithnonparametrictests.
Table4.ConfigurationtypesforthesimulatedsessionsintheNAVSIM(Yyes,Nno/scale0to5,where0standsforno
answer,1fordisagreeand5fortotallyagree)
__________________________________________________________________________________________________
WhichtypeofsessionbestsuitthedevelopmentoftechnicalandNTS?0 1 2 3 4 5
x

__________________________________________________________________________________________________
Playingthescenario,withnointerruption/instructorinthecontrol roommonitoring6 11,1
Playingthescenario,withnointerruption/activepresenceoftheinstructorinthebridge1 2 1 3 3,9
Playingthescenario,withinterruptions,forcoachingandexplanationsoftheinstructor,inthebridge
3 4 4,6
Playingthescenario,withnointerruption/activepresenceoftheinstructorinthecontrolroom4 3 3,4
__________________________________________________________________________________________________
259
Table5.InstructorsperceptionontheuseoftheNAVSIMasaneducationaltool(scale0to5,where0standsfornoanswer,
1fordisagree/never/verybad,and5fortotallyagree/always/verygood)
__________________________________________________________________________________________________
0 1 2 3 4 5
x
__________________________________________________________________________________________________
PartII
ArethenumberofinstructorsinNAVSIMsessiontrainingenough?2 1 2 4 1 3,3
WhatisthequalityoftheNAVSIMfacilities?2 1 6 1 4,0
Arethenumberoftrainingsessionssufficient?2 5 3 0 3,4
ThetotalnumberofNAVSIMtraininghoursfor
Navygraduatedegreeprogramissufficient 4 5 1 3,2
ThetotalnumberofNAVSIMtraininghoursforMarine, EngineersandAdministrationgraduate 6 3 12,3
degreeprogramissufficient
Importanceoflongtrainingsessions(>12hours)2 1 0 1 5 1 3,6
PartIII
Doyouagree
thattraininginNAVSIMisrelevantforthedevelopmentofbothtechnicalandNTS24 4 4,5
ofthefutureOOW
HowimportantisthedevelopmentofNTSintheNAVSIM?1 6 3 4,2
Doyouperformbriefingsanddebriefings?2 4 4 4,2
DoyouencourageOOW
traineestoassignroles/tasksandclarifytheresponsibilitiestotheremaining 1 4 5 4,4
membersoftheirteam?
Doyouevaluatethetraineesindividuallyaftereachsession?2 6 2 4,0
Doyouevaluatethetraineesasateamaftereachsession?1 6 3 4,2
Do
youencourageOOWtraineestomonitorthetasksandsustainacommonsituationawarenesswithin 1 5 4 4,3
theteam?
Doyouencourageteamwork?22 6 4,8
Doyouencouragedecisionmakinginsafetycriticaloruncomfortablesituations?27 1 4,1
Doyouencouragetheuse
offormalcommunicationformswithintheteam?28 5,0
Doyouevaluatetheradiocommunicationprocedureswithothershipsandshorestations?2 1 0 3 4 4,3
Istheindividualtrainingsession,precededbyplanningwork?2 1 7 3,9
Isthegrouptrainingsession,precededbyplanning
work?2 1 1 4 2 3,9
Justbeforethesession,doyoubriefthestudentswiththesessiongoals,planandevaluation
methodology?1 1 4 4 4,3
Justafterthesession,doyoudebriefthestudentswithananalysisofthesession,lessonslearned 13 6 4,7
andgoodpractices?
DoyouconsiderthatNAVSIMtraininghelpsthedevelopmentofleadershipskills?24 4 4,5
DoyouencouragetheOOWtraineetosetprioritiesinaccordancewiththesituations3 7 4,7
__________________________________________________________________________________________________
The results of the KruskalWallis test gave high
degree of significance in 7 over 13 questionsfor the
Academics Years, and 9 over 13 questions for the
graduate degree programs. The MannWhitney tests
(multiplecomparisons) weremade forthe questions
withhighdegreeofsignificance.
Theresults
expressacleartrendfromthe2
nd
tothe
5
th
years, namely in regards to the increasing
instructor’s engagement to follow communication
standards, providing team working instructions and
decisionmaking in stressful situations. Differences
were also found between the graduate programs,
especiallybetweentheNavygraduatesandgraduates
fromtheotherprograms.Thiscouldbeconnectedto
the fact that
until the end of the 2
nd
year they all
attendthesamecourses,laterontheyfollowdifferent
curriculum,withtheNavygraduatesattendingmore
courses and more demanding tasks in the SIMNAV
sessions. This is also reflected in the instructor
behaviour and, on the requirements, he/she sets for
thetrainees.
In general, more than half
of the participants
considered that they are motivated to manage tasks
andtosetresponsibilitieswithintheteam.
Additionally,about2/3reportedthattheyusually
have briefings and debriefings of the sessions. 2/3
reports that the NAVSIM sessions helped them to
developtheir leadership skills.They considered that
theyaremostly
evaluatedasateamintheNAVSIM
sessionsratherthanasindividuals.
Table6.SummaryresultsofHFACSanalysis(levels1,2and
3)
_______________________________________________
HFACSFactorsN %
_______________________________________________
Unsafeacts20 100
Errors20 100
Skillbasederrors11 55
Decisionerrors15 75
Perceptualerrors17 85
Violations9 45
Routineviolations2 10
Exceptionalviolations8 40
Preconditionsforunsafeacts20 100
Environmentalfactors20 100
Physicalenvironment14 70
Hydro
METOCphenomena10 50
Visibilityorlighting5 25
Technologicalenvironment17 85
Shipbuildingbridgedesign2 10
Radar,ECDIS,NAVAIDSfailure6 30
Nonuseormisuseofinstruments17 85
Conditionsofoperators12 60
Adversementalstate12 60
AffectedSA3
15
Attentiondeficitworkload12 60
Complacency1 5
Personnelfactors19 95
SRM19 95
Intershipcommunication9 45
BRM17 85
Shipshorecommunications2 10
Intrashipcommunication4 20
Unsafeleadership19 95
Inadequateleadership16 80
Plannedinappropriateoperations15
 75
Leadershipviolations11 55
Failuretocorrectknownproblem12 60
_______________________________________________
260
More than half mentioned that instructors
encourage the monitoring tasks and support the
development of situation awareness. While 60% felt
thattheyareencouragedtoworkasateam,butonly
1/3 referred that they are encouraged to take
decisions.
In relation to the type of sessions, their opinions
agreed
with the instructors, preferring sessions with
interruptions, for coaching and explanations of the
instructor, with him in the bridge. This was
highlighted in the openend questions, where they
reported that sessions were generally well planned
buttheinstructorsshouldbemoretimeinthebridge,
and they would like
to have much more time in
simulatedtraining.
When asked about the evaluation process, they
referredthatitshouldbemorefrequentandobjective.
80% of the participants considered that the
proficiencyoftheinstructorisgoodorverygood.
3.4 HFCASofNavyaccidents
Table 6 present the results of
the HFACS analysis,
onlyshowingthefirst3levels,thatweconsideredto
be more directly connected with the NTS. The
causality factors with higher relevance are the
decisionandperceptualerrors,nonuseormisuseof
instruments, BRM, inadequate leadership and
inappropriateplanning.
4 DISCUSSIONANDCONCLUSIONS
4.1 Non
technicalskills
From the results, we may see that five of the most
commonNTSwerealsoaddressedbytheinstructors
in the survey. Those are also closely related with
causality factor revealed from the analyses of the
accidentsreports.Subsequently, from the correlation
analysis of the literature review, surveys and
accidentsanalysis,fiveNTScategoriesweredefined:
Leadership,SituationalAwareness,Communications,
TeamworkandDecisionmaking.
Some may argue that the methodology should
have include a survey directed to the practitioners,
this options was discussed and considering the
extensivestudiesalreadymadebasedonprofessional
focusedgroup,weproposedto
followthatworkand
combiningitwithdifferentperspectives,eventhough
withsomeprofessionalopinionsthelecturers.
Notwithstandingtheadoptedselection,weshould
notice the fact that several factors identified in the
analysis of the accidents, have been already
addressed by the Navy, mostly throughout the
implementation of new procedures,
instructions,
changes in the training programs and qualification
processes.
4.2 Developingnontechnicalskillsinsimulators
Theexistingtrends foundovertheacademicyears,is
understandable, since in the first two years the
education is heavily focused on the technical skills,
whichcompelsforamoreindividualtraining.
Fromthe
results,wemaysaythatthereisastrong
engagement in strengthening the communication
skillswithintheteam,thelowestconcernwithradio
communicationsprocedurescouldberelatedtoother
courses,such as GMDSSand radiocommunications.
However,thelecturers,whenplanningthescenarios
andevaluatingthesessions,shouldbearin
mindthat
communication problems with other radio stations
arealsoacontributingfactorforaccidents.
Both surveys show that simulated training
demandsmoreinvolvementfromtheinstructors,ora
different type of pedagogical approach. These
students’andinstructors’perceptionsareinlinewith
what Emad (2010) and Magdy (2016) claim
over a
moreinvolvingroleofthelecturersintheteamunder
training.Studentsundertakingtasksandworkingasa
team in the simulator, seek for more cues and
guidancethantheonesprovidedbythewarningsand
alarmsrelated withthe effectiveness of their actions
anddecisions.
This involvement is
quite like what happens on
onboardtraining, sincethetraineeisdeeplyinvolved
intheshipteam.Thus,whileshipshaveoneortwo
trainees in the bridge team, simulators should have
theinstructorjoininginthetraineeteam,sotheymay
learnwithhim.
To increase student’s autonomy,
initiative was
made to allow them to use the NAVSIM for self
training and preparation of their assignments. This
decisionrequired the establishment of a certification
process to guarantee that the students had the
minimum qualification to operate the NAVSIM
control. A significant growth of the NAVSIM was
noticed,fromthe
3
rd
yearon,withnightimeuse.
4.3 Behaviouralmarkerssystemsinsimulators
NexttotheselectionoftheNTS,newiterationprocess
wasconductedtodesigntheappropriatebehavioural
markers. We start to review the existent marking
systems,butnowfocusedontheidentifiedskills.This
analysisalsoconsideredthe
descriptionspresentedin
the accident reports. Finally, we completed this
process with observation of simulated sessions, to
evaluatetheeffectivenessofthemarkingsystemand
thepracticalchallengesfacedbytheratter.Theactual
versionofmarkingsystemisshownintable7.
It soon become obvious that prior to the
implementation of any marking system it would be
necessary to prepare the instructors on its usage.
Furthermore, it would be necessary to collect their
feedback. A hot debate emerged among the
instructors and control staff, about the effectiveness
and usability of the form, which finished to
emphasised the necessity of
new functions in the
simulator that could support the evaluation of the
behavioural marker. Recently, Olaiya (2016) pointed
out the importance of establishing specification
requirements for the simulators, driven from the
maritimeuserneeds.
261
Table7.Behaviouralmarkingmodel
_______________________________________________
SkillBehaviouralmarker
_______________________________________________
LeadershipTakestheinitiative
Setintentionsandgoals
Establishandcontrolstandards
SituationalMonitorandreportschangesof
situations
AwarenessCollectsexternalinformation
Identifiespotentialdangerorproblems
CommunicationsSharesinformation
Keepsacontinuous,clearandeffective
flowofinformation
Promotesaconstructiveenvironment
forcommunications
Team
workConsidersalltheelementsoftheteam
Coordinatesthetasksoftheteam
Assessthecapabilitiesandcorrects
procedures
Decisionmaking Establishesalternativelinesofaction
Assessandverifiestheconsequencesof
thedecisionandactions
Considersandshareswiththeothers,
therisksofthe
differentlinesof
action
_______________________________________________
The navy operates two NAVSIMs, one at the
Naval Academy and the second at the Tactical
Training Centre. While the first serves the cadet
course program, the last is used to train the ship’s
navigation team. When, drawing a behavioural
markingsystemforbothfacilities,weconsiderthatit
must account
for the differences in the trainees and
training goals. The Naval Academy is more focused
in the understanding and initial development of the
NTSandthestudentssailingexperienceisonlybased
on the sea training voyages, whereas the Tactical
Training Centre is oriented for more contextualized
training to commissioned
practitioners, that have
beenworkingasateamforsometime.These teams,
whiletraining,aredeeplyfocusedonthefulfilmentof
the specific performance standards set by the Naval
FleetCommand.Thesefactorswerealreadyreflected
byDevitt&Holford(2010),whentheyconsideredthe
influences of the contextual differences
and the
organizationalrequirementsandneeds.
The availability of a marker system goes in line
withthesurvey’sresults,wereitwaspointedoutthe
necessityofmorerecurrentandobjectiveassessments.
Moreover,itopensnewlinesofresearchinthefield
of assessment in simulated training, since it is
necessarytodesigncomputerisedmetricstosupport
thebehaviouralmarkingmodel.Forinstance,through
the correlation of communications between
individuals and interactions with the workstations,
like RADAR or ECDIS. One would expect that,
following the appearance of a new contact in the
RADARdisplay,itwouldbereportedbytheRADAR
operator and that report should trigger new
interactions with the system, such as AIS or the
lookout’sdigitalPelorus.
The results support the fact that the envisage
changes in the instructor roles demands a larger
numberofpersonnel,asitisnotconceivabletostayat
thecontrolroomassessing
severalbridgesteams.
The Quality Management System of the Naval
Academy, includes a benchmarking process which
collects the assessments of several internal and
external stakeholders. One of the processes refers to
the selfassessment of competences and skills of the
youngofficer (one year aftergraduation) along with
the assessment of
captains and senior officers. This
informationwillbeextremelyvaluabletosupportthe
refinementofthisinitialmarkingsystem.
Previous comprehensive studies on behavioural
marking system are grounded in accident causal
factorsandanalysisoftheworkdomain,butwethink
that this model broads the existing work by
attempting
tobygrasptheeducationalfactors.
4.4 Conclusions
This paper presents the research undertaken to
provideabeha viouralmarkingschemetosupportthe
evaluation of nontechnical skill in NAVSIM. The
methodology considered previous studies and other
behavioural marking framework, but most
importantly it was shaped to the context of its
use:
educationalprogram,shipnavigationsimulatorsand
Portuguesenavy.
Very few results are found on the validation of
behavioural markers in simulated environment,
taking into consideration the distinctive needs and
requirements of educational and training program.
Current work sets for the preparation of workshops
for assessors to get feedback about
the markers.
Followupworkwillgoon,asaniterativeprocessto
further develop this first marking system. Within
three years we will have the first fiveyear program
cyclecomplete,alongwiththefeedbackreportsfrom
theseniorofficersoftheshipsandtheselfevaluations
oftheyoung
officers.
REFERENCES
Barnett,M.,Gatfield,D.,&Pekcan,C.(2006).Nontechnical
skills: the vital ingredient in world maritime
technology?InProceedingsoftheInternational Conference
on World Maritime Technology. London: Institute of
Marineengineering,Scienceandtechnology.
Bué,I.M.G.,Lopes,F.C.,&Semedo,Á.(2015).TheUse
of
thePortugueseNavalAcademyNavigationSimulatorin
Developing Team Leadership Skills. TransNav,
InternationalJournalonMarineNavigationandSafetyofSea
Transportation,(June),83–88.
Celik, M., & Cebi, S. (2009). Analytical HFACS for
investigating human errors in shipping accidents.
AccidentAnalysis&Prevention,41(1),66–75. 
Chauvin,C.,Clostermann,
J.P.,&Hoc,J. M.(2009).Impact
oftrainingprogramson decisionmaking and situation
awarenessoftrainee watchofficers.SafetyScience,47(9),
1222–1231.
Chauvin, C., Lardjane, S., Morel, G., Clostermann, J.P., &
Langard,B.(2013).Humanandorganisationalfactorsin
maritime accidents: Analysis of collisions at sea
using
theHFACS.AccidentAnalysisandPrevention,59,26–37.
Chen,