255
1 INTRODUCTION
Over the last decades, the maritime industry has
witnessed some rewarding outcomes from the
adoption of the Crew Resource Management
frameworkandtheuseofsimulators inthetrainingof
bridgeofficers.Still,therearedoubtsamongmaritime
professionalsabouttheeffectivenessofsuchtraining
in the improvement of ma
ritime safety (Barnett,
Gatfield,&Pekcan, 2006). In aneducational context,
the situation is more problematic since, among the
requirementsofdevelopingtechnicalskills,thereisa
needtobuildandevaluatethenon‐technicalskillsof
cadets with little experience in bridge team
management. For some time, the Portuguese Naval
Academy have been using the ship’s navigation
simulator (NAVSIM) to conduct tea
m leadership
model courses (Bué, Lopes, & Semedo, 2015). More
recently, namely after the latest NAVSIM’s upgrade
in 2004, there was an overall perception among
students and instructors that the complexity of the
trainingneededanewapproachforthedevelopment
andassessmentofnon‐technicalskills.Thus,astudy
tofurt
herassesshowthesimulatorisusedandhow
to improve the evaluation and assessment of non‐
technical skills during navigation training was
initiated.The adopted approach aimed to develop a
framework supported by previous research results
andcont
extualizingtheneedsofthePortuguesenavy,
not only through lecturer observations and
assessment of the students, but also by analysing
Navy accidents reports over the last two decades.
Thisresearchisanimportantcontributiontoincrease
the effectiveness of NAVSIM training sessions, by
Development of a Behavioural Marker System for
Rating Cadet’s Non-Technical Skills
V.P.daConceição
ChalmersUniversityofTechnology,Göteborg,Sweden
UniversityofLisbon,Lisbon,Portugal
PortugueseNavalSchool,Lisbon,Portugal
J
.C.Basso&F.C.Lopes
PortugueseNavalSchool,Lisbon,Portugal
J
.Dahlman
ChalmersUniversityofTechnology,Göteborg,Sweden
NavalPostgraduateSchool,Monterey,CA,USA
ABSTRACT: Despite the adoption of crew resource management training for bridge teams over the last
decades,thetrainingisstillheavilyfocussedontechnicalachievements.Inaneducationalcontext,thesituation
ismoreproblematical,sincewithrequirementofdevelopingthetechnicalskills,thereisa needtobuildand
evaluatethenon‐technicalskillsofcadetswithl
ittleexperienceinbridgeteammanagement.Inparallelwith
theapplicationofteamleadershipmodels,thePortugueseNavalAcademyconductedaresearchtoimprove
thedevelopmentandassessmentofnon‐technicalskillsinbridgesimulators.Thispa
perdescribesthemethod
usedto identify thekey non‐technical skillsrequired for navalcadetsand todevelopa behavioural marker
systemfortheirmeasurement.Aliteraturereviewofbehaviouralmarkersystemswassupplementedwithan
analysis of interviews conducted with students and simulator instructors. Additionally, furt
her analysis of
PortugueseNavyaccidentsreportswasmade,applyingtheHFACSframeworktoidentifythe relevantnon‐
technicalskillsinvolvedintheaccidents.Theresultingratingsystemcovers five skill categories (leadership,
situationalawareness,communication,teamworkanddecisionmaking),eachonewiththreeratingelements.
Theframeworkiscurrentlyunderevaluationtestsinbridgesimulatorssessions,withinaneducationalcontext.
http://www.transnav.eu
the International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 11
Number 2
June 2017
DOI:10.12716/1001.11.02.07
256
addinganobjectiveframeworkfortheassessmentof
non‐technical skills to the present technical skill
developmentprogram.
1.1 NAVSIMtrainingandeducation
Regardless of the general consensus over the
advantagesofnavigationsimulatorsineducationand
training,namelyinrelationtoincreasedsafety(Cross,
2012; Magdy Ali Elashkar, 2016),
their effectiveness
has been questioned. McCallum and Smith (2000)
evaluated simulators and provided a framework for
their assessment. From another perspective, Cross
(2003, 2007), Emad and Roth (2008) researched the
development of competence‐based training and
assessment in simulators. Other researchers have
shown that training programs display some
weaknesses in
the development of decision making
skills,whenaddressing themanagement ofcomplex
situations (Chauvin, Clostermann, & Hoc, 2009).
Other studies emphasize issues associated with the
effectiveness of simulation training, such as the
advantagesofexhaustivetrainingofunsafesituations
balancedwiththeriskofimprintingriskybehaviours
(Forsman,Hill,Dobbins,Brand,
&MacKinnon,2012).
Newinsightsonpedagogicalstrategiesarecalling
for a more innovative role of the lecturer. Students
need to learn by experiencing their working
environment and performing tasks with experts.
Thus, instructor shall gradually leave space for the
studentsandcoachthemintheirunderstandingofthe
undergoing
processes (Emad, 2010; Magdy Ali
Elashkar,2016).Arecentstudy,whichwasbasedona
survey to assess training in simulators at the
MerchantMarineAcademyofAthens,Tsoukalasetal.
(2015) concluded that adjustments in the course
programs and methodologies were necessary. They
specificallysuggested modificationtomethodologies
used
in the evaluation of trainees and the teaching‐
designforproblem‐solving.
1.2 Non‐technicalskills(NTS)
Thisterm,commonlyusedinsafetycriticaldomains,
hasbeendefinedbyFlin,O’Connor&Crichton(2008)
as the cognitive, social and personal resource skills
that complement technical skills, and contribute to
safeand
efficienttaskperformance.Still,thereareno
straight guidelines to classify NTS, since their
relevanceandappearancearestronglycoupledtothe
workingcontext.Yet,NTShavetwodissimilareffects,
since good skills reduce the possibility of human
errorsand,ontheotherside,poorskillincreasesthe
chance of
error occurrence or extend their
consequences. Regardless of the research made to
addressthedevelopmentofNTSinthemaritimefield
(Barnett et al., 2006; Devitt & Holford, 2010;
Hetherington, Flin, & Mearns, 2006; Long, 2011;
MagdyAliElashkar,2016;Saeed, 2010),Hayward&
Lowe (2010) identified significant shortfalls in
maritime
training,whencomparedtootherdomains.
Basedontheexistingfindings,wemaycommonly
agree with the following list of non‐technical skills,
frequentlyrecognisedwhenaddressingsafetyissues:
Situationalawareness;
Assertiveness, ability to criticize and to be
criticized;
Decisionmaking;
ProblemSolving;
Leadership,ability
touseauthority
Managingstress;
Workload management, Multi‐tasking and
selectiveattention;
Teamwork,teambuildingandcooperation;
Communication,Inter‐teamcommunication;
Copingwithfatigue;
Timemanagement,planningandpreparing
Situationmonitoring
Anticipationoffuturestates;
Settingpriorities;
CrisisManagement.
1.3
Behaviouralmarkerssystems
Followingthefirstbeha viouralmarkingsystemmade
bytheUniversityofTexas(UTBehaviouralMarker),
created for pilots, others have been developed for
severalworkingdomains,suchasthe:
Pilots’non‐technicalskills(NOTECHS)inaviation;
Non‐TechnicalSkillsforSurgeons(NOTSS);
Anaesthetists’Non‐
TechnicalSkills(ANTS);
TraumaNon‐TechnicalSkills(T‐NOTECHS);
Well Operations Crew Resource Management
(WOCRM)inoffshorewellcontrol.
In the maritime industry, few developments are
foundwithafirmemploymentofamarkingscheme
for the Bridge Resource Management (BRM)
framework. After undertaking a study for the
development
ofresourcemanagementandleadership
behavioural markers for shipping, Devitt & Holford
(2010) pointed out some factors that have to be
considered when establishing behavioural markers,
namely:
Culturalimplications;
Shipboardandshoresidecontextualdifferences;
Specificorganisationalneedsandrequirements;
TrainingofBehaviouralmarkerassessors.
Their
behavioural markers were yet to be
validated in simulated environments. In another
research, Long (2011) developed a marking
framework (Nontechnical Skills for Officers of the
Deck‐NTSOD) to be used for the evaluation of
Officers of deck in the US Navy. It comprises four
categories (leadership, decision‐making,
communications and situation
awareness), with ten
rating elements, on a scale of 1 to 4. This prototype
wasinitiallydesignedtobeusedonboard,however,
the author considered the need for further work,
namelyitsimplementationinsimulationtraining for
moretestsandrefinementsoftheratingsystem.
1.4 TheHFACSFramework
Grounded in Reason’s “Swiss cheese” model of
accident causation, the Human Factor Analysis and
Classification System (HFACS) was designed to
provide a classification scheme and used for the
systematicidentificationofaccidentcausationfactors
(Shappell&Wiegmann,2000).TheHFACScomprises
fourhierarchicallevelsof failures, from UnsafeActs
atthe
front‐enduptotheOrganizationalInfluenceat
257
the back‐end, passing by Preconditions for Unsafe
Acts and Unsafe Supervision. After its initial
application in military aviation accidents analysis,
severalderivedmodelshavebeendesignedforother
domains such as the rail, mining and medicine
industry. Adaptation to the maritime industry, has
also been undertaken by several authors
(Celik &
Cebi,2009;Chauvin,Lardjane,Morel,Clostermann,&
Langard, 2013; Chen et al., 2013; Rothblum et al.,
2002).TheHFACS‐MAmodeldesignedbyChenetal
(2013)includesanadditionallevel(Otherfactors),set
above the organizational level, to address the
legislation and political factors. To perform the
analysisofcollisionsatsea,Chauvinetal(2013)also
made some adaptions to consider factors such as
communicationandBRM.
Additionally, it was found that, despite the
positive results on inter‐ and intra‐rater reliability
tests of the HFACS framework, it still needs
improvements in its implementation (Ergai et al.,
2016).
2 METHODOLOGY
TheproposedmethodologyaimedtodevelopanNTS
behaviourmarkingsystemtobeusedintheNAVSIM
of the Portuguese Naval Academy. Technical skills
were not consideredin this research,since there are
alreadyobjectivelyevaluatedthru thecomplianceof
proceduresandtheeffectivenessofthedecisions
and
actions. Thus, we considered a four‐step process to
design a framework that should be coupled to the
navy needs and context. The first step was a
comprehensive literature review to create a list of
NTSthathavebeeneffectivelyusedinsafetycritical
domain,whichwouldbeusefulfor
theOfficerofthe
Watch (OOW). This was followed by focus group
questionnaires to NAVSIM instructors, aiming at a
bettercategorizationoftheNTS.Thethirdstepaimed
at the contextualization of the navy needs, thus we
performed an analysis of accident reports involving
navy vessels, complemented by a second
set of
questionnaires involving NAVSIM trainees. The last
step consisted of the qualitative analysis of the
collected data and design of the NTS behaviour
markingsystem.
2.1 Questionnaires
All questionnaires were completed over the first
semesterof2016.Thefirstquestionnairewasgivento
lecturers and instructors who use or have
recently
usedbothNAVSIMfromtheNavalAcademyandthe
Tactical Training Centre, in training and teaching
sessions, making a total of 10 participants. The
questionnairewascomposedbythreeparts.Thefirst
onecoveredthedemographicdataincludinggender,
age, years of experience and attended courses. The
secondpart,
aimedtheevaluationoftheeducational
programinrelationtotheuseoftheNAVSIMandthe
characterization of the simulated sessions (9
questions).Thelastpartfocusedontheassessmentof
howtheNTSaredevelopedandwhichareconsidered
themostrelevant(24questions).
The second questionnaire was directed
towards
student perceptions over the educational program
around the use of the simulator. The questionnaire
waspresented to all students, except those from the
1stacademicyear,fromallgraduate degreeprogram
that use the NAVSIM, totalling 139 participants,
representing90%ofthepopulation(seeTable1).The
navygraduate
degreeprogramme(63%ofthecadets)
has more courses with modules conducted in the
NAVSIM. This questionnaire also comprises three
parts. The demographic part collected age, gender,
academicyearandcourseprogram. The second part
aimed the evaluation of the educational program in
regardstotheirtrainingasOOW(10questions).
The
third part pointed the student perception over the
development of their technical and non‐technical
skillsinsimulatedtraining(22questions).
Table1.Studentparticipants
_______________________________________________
Academicyear Total
Graduateprog. 1
st
2
nd
3
rd
4
th
5
th
N %
_______________________________________________
Navaladministration 4 4 5 4 17 17100%
Weapons,elect.Eng. 5 2 3 3 13 13100%
MechanicalEng. 7 6 3 5 21 21100%
Marine0 0 2 1 3 0 0%
Navy28 28 24 18 98 8890%
_______________________________________________
Total44 40 37 33 154 13990%
_______________________________________________
Participants(N) 42 33 35 29 139
95% 83% 95% 88% 90%
_______________________________________________
Both questionnaires involved mostly close‐end
questions of multiple‐choice and ordering. Seven
open‐ended questions were included, three for the
students and four for the instructors. Before the
implementation, a pre‐test was performed on 3
individuals, to validate the adequacy of the
questionnaires.Theanalysiswasperformedwiththe
SPSSv.20software.
2.2 Accidentsanalysis
The analysis of the accidents was made with the
HFACS‐ Coll framework proposed by Chauvin et al
(2013), with minor changes to adapt it to the cases
under analysis, namely to account the groundings.
The accident reports used in this analysis were
obtained from
investigation reports set up by
Portuguese Navy, on accidents involving navy
vessels. All the navigation accidents cases were
considered, totalling 20 cases. They involve 8
collisions, 5 groundings and 7 collision in mooring
manoeuvres, from 1995 to 2016. The accident events
occurredwithinthefollowingcontext:
Oceannavigation2
Coastalnavigation4
Restrictedwaternavigation14
Closemanoeuvres6
Trainingprograms4
Thecodingprocesswasseparatelycarriedoutby
two analysts, one of them an experienced mariner,
usingqualitativeanalysissoftware(NVIVO).
Two iteration were made and the inter‐rater
reliability was assessed by measuring
percentage
agreementsandKappacoefficients.Asitisshownin
Figure 1, the second iterationsignificantly increased
258
the degree of agreement. The adjustments of the
HFACS framework were performed during the
codingprocess,afterdiscussionsbetweenthecoders.
Figure1.Kappadistributiondiagramofthecodingprocess
3 RESULTS
3.1 Non‐technicalskills
The NTS were mostly retrieved from existing
behavioural marker systems, the UT Behavioural
Marker(Klampferetal.,2001),NOTECHS(Flinetal.,
2003),ANTS(Fletcheretal.,2004),NOTSS(Flin,Yule,
Paterson‐Brown, Rowley, & Maran, 2006), and
NTSOD (Long, 2011), we also considered skills
identifiedby Flin etal. (2008)and Devitt & Holford
(2010). The literature review resulted in a list of 13
categories(seetable3),whereagreateremphasiswas
found in skills like communication, leadership;
situational awareness, decision‐making and team
work. The criteria used for choice and descriptions
were
guided by the necessity to: 1) cover all the
critical behaviour; 2) avoid overlap between
categories;3)useterminologyeasilyperceivedwithin
the organization; 4) be easily accessible and
observable;and5)bemeasurableinsomedimensions,
evenqualitatively.
3.2 Lecturersurvey
The participants, were all navy officers, with an
average
of 6,5 years of experience in NAVSIM
training,90%male,twoyoungerthan35yearsand4
olderthan 45,none withany documentedsimulator
training education. The majority (80%) considered
thatthetrainingintheNAVSIMcouldbeimproved.
The participants use the NAVSIM for four different
coursesprograms,
asitcanbeseeninthetable2.
Table3presentstheNTSthatwereconsideredas
themostrelevant,byaskingthe participanttoselect
fiveofthe13categoriesofNTS.
Twoquestionsweredirectedfortheperceptionof
thebestsimulatedsession,onetoknowwhichtype
of
session is usually used and a second to identify the
recommended type for the development of non‐
technicalskills.LookingattheresultsinTable4,we
mayseethattherecommendedtypeofsessionisthe
one that flavour a greater integration of the lecturer
withintheteam
activity.
Table2.CourseprogramsconductedintheNAVSIM
_______________________________________________
Whichcourseprogramdoyoumanage?Y N
_______________________________________________
Tacticalnavigation/navaloperations 6 2
Seamanship3 5
Leadership/organizationalbehaviour 1 7
Navigation7 1
_______________________________________________
Table3.IdentificationofthemostrelevantNTS
_______________________________________________
NTSN Order
_______________________________________________
Decisionmaking9 1
Situationalawareness8 2
Leadership6 3
Taskplanningandmanagement6 3
Monitoring,vigilance6 3
Teamwork5 4
Communication5 4
Assertiveness3 5
Managingstress1 6
Perception,intuition1 6
Copingwithfatigue1 6
Energy,mentalalertness0 12
Workloadmanagement
0 12
_______________________________________________
Table5presentsthesomeresultsofthelecturers’
survey. While the first set of questions attempts to
characterize the context of the training session, the
secondtriestogivesussomeunderstandingoverthe
participant perception in the development of non‐
technicalskills.
3.3 Studentssurvey
The participants of this
survey were all students of
the Naval Academy, being between 18 and27 years
old, 41.7% being between 21 and 22, 79% male.
Bearing in mind that 97,1% have already performed
tasks in a bridge team, 92.8% considered that
simulation training could be improved and almost
halfofthemconsidered
longtermsessionsasagood
waytoincreasetheir skills.Threedifferenttestswere
performed with the collected data, Kolmogorov‐
Smirnov, Kruskal‐Wallis and Mann‐Whitney.
Kolmogorov‐Smirnovtestshownthatthedatadoesn’t
have a normal distribution, therefor we choose to
proceedwithnon‐parametrictests.
Table4.ConfigurationtypesforthesimulatedsessionsintheNAVSIM(Y‐yes,N‐no/scale0to5,where0standsforno
answer,1fordisagreeand5fortotallyagree)
__________________________________________________________________________________________________
WhichtypeofsessionbestsuitthedevelopmentoftechnicalandNTS?0 1 2 3 4 5
x
__________________________________________________________________________________________________
Playingthescenario,withnointerruption/instructorinthecontrol roommonitoring6 11,1
Playingthescenario,withnointerruption/activepresenceoftheinstructorinthebridge1 2 1 3 3,9
Playingthescenario,withinterruptions,forcoachingandexplanationsoftheinstructor,inthebridge
3 4 4,6
Playingthescenario,withnointerruption/activepresenceoftheinstructorinthecontrolroom4 3 3,4
__________________________________________________________________________________________________
259
Table5.InstructorsperceptionontheuseoftheNAVSIMasaneducationaltool(scale0to5,where0standsfornoanswer,
1fordisagree/never/verybad,and5fortotallyagree/always/verygood)
__________________________________________________________________________________________________
0 1 2 3 4 5
x
__________________________________________________________________________________________________
PartII
ArethenumberofinstructorsinNAVSIMsessiontrainingenough?2 1 2 4 1 3,3
WhatisthequalityoftheNAVSIMfacilities?2 1 6 1 4,0
Arethenumberoftrainingsessionssufficient?2 5 3 0 3,4
ThetotalnumberofNAVSIMtraininghoursfor
Navygraduatedegreeprogramissufficient 4 5 1 3,2
ThetotalnumberofNAVSIMtraininghoursforMarine, EngineersandAdministrationgraduate 6 3 12,3
degreeprogramissufficient
Importanceoflongtrainingsessions(>12hours)2 1 0 1 5 1 3,6
PartIII
Doyouagree
thattraininginNAVSIMisrelevantforthedevelopmentofbothtechnicalandNTS24 4 4,5
ofthefutureOOW
HowimportantisthedevelopmentofNTSintheNAVSIM?1 6 3 4,2
Doyouperformbriefingsanddebriefings?2 4 4 4,2
DoyouencourageOOW
traineestoassignroles/tasksandclarifytheresponsibilitiestotheremaining 1 4 5 4,4
membersoftheirteam?
Doyouevaluatethetraineesindividuallyaftereachsession?2 6 2 4,0
Doyouevaluatethetraineesasateamaftereachsession?1 6 3 4,2
Do
youencourageOOWtraineestomonitorthetasksandsustainacommonsituationawarenesswithin 1 5 4 4,3
theteam?
Doyouencourageteamwork?22 6 4,8
Doyouencouragedecisionmakinginsafetycriticaloruncomfortablesituations?27 1 4,1
Doyouencouragetheuse
offormalcommunicationformswithintheteam?28 5,0
Doyouevaluatetheradiocommunicationprocedureswithothershipsandshorestations?2 1 0 3 4 4,3
Istheindividualtrainingsession,precededbyplanningwork?2 1 7 3,9
Isthegrouptrainingsession,precededbyplanning
work?2 1 1 4 2 3,9
Justbeforethesession,doyoubriefthestudentswiththesessiongoals,planandevaluation
methodology?1 1 4 4 4,3
Justafterthesession,doyoudebriefthestudentswithananalysisofthesession,lessonslearned 13 6 4,7
andgoodpractices?
DoyouconsiderthatNAVSIMtraininghelpsthedevelopmentofleadershipskills?24 4 4,5
DoyouencouragetheOOWtraineetosetprioritiesinaccordancewiththesituations3 7 4,7
__________________________________________________________________________________________________
The results of the Kruskal‐Wallis test gave high
degree of significance in 7 over 13 questionsfor the
Academics Years, and 9 over 13 questions for the
graduate degree programs. The Mann‐Whitney tests
(multiplecomparisons) weremade forthe questions
withhighdegreeofsignificance.
Theresults
expressacleartrendfromthe2
nd
tothe
5
th
years, namely in regards to the increasing
instructor’s engagement to follow communication
standards, providing team working instructions and
decision‐making in stressful situations. Differences
were also found between the graduate programs,
especiallybetweentheNavygraduatesandgraduates
fromtheotherprograms.Thiscouldbeconnectedto
the fact that
until the end of the 2
nd
year they all
attendthesamecourses,laterontheyfollowdifferent
curriculum,withtheNavygraduatesattendingmore
courses and more demanding tasks in the SIMNAV
sessions. This is also reflected in the instructor
behaviour and, on the requirements, he/she sets for
thetrainees.
In general, more than half
of the participants
considered that they are motivated to manage tasks
andtosetresponsibilitieswithintheteam.
Additionally,about2/3reportedthattheyusually
have briefings and debriefings of the sessions. 2/3
reports that the NAVSIM sessions helped them to
developtheir leadership skills.They considered that
theyaremostly
evaluatedasateamintheNAVSIM
sessionsratherthanasindividuals.
Table6.SummaryresultsofHFACSanalysis(levels1,2and
3)
_______________________________________________
HFACSFactorsN %
_______________________________________________
Unsafeacts20 100
Errors20 100
Skill‐basederrors11 55
Decisionerrors15 75
Perceptualerrors17 85
Violations9 45
Routineviolations2 10
Exceptionalviolations8 40
Preconditionsforunsafeacts20 100
Environmentalfactors20 100
Physicalenvironment14 70
Hydro
‐METOCphenomena10 50
Visibilityorlighting5 25
Technologicalenvironment17 85
Shipbuilding‐bridgedesign2 10
Radar,ECDIS,NAVAIDSfailure6 30
Non‐useormisuseofinstruments17 85
Conditionsofoperators12 60
Adversementalstate12 60
AffectedSA3
15
Attentiondeficit‐workload12 60
Complacency1 5
Personnelfactors19 95
SRM19 95
Inter‐shipcommunication9 45
BRM17 85
Ship‐shorecommunications2 10
Intra‐shipcommunication4 20
Unsafeleadership19 95
Inadequateleadership16 80
Plannedinappropriateoperations15
75
Leadershipviolations11 55
Failuretocorrectknownproblem12 60
_______________________________________________
260
More than half mentioned that instructors
encourage the monitoring tasks and support the
development of situation awareness. While 60% felt
thattheyareencouragedtoworkasateam,butonly
1/3 referred that they are encouraged to take
decisions.
In relation to the type of sessions, their opinions
agreed
with the instructors, preferring sessions with
interruptions, for coaching and explanations of the
instructor, with him in the bridge. This was
highlighted in the open‐end questions, where they
reported that sessions were generally well planned
buttheinstructorsshouldbemoretimeinthebridge,
and they would like
to have much more time in
simulatedtraining.
When asked about the evaluation process, they
referredthatitshouldbemorefrequentandobjective.
80% of the participants considered that the
proficiencyoftheinstructorisgoodorverygood.
3.4 HFCASofNavyaccidents
Table 6 present the results of
the HFACS analysis,
onlyshowingthefirst3levels,thatweconsideredto
be more directly connected with the NTS. The
causality factors with higher relevance are the
decisionandperceptualerrors,non‐useormisuseof
instruments, BRM, inadequate leadership and
inappropriateplanning.
4 DISCUSSIONANDCONCLUSIONS
4.1 Non‐
technicalskills
From the results, we may see that five of the most
commonNTSwerealsoaddressedbytheinstructors
in the survey. Those are also closely related with
causality factor revealed from the analyses of the
accidentsreports.Subsequently, from the correlation
analysis of the literature review, surveys and
accidentsanalysis,fiveNTScategoriesweredefined:
Leadership,SituationalAwareness,Communications,
TeamworkandDecisionmaking.
Some may argue that the methodology should
have include a survey directed to the practitioners,
this options was discussed and considering the
extensivestudiesalreadymadebasedonprofessional
focusedgroup,weproposedto
followthatworkand
combiningitwithdifferentperspectives,eventhough
withsomeprofessionalopinions–thelecturers.
Notwithstandingtheadoptedselection,weshould
notice the fact that several factors identified in the
analysis of the accidents, have been already
addressed by the Navy, mostly throughout the
implementation of new procedures,
instructions,
changes in the training programs and qualification
processes.
4.2 Developingnon‐technicalskillsinsimulators
Theexistingtrends foundovertheacademicyears,is
understandable, since in the first two years the
education is heavily focused on the technical skills,
whichcompelsforamoreindividualtraining.
Fromthe
results,wemaysaythatthereisastrong
engagement in strengthening the communication
skillswithintheteam,thelowestconcernwithradio
communicationsprocedurescouldberelatedtoother
courses,such as GMDSSand radiocommunications.
However,thelecturers,whenplanningthescenarios
andevaluatingthesessions,shouldbearin
mindthat
communication problems with other radio stations
arealsoacontributingfactorforaccidents.
Both surveys show that simulated training
demandsmoreinvolvementfromtheinstructors,ora
different type of pedagogical approach. These
students’andinstructors’perceptionsareinlinewith
what Emad (2010) and Magdy (2016) claim
over a
moreinvolvingroleofthelecturersintheteamunder
training.Studentsundertakingtasksandworkingasa
team in the simulator, seek for more cues and
guidancethantheonesprovidedbythewarningsand
alarmsrelated withthe effectiveness of their actions
anddecisions.
This involvement is
quite like what happens on
on‐boardtraining, sincethetraineeisdeeplyinvolved
intheshipteam.Thus,whileshipshaveoneortwo
trainees in the bridge team, simulators should have
theinstructorjoininginthetraineeteam,sotheymay
learnwithhim.
To increase student’s autonomy,
initiative was
made to allow them to use the NAVSIM for self‐
training and preparation of their assignments. This
decisionrequired the establishment of a certification
process to guarantee that the students had the
minimum qualification to operate the NAVSIM
control. A significant growth of the NAVSIM was
noticed,fromthe
3
rd
yearon,withnightimeuse.
4.3 Behaviouralmarkerssystemsinsimulators
NexttotheselectionoftheNTS,newiterationprocess
wasconductedtodesigntheappropriatebehavioural
markers. We start to review the existent marking
systems,butnowfocusedontheidentifiedskills.This
analysisalsoconsideredthe
descriptionspresentedin
the accident reports. Finally, we completed this
process with observation of simulated sessions, to
evaluatetheeffectivenessofthemarkingsystemand
thepracticalchallengesfacedbytheratter.Theactual
versionofmarkingsystemisshownintable7.
It soon become obvious that prior to the
implementation of any marking system it would be
necessary to prepare the instructors on its usage.
Furthermore, it would be necessary to collect their
feedback. A hot debate emerged among the
instructors and control staff, about the effectiveness
and usability of the form, which finished to
emphasised the necessity of
new functions in the
simulator that could support the evaluation of the
behavioural marker. Recently, Olaiya (2016) pointed
out the importance of establishing specification
requirements for the simulators, driven from the
maritimeuserneeds.
261
Table7.Behaviouralmarkingmodel
_______________________________________________
SkillBehaviouralmarker
_______________________________________________
LeadershipTakestheinitiative
Setintentionsandgoals
Establishandcontrolstandards
SituationalMonitorandreportschangesof
situations
AwarenessCollectsexternalinformation
Identifiespotentialdangerorproblems
Communications Sharesinformation
Keepsacontinuous,clearandeffective
flowofinformation
Promotesaconstructiveenvironment
forcommunications
Team
workConsidersalltheelementsoftheteam
Coordinatesthetasksoftheteam
Assessthecapabilitiesandcorrects
procedures
Decisionmaking Establishesalternativelinesofaction
Assessandverifiestheconsequencesof
thedecisionandactions
Considersandshareswiththeothers,
therisksofthe
differentlinesof
action
_______________________________________________
The navy operates two NAVSIMs, one at the
Naval Academy and the second at the Tactical
Training Centre. While the first serves the cadet
course program, the last is used to train the ship’s
navigation team. When, drawing a behavioural
markingsystemforbothfacilities,weconsiderthatit
must account
for the differences in the trainees and
training goals. The Naval Academy is more focused
in the understanding and initial development of the
NTSandthestudentssailingexperienceisonlybased
on the sea training voyages, whereas the Tactical
Training Centre is oriented for more contextualized
training to commissioned
practitioners, that have
beenworkingasateamforsometime.These teams,
whiletraining,aredeeplyfocusedonthefulfilmentof
the specific performance standards set by the Naval
FleetCommand.Thesefactorswerealreadyreflected
byDevitt&Holford(2010),whentheyconsideredthe
influences of the contextual differences
and the
organizationalrequirementsandneeds.
The availability of a marker system goes in line
withthesurvey’sresults,wereitwaspointedoutthe
necessityofmorerecurrentandobjectiveassessments.
Moreover,itopensnewlinesofresearchinthefield
of assessment in simulated training, since it is
necessarytodesigncomputerisedmetricstosupport
thebehaviouralmarkingmodel.Forinstance,through
the correlation of communications between
individuals and interactions with the workstations,
like RADAR or ECDIS. One would expect that,
following the appearance of a new contact in the
RADARdisplay,itwouldbereportedbytheRADAR
operator and that report should trigger new
interactions with the system, such as AIS or the
lookout’sdigitalPelorus.
The results support the fact that the envisage
changes in the instructor roles demands a larger
numberofpersonnel,asitisnotconceivabletostayat
thecontrolroomassessing
severalbridgesteams.
The Quality Management System of the Naval
Academy, includes a benchmarking process which
collects the assessments of several internal and
external stakeholders. One of the processes refers to
the self‐assessment of competences and skills of the
youngofficer (one year aftergraduation) along with
the assessment of
captains and senior officers. This
informationwillbeextremelyvaluabletosupportthe
refinementofthisinitialmarkingsystem.
Previous comprehensive studies on behavioural
marking system are grounded in accident causal
factorsandanalysisoftheworkdomain,butwethink
that this model broads the existing work by
attempting
tobygrasptheeducationalfactors.
4.4 Conclusions
This paper presents the research undertaken to
provideabeha viouralmarkingschemetosupportthe
evaluation of non‐technical skill in NAVSIM. The
methodology considered previous studies and other
behavioural marking framework, but most
importantly it was shaped to the context of its
use:
educationalprogram,shipnavigationsimulatorsand
Portuguesenavy.
Very few results are found on the validation of
behavioural markers in simulated environment,
taking into consideration the distinctive needs and
requirements of educational and training program.
Current work sets for the preparation of workshops
for assessors to get feedback about
the markers.
Followupworkwillgoon,asaniterativeprocessto
further develop this first marking system. Within
three years we will have the first five‐year program
cyclecomplete,alongwiththefeedbackreportsfrom
theseniorofficersoftheshipsandtheself‐evaluations
oftheyoung
officers.
REFERENCES
Barnett,M.,Gatfield,D.,&Pekcan,C.(2006).Non‐technical
skills: the vital ingredient in world maritime
technology ?InProceedingsoftheInternational Conference
on World Maritime Technology. London: Institute of
Marineengineering,Scienceandtechnology.
Bué,I.M.G.,Lopes,F.C.,&Semedo,Á.(2015).TheUse
of
thePortugueseNavalAcademyNavigationSimulatorin
Developing Team Leadership Skills. TransNav,
InternationalJournalonMarineNavigationandSafetyofSea
Transportation,(June),83–88.
Celik, M., & Cebi, S. (2009). Analytical HFACS for
investigating human errors in shipping accidents.
AccidentAnalysis&Prevention,41(1),66–75.
Chauvin,C.,Clostermann,
J.‐P.,&Hoc,J. ‐M.(2009).Impact
oftrainingprogramson decision‐making and situation
awarenessoftrainee watchofficers.SafetyScience,47(9),
1222–1231.
Chauvin, C., Lardjane, S., Morel, G., Clostermann, J.‐P., &
Langard,B.(2013).Humanandorganisationalfactorsin
maritime accidents: Analysis of collisions at sea
using
theHFACS.AccidentAnalysisandPrevention,59,26–37.
Chen,S.T.,Wall,A.,Davies,P.,Yang,Z.,Wang,J., &Chou,
Y. H. (2013). A Human and Organisational Factors
(HOFs) analysis method for marine casualties using
HFACS‐Maritime Accidents (HFACS‐MA). Safety
Science,60,105–114.
Cross, S.
J. (2003). Enhancing Competence Based Training
and Assessment for Marine Engineers through the
Realism of Virtual Presentation. In International
ConferenceonMarineSimulationandShipManeuverability
MARSIM’03(pp.1–7).Kanazawa,Japan.
262
Cross, S. J. (2007). Competence Based Learning and
Evaluation: Developments and Non‐Developments in
MET.InIFSMA33rdAnnualGeneralAssembly.Antwerp,
Belgium.
Cross, S. J. (2012). Aspects of simulations in MET‐
improvingshippingsafetyandeconomy.InInternational
MaritimeLecturersAssociation20thConference.
Devitt,K. R.,& Holford, S.
D. (2010).The development of
resource management and leadership behavioural
markers for the Merchant Navy. In Proceedings of the
Maritime Human Resource Solutions Conference. St John’s
Newfoundland, Canada. Retrieved from
http://ssudl.solent.ac.uk/1426/
Emad, G. R. (2010). Rethinking Vocational Education and
training:HaulinginfromMaritimeEducation. InR.V.
Nata(Ed.),
Progress in Education(Vol. 22, pp. 125–140).
NovaSciencePublishers.
Emad,G.R.,&Roth,W.‐M. M.(2008).Contradictionsinthe
practicesoftrainingforandassessmentofcompetency:
Acase studyfromthe maritime domain. Educationand
Training,50(3),260–272.
Ergai, A., Cohen, T., Sharp, J., Wiegmann,
D.,
Gramopadhye,A.,&Shappell, S.(2016).Assessmentof
the Human Factors Analysis and Classification System
(HFACS): Intra‐rater and inter‐rater reliability. Safety
Science,82(May),393–398.
Fletcher,G.,Flin,R.,McGeorge,P.,Glavin,R., Maran,N.,&
Patey,R.(2004).Ratingnon‐technicalskills:developing
a behavioural marker system
for use in anaesthesia.
Cognition,Technology&Work,6(3),165–171.
Flin, R., Martin, L., Goeters, K.‐M., Hörmann, H.‐J.,
Amalberti, R., Valot, C., & Nijhuis, H. (2003).
Development of the NOTECHS (non‐technical skills)
system for assessing pilots’ CRM skills. Human Factors
andAerospaceSafety,3(2),95–117.
Flin, R., O’Connor, P., & Crichton, M. (2008). Safety at the
sharp end: a guide to non‐technical skills. Aldershot
England:Ashgate.
Flin, R., Yule, S., Paterson‐Brown, S., Rowley, D. I., &
Maran, N. (2006). The Non‐Technical Skills for Surgeons
(NOTSS) System Handbook v1.2: Structuring observation,
ratingand
feedbackofsurgeons’behavioursintheoperating
theatre. University of Aberdeen. Retrieved from
http://www.abdn.ac.uk/iprc/notss
Forsman,F.,Hill,J.,Dobbins,T.,Brand,T.,&MacKinnon,S.
N.(2012).ExploitingSimulationtoEnhanceOperational
Effectiveness and Interoperability in Littoral Maritime
Operations. In STO Modelling and Simulation Group
Conference.
Hayward, B. J., &
Lowe, A. R. (2010). Chapter 12 – The
Migration of Crew Resource Management Training. In
CrewResourceManagement(pp.317–342).
Hetherington, C., Flin, R., & Mearns, K. (2006). Safety in
shipping: The human element. Journal of Safety Research
(Vol.37).London.
Klampfer,B.,Flin,R.,Helmreich,R.L.,Hausler, R.,
Sexton,
B., Fletcher, G., … Dieckmann, P. (2001). Enhancing
Performance in High Risk Environments,
Recommendations for the Use of BehaviouralMarkers.
In Behavioral Markers Workshop, GIHRE. Zürich,
Switzerland.: Gottlieb Daimler and Karl Benz
Foundation.
Long, W. M. (2011). The development of a prototype
behavioral marker system for US Navy officers
of the
deck.SafetyScience,49(10),1381–1387.
Magdy Ali Elashkar. (2016). the Use of Simulation
TechniquesintheDevelopmentofNon‐TechnicalSkills
for Marine Officers. International Journal of General
EngineeringandTechnology(IJGET),5(5),19–26.
McCallum, M. C., & Smith, M. W. (2000). Evaluating
Simulators for use
in assessment of mariner
proficiencies.In Proceedings of MARSIM 2000 (pp. 1–8).
Orlando,Florida.
Olaiya, J. Oi. (2016). Marine Simulators: Technical And
PerformanceSpecifications‐AParadoxicalParallelism ?
InIAMU.
Rothblum, A., Wheal, D., Withington, S., Shappel, S. A.,
Wiegmann,D.A.,Boehm,W.,&Chaderjian,M.(2002).
Improving incident
investigation through inclusion of
humanfactors.UnitedStatesDepartmentofTransportation
‐‐ Publications & Papers, (32). Retrieved from
http://digitalcommons.unl.edu/usdot/32/
Saeed,F.(2010).TheImportanceofNon‐TechnicalSkillsin
the Maritime Education. In International Association of
MaritimeUniversitiesAGA11(pp.103–114).Busan,South
Korea.
Shappell, S., & Wiegmann, D. (2000). The
Human Factors
Analysis and Classification System – HFACS. Federal
Aviation Administration Technical Report DOT/FAA/AM‐
00/7. Springfield, Virginia: National Technical
InformationService.
Tsoukalas, V. D., Papachristos, D. A., Stefanakou, A. A.,
Tsoumas, N. K., & Nikitakos, N. (2015). Questionnaire
assessment of training in a marine simulator. WMU
JournalofMaritime
Affairs,14(2),293–312..
