229
1 INTRODUCTION
The complexity of maritime navigation is
continuously increasing. The growth in the world
fleet size and the changing characteristics of vessels
increasestheneedforskillednavigators.Assessment
of navigation skills is therefore an ever more
importantissue.However,themethodsforassessing
the performance of human operators have mostly
remained unchanged. Currently, the performa
nce
assessmentduringtraininginsimulatorsisdoneasa
subjectiveassessmentbydomainexperts.
Research suggests limitations to the reliability of
subjective assessments.In principle, all fair tests are
designed to differentiate between those that have a
trait (e.g. being competent) and those tha
t lack the
trait (e.g. those that are not competent). However,
since a human subject matter expert (SME) is the
assessment tool of the traineesʹ performance within
thesimulator,theassessmentisaffectedbythebiases
thatfollowsfromsuchasubjectiveevaluation(Manca
et al., 2012; Nazir & Manca, 2015). A bia
s in
assessment involves the tendency to systematically
shift the evaluation away from a consistent score
(Kahneman,2011;Allen&Yen,1979).Thepresenceof
biases lowers the reliability of the assessment
(Cronbach et al., 1972; Freedman, 2009). Biases can
arise from the fact that humans are not perfectly
rational decision‐ma
kers (Simon, 1979). Humans
show non‐optimal decision making and judgement
eveninsituationswhereallnecessaryinformationis
available to make an optimal decision (Kahneman,
Slovic&Tversky,1982).Also,humanassessmentsdo
nothaveperfecttest‐retestreliabilitybutcanvaryasa
function of ti
me (Fried & Feldman, 2008). Hence,
identical performances at differenttimes can lead to
differentassessments.Biasesinassessmentinvolving
humanjudgmentareageneralphenomenonandare
muchresearchedinfieldssuchasmedicine(Higgins
&Altman,2008;Higginsetal.,2011),andpsychology
(Kahneman et al., 1982; Kahneman, 2011). These
Towards Automated Performance Assessment for
Maritime Navigation
K.I.Øvergård,S.Nazir&A.S.Solberg
UniversityCollegeofSoutheastNorway,Borre,Norway
ABSTRACT:Thispaperpresentstheoutcomeofapre‐project that resultedinaninitialversion(prototype)of
an automated assessment algorithm for a specific maritime operation. The prototype is based on identified
controlrequirementsthathumanoperatorsmustmeettoconductsafenavigation.Currentassessmentmethods
of navigation in simulators involve subject ma
tter experts, whose evaluations unfortunately have some
limitations related to reproducibility and consistency. Automated assessment algorithms may address these
limitations.Foraprototype,ouralgorithmhadalargecorrelationwithevaluationsperformedbysubjectmatter
expertsinassessmentofnavigationroutes.Theresultsindicat
ethatfurtherresearchinautomatedassessment
ofmaritimenavigationhasmerit.Thealgorithmcanbeasteppingstoneindevelopingaconsistent,unbiased,
andtransparentassessmentmoduleforevaluatingmaritimenavigationperformance.
http://www.transnav.eu
the International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 11
Number 2
June 2017
DOI:10.12716/1001.11.02.03
230
limitations apply to all subjective assessments –
including the evaluations done by subject matter
experts (SMEs). Factors such as time of the day,
fatigue, mood, and low blood sugar levels can also
negativelyaffecttheoutcomeofanexpertevaluation
(Danzigeretal.,2011).
We suggest the use of automated
assessment
algorithms to support the subjective assessment by
SMEs.Unfortunately, thereare numerouschallenges
toautomatingtheassessmentofmaritimenavigation.
First, navigation is an open goal‐oriented work task
whichischaracterizedbyhavingmultipledegreesof
freedom. Second, navigators have the freedom to
choose the sequence and
timing of work tasks (i.e.
there are few procedures that prescribe how a task
shallbedoneinaspecificsituation).Third,navigation
and manoeuvring must be context‐sensitive and
adaptive since vessels move around in a cluttered
environment with multiple obstacles or objects (e.g.
rocks, land, other ships
or objects in the water).
Fourth,thedegreesoffreedomarealsoexceptionally
largebeca useitoftenexistsnumerousacceptableways
of reaching a destination (e.g. sailing from A to B),
meaningthatthereareseveralpossibleroutesthatall
conformtotherequirementsforsafetyandefficiency
(Bjørkli et
al., 2007). Fifth, maritime navigation is
relatedtoanumberofconstraintsrelatedtophysical
laws,operationallimits,societallaws/regulations,and
organizational goals related to safety, economy, and
theenvironment(Øvergård,2012).
The sensitivity and complexity of making
automated assessments of maritime navigation and
manoeuvring has refrained many researchers
from
developing automated methods and procedures to
assess navigation performance in real time. One
exception is the Navigational Risk Detection and
Assessment System (NARIDAS; Gauss, Rötting &
Kersandt,2007;Gauss&Kersandt,2005;Hederström,
Kersandt & Müller, 2012) – a system that combines
multiple parameters to form a risk assessment of
navigation. This system has focused on risk
assessmentofnavigation,andnotontheassessment
ofnavigationalperformanceassuch.
2 CURRENTSTATEOFAUTOMATED
ASSESSMENTOFOPERATORPERFORMANCE
A literature survey on automated procedures for
operator performance assessment suggests more
research is needed. The examples that exists comes
from
the aviation (Johannes et al., 2007), the naval
(McCormack,2007;Bjørkli&Øvergård,2012),andthe
surgerydomain(Fried&Feldman,2008).Johanneset
al.(2007)validatedanautomatedassessmentmethod
in a flight simulator by showing high correlations
betweentheoutcomeoftheassessmentalgorithmand
expert trainerʹs
rating of the operatorʹs simulator
performance. However, this approach requires a
human expert trainer to visually identify the
behaviour/actions made by the trainee‐limiting the
automaticityofthealgorithm.
Todate,thereexistnofullyfunctionalautomated
assessment systems that are adapted for tasks with
large degrees of freedom. A
limited number of
currentobjective assessment algorithms is employed
on procedure‐based work scenarios where the
sequence and timings of actions can be pre‐defined.
Examplesofassessmentsystemsforprocedure‐based
scenarios are the K‐SIM ® Polaris – Ships Bridge
simulator (Kongsberg Maritime, 2017) and systems
for the
automated assessment of operators’
performance in a petrochemical process simulator
(Mancaetal.,2012;Manca&Brambila,2011;Naziret
al., 2013; Nazir et al., 2015). However, both systems
focus on procedure‐based work tasks, and are not
designedtohandleopengoal‐orienteddynamicwork
taskssuchas
coastalnavigation.
Itisbecoming increasinglyimportant toestablish
an unbiased evaluation system that can contribute
consistent, unbiased, and transparent evaluation of
operators’ skills and competencies. To meet these
challenges, we have quantified some of the control
requirements (Petersen, 2004; Bjørkli et al., 2007;
Øvergård et al., 2010) that a
navigator must meet to
conductnavigationinasafemanner.Theresearchis
partoftheGruNTpre‐project.
3 AIMOFTHISPAPER
Themainaimofthispaperistopresentthefirststeps
toward making an automated assessment algorithm
for dynamic goal‐oriented work tasks, such as
maritime navigation. It presents the outcome of the
GruNTpre‐project,whichincludesthefirstvalidation
studyofasimpleformofthisalgorithm.
4 METHOD
Identificationofcontrolrequirementsforthesafetyof
navigationwas doneusing openinterviews with six
SMEs who all held deck‐officer certifications. Open
interviews
were chosen to allow the SMEs full
freedom to talk about important parameters and
requirements they believe are important in the
assessment of navigation. Several control
requirements were identified. Control requirements
wherecomparedtotheinformationthatwasavailable
inthe log systemofthe K‐Sim® simulator. We then
selected the control requirements that had relevant
parameters in the logging system. The control
requirementsselectedforuseinthepre‐projectare:1)
distancetolandbasedonownshiplength,2)distance
tomovingobjects(vessels)basedonownshiplength,
3)distancetofloatingobjectsbasedonship
length,4)
the deviation between ship heading and heading of
dock(meaningthattheshipshouldbeparalleltothe
dock during the last part of docking), and 5) the
minimumdepth below the ship´s keel (the so‐called
‘safetydepth’).
BasedupontheinputfromtheSME´s,wedefined
hundred‐point limits (HPL) and zero‐point‐limits
(ZPL)foreach of theparameters tofit thesimulator
model of the vessel “Thor Magni” (IMO 9679024).
‘ThorMagni’isa64.40meterslongoffshorevessel.A
draughtof5.70meterswasselectedforthevesselin
thescenarios.TheHPLand
ZPLvaluesforthevessel
aregivenintable1.
231
Table1.PerformanceIndicatorsand Control Requirements
for“ThorMagni”
_______________________________________________
PerformanceIndicatorsHPL ZPL
Distancetoland>539m <263m
Distancetosmallfloatingobjects>270m <132m
Distancetomovingobjects(e.g.vessels) >1058m<539m
Safetydepth(clearanceunderkeel,aft) HPL>5m>ZPL
DeviationbetweenHeadingofvessel <3deg. >5deg.
andHeadingofdockat10meters’
distance(measuredindegrees)
_______________________________________________
NOTE:Thesafetydepthisthesameastheminimumwater
depthbelowthevessel´skeel–correspondingtoawater
depthof5.7+5=10.7meters.HPL=Hundred‐pointlimit,
ZPL=Zero‐pointLimit,m=meters,deg.=degrees
Ifthescoreofoneoftheparameterswasabovethe
HPL the parameter was scored as 100 points. If the
score was below the ZPL a score of 0 points was
given.IftheparameterwasbetweentheHPLandthe
ZPLascoreequaltothelinearinterpolationbetween
thesescoreswasgiven.Forexample,ifthedistanceto
landwas401meters a scoreof 50points was given,
indicating that the vessel did not have an optimal
positingrelativetoland.Thecalculationisshownin
table2.
Table2.Scorecalculationfordistancetoland
_______________________________________________
SituationD>HPL HPL≥D≥ZPL ZPL>D
Scorecalculation 100
*100
DZPL
HPL ZPL
0
_______________________________________________
NOTE:D=Distancetoland,HPL=Hundred‐pointlimit,
ZPL=Zero‐pointlimit
4.1 Creation ofroutesforassessment
Thethirdauthorcreated20differentroutesusingthe
KongsbergK‐Sim
®
Navigation simulator version2.2.
AllroutesstartedjusteastofMefjordbåenintheOslo
fjordandthedestinationwasthedeep‐waterdockin
Horten on the western side of the Oslo fjord. The
routes were intentionally made of different quality
(from excellent to poor) – thereby creating variance
that
would allow us to measure the extent of
covariancebetweentheassessmentmadebytheSMEs
andtheassessmentalgorithm.
Datafromthesesimulatortrialswereloggedonce
persecondandsavedinEXCEL‐files.Theparameters
werethentransformedintoscoresbetween100and0
per the limits described in
Table1. The minimum
score for each of the five parameters (during the
wholesession)wasusedasrepresentativeparameter
scores for each of the 20 trials. The mean of the
minimum scores were then used to calculate an
overallscoreforeachscenario.
4.2 Validationofassessmentalgorithm
Validation
wasconductedintheinitialphasetoreach
consistent results. Two dedicated SMEs were
requested to rank the 20 developed routes. In
addition,anotherSMEgaveanindividualrankingof
the 20 routes. The rankings done by the SMEs were
independent of each other. The SMEs were not
informed of the
output from the assessment
algorithm.
The ranking of the 20 routes were done by
showing printed images of the routes to the SMEs.
ExamplesoftheimagesareseeninFigures1a‐b.We
also gave additional images showing the closest
passage between a vessel and other vessels for each
route – allowing the experts to assess whether the
“ThorMagni”wastooclosetoland,othervesselsand
floatingobjects.
We acknowledge that these pictures are not a
suitable way to assess navigational performance
duringtrainingoreducation.However,ourintention
wastoseewhetherahumanevaluationofa
reduced
setofinformationwassimilarastheoutputfromour
simple algorithm. Future research will of course
involvemore complexalgorithmsand assessment of
real‐timesimulator‐basednavigation.
5 RESULTS
We first took the mean scores from the assessment
algorithmsandrankedthemusingmeanranks.Ifthe
samescoreswereobtainedformultiplerouteseachof
these were assigned the mean rank (for example,
threeroutesscored100points,thesewhereallgiven
therankof1+2+3=6/3=2).
Ratingoftheeffectsizesas‘small’,‘medium’and
‘large’ are done in accordance with Cohen´s (1988)
classification of effect sizes. We compared the
association between the algorithm´s ranking of the
routes with the two independent rankings from the
SMEs. Spearman´s Rank Correlation Coefficient (r
s)
betweenthe algorithmandSME 1&2 waslarge (r
s=
0.61,95%CI[0.177,0.885]).
Togiveavisualrepresentationofthecorrelations
wehaveprintedscatterplotsshowingtherelationship
betweenthealgorithmandSME1&2(seeFigure2).
The correlation algorithm and SME 3 was also
large(r
s=0.551,95%CI[0.117,0.859]).Figure3shows
agraphicalrepresentationoftherelationshipbetween
SME3andthealgorithm.
ThecorrelationbetweentheSME1&2andSME3
was also large (r
s = 0.815, 95% CI [0.542, 0.942]),
indicating that the SMEs agreed to a larger extent
witheachotherthantheydidwiththealgorithm.
232
Figures1a‐b:InthemiddleoftheimagesyouseeBastøyisland.ThestartingpointofeachrouteisjusteastofMefjordbåen
whichisthebeaconintheimage´slowerend.TherouteswenttoHortenharborwhichisseenintheupperpartofFigure
1a‐b.The
figuresarecreatedoutofmultiplescreenshotsfromtheK‐SIM®NavigationSimulator.
Figure2.SME1&2vs.Algorithm
Figure3:SME3vs.Algorithm
6 DISCUSSION
This paper describes the work done in the GruNT
pre‐project. The project aimed to do a proof‐of‐
concept test of an automated assessment algorithm
for the assessment of a simple scenario involving
maritimenavigationandmanoeuvring. Atotal of20
sailingroutesbetweenMefjordbåenandHortenin
the
Oslo fjord were evaluated and ranked on two
different occasions by different SMEs. The SME´s
rankings were then compared with the assessment
algorithm´srankingofthesame20sailingroutes.
The results showed large rank‐based correlations
(‘large’ is >.50 per Cohen´s (1992) classification of
Effect Sizes) between the
SMEs and the assessment
algorithm (r
s = 0.515 and 0.610). The correlations
indicate that there is a quite good fit between the
algorithmand theSMEs when evaluating a reduced
andverysimplecaseofcoastalnavigation.
The agreement between SMEs and the algorithm
demonstrates the algorithm´s criterion validity by
showingthatthereexistscovariancebetween
multiple
different measurements of the same phenomenon
(Fried & Feldman, 2008). The control requirements
alsohavefacevalidity,asdistancetoobjects,land,and
other vessels are important factors with respect to
collisions and grounding. The deviation of the
vessel´s heading to the heading of the dock is
admittedlyonly
relevantforasub‐setoftheavailable
docking procedures. Therefore, the algorithm needs
to be further improved soit becomes more complex
(moreparametersandmulti‐dimensionalparameters)
and that it allows for more general navigation
scenarios. This is also supported by the fact that
correlationbetweenthe SME´srankings
were higher
(r
s=0.813)thanbetweenSMEsandthealgorithm(rs=
0.61 and 0.515), indicating that the assessment
algorithm lacks some criteria that the human SMEs
233
areusingand/orthatthelimits(HPLandZPL)needs
to be altered. A notable case of mismatch between
SMEs and algorithm is the route which have been
rankedasnumber15bybothSME1&2 andSME3but
whichisrankedsecondbest(2)bythealgorithm,see
Figure
2and3).
Themodelatthisstageissimplifiedtoassessonly
a few aspects of navigation. For example,
environmentalandefficiencyaspectsarenotassessed
atall,andthereisaneedformanymoreparameters
to assess safety of navigation and manoeuvring. In
contrast, SMEs tend to
evaluate the simulations as
real navigation exercises. However, the correlation
achieved between the model and the SMEs indicate
that the proof‐of‐concept assessment algorithm has
merit–despiteitsapparentsimplicity.
7 LIMITATIONS
Thelimitationstowardsassessingrealnavigationare
apparent, and the authors´ acknowledge this. The
limited nature of
the assessed scenario, the limited
number of parameters in the assessment algorithm,
andthewaythattheSMEs assessedand rankedthe
20 sailing routes are all points of improvement.
Hence,wedonotattempttogeneralizeourfindings
beyondsayingthatwethinkthatourconceptualidea
forautomated
assessmentofmaritimenavigationhas
merit.
Relatedtothescenarioweused,wearealsofaced
with the fundamental difference between coastal
navigation and harbour manoeuvring. During
harbourmanoeuvringthefocusis oncontrollingthe
forces between the ship and the waters to ensure
desired movement. During coastal navigation, less
focus
is needed on force controlling since the vessel
generally moves in one direction with less sharp
turns. In such circumstances a focus on safety
distances, and displaying intentions to other ships
maybemoreimportant.
The limits described in Table 1 are based upon
inputfromonlysixSMEs.Hence,
wedonotknowif
theselimitsaresomethingthatthelargepopulationof
navigatorswouldagreeto.Thiswillbethefocusfor
furtherresearch.
Also,ourassessmentalgorithmisatpresentonly
designed for assessing technical skills such as
handling and navigation of the vessel. It cannot
(currently) assess
aspects related to the interaction
between humans during navigation. This includes
features like non‐technical skills (Flin et al., 2008),
situation awareness (Endsley, 1995), or team
communication (Øvergård et al., 2015). To assess
these‘soft’skillswe mustcombine otherassessment
methodologieswithourassessmentalgorithm.Thisis
ofcourse
oneofthefutureresearchchallenges.
The model does not yet describe a real‐world
navigationsituation,butratherasimplifiedmodel.At
thisstageofresearch,theresults presentedhereinis
encouraging,andindicatesthat furtherresearch into
automated performance assessment in the maritime
domainiswarranted.Theneedfor
furtherresearchis
also supported by the fact that the industry has
shown interests in systems that can assess the
performanceoftrulyautonomousvessels.Thecurrent
model, albeit limited. illustrates some of the
methodological challenges and give an indication of
the feasibility of automated algorithm‐based
assessmentinthe
maritimedomain.
8 FUTURERESEARCH
Futureresearchwillfocusonidentifyingmorecontrol
requirementsforsafe,efficientand‘green’navigation.
Theaimwillbetoidentifythelimitsforthesecontrol
requirements by investigating a large multi‐national
sampleofexperiencednavigators.
Anotherresearchchallengeishowtocombineand
to
createconsistentweightsfortheassessmentscores
ofsafe,efficientand‘green’navigation.Methodssuch
as Analytic Hierarchy Process (AHP; Saaty, 1980)
exist,butthisapproachcannotsolveproblemswhere
theweightschangedynamically(Saaty,2007).Hence,
we will research ways to ensure a consistent set of
weightsbetweendifferent
parametersthatwillallow
us to assess navigation and manoeuvring in both
openandconfinedwaters.
Also,weaimtoextractinformationfromAIS‐data
about historical sailing routes to determine where
vesselsnormallynavigate.Baseduponthis,wehope
to supplement the data we get from talking to
experienced navigators
by also identifying the
statistical distributions of acceptable distances
betweenvessels,land,floatingobjectsasafunctionof
characteristicsofthevesselandthesituation.
The research presented may also have relevance
for the automated assessment of the performance of
truly autonomous vessels. Further research into
automated assessment is likely
to consider the
developmentinthefieldofautonomousvessels.
9 CONCLUSION
Wehavedevelopedasimpleversionofanautomated
assessmentmodulebasedupon thequantificationof
control requirements for safe navigation. There is a
large degree of agreement between SMEs and our
assessment algorithm, indicating that our simple
proof
‐of‐concept model may have merit. We believe
themodelpresentedinthispapermaybeastepping
stoneintolargerresearchefforts.
ACKNOWLEDGEMENTS
The GruNT pre‐project was funded by Kongsberg
DigitalandtheOslofjordRegionalResearchFundin
Norway(projectnumber258894).
REFERENCES
Allen,M.J.&Yen,W.M. (1979).IntroductiontoMeasurement
Theory.Belmont,CA:Wadsworth.
234
Bjørkli,C.A.,Øvergård,K.I.,Røed,B. K.,&Hoff,T.(2007).
Control Situations in High‐Speed Craft Operation.
Cognition, Technology, and Work, 9, 67‐80. doi:
10.1007/s10111‐006‐0042‐z
Bjørkli,C.A.&Øvergård,K.I.(2012).Automatedassessment
ofdocking maneuvers: Whendo we know when
anoperator
performs well? Presentation at Scandinavian Maritime
Conference2012,28‐29NovemberatVestfoldUniversity
College,Horten,Norway.
Cronbach,L.J.,Gleser,G.C.,Nanda,H.,&Rajaratnam,N.
(1972). The Dependability of Behavioral Measurements.
London,England:JohnWiley.
Danziger, S., Levav, J., & Avnaim‐Pesso, L. (2011).
Extraneous
factorsinjudicialdecisions.Proceedingsofthe
National Academy of Sciences, 108(17), 6889‐6892. doi:
10.1073/pnas.1018033108
Endsley, M. R. (1995). Toward a theory of situation
awareness in dynamic systems.Human Factors: The
JournaloftheHumanFactorsandErgonomicsSociety,37(1),
32‐64.doi:10.1518/001872095779049543
Flin,R.H.,O
ʹConnor,P.,&Crichton,M.(2008).Safetyatthe
sharp end: a guide to non‐technical skills. Aldershot,
England:Ashgate.
Freedman, D. A. (2009). Statistical Models: Theory and
Practice, rev. ed. Cambridge, England: Cambridge
UniversityPress.
Fried,G.M.,&Feldman,L.S.(2008).Objectiveassessment
of technical performance. World
Journal of Surgery, 32,
156‐160.doi:10.1007/s00268‐007‐9143‐y
Gauss, B., & Kersandt, D. (2005). NARIDAS‐Navigational
Risk Detection and Assessment System for the Ship’s
Bridge. InProceedings of the International Conference
on Computational Intelligence for Modelling, Control
andAutomation,2005(Vol.2,pp.612‐617).IEEE.
Gauss, B., Rötting, M., & Kersandt, D. (2007). NARIDAS–
evaluation of a risk assessment system for the ship’s
bridge. InHuman Factors in Ship Design, Safety and
Operation. RINA‐The Royal Institution of Naval
Architects.InternationalConference.
Hederström, H., Kersandt, D., & Müller, B. (2012). Task‐
orientedstructureofthenavigation
processandquality
controlofitspropertiesbyanauticaltaskmanagement
monitor(ntmm).EuropeanJournalofNavigation,10(3).
Higgins, J. P. T.& Altman, D. G. (2008). Assessing risk of
biasinincludedstudy.InJ.P.T.HigginsandS.Green
(eds.). Cochrane Handbook for Systematic Reviews of
Interventions
(pp. 187‐242). West Sussex, England: John
Wiley&Sons.
Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P.,
Moher, D., Oxman, A. D., Savovic, et al. (2011). The
Cochranecollaborationʹstoolforassessingriskofbiasin
randomised trials. British Medical Journal, 343(7829),
d5928.doi:
10.1136/bmj.d5928
Kahneman, D. (2011). Thinking, Fast and Slow. New York,
NY:Farrar,StrausandGiroux.
Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment
under uncertainty: Heuristics and biases. Cambridge,
England:CambridgeUniversityPress.
KongsbergMaritime(2017)K‐SimNavigation–Kongsberg.
Web site Kongsberg Maritime [Available at]
https://www.kongsberg.com/en/kongsberg‐
digital/maritime%20simulation/k‐
sim%20navigation%20‐page/
Manca, D., Nazir, S., Colombo, S., & Kluge, A. (2014).
Procedure for automated assessment of industrial
operators.ChemicalEngineeringTransactions,36,391‐396.
doi:10.3303/CET1436066
Manca,D.,&Brambilla,S.(2011).Amethodologybasedon
the Analytic Hierarchy Process for the quantitative
assessmentofemergencypreparednessand
responsein
road tunnels. Transport Policy, 18(5), 657‐664. doi:
10.1016/j.tranpol.2010.12.003
Manca, D., Nazir, S., Lucernoni, F., & Colombo, S. (2012).
Performanceindicators for the assessmentofindustrial
operator.ComputerAidedChemicalEngineering,30,1422‐
1426.Doi:10.1016/B978‐0‐444‐59520‐1.50143‐3.
McCormack, W. (2007). Automated Operator and System
Performance Assessment. In T. Bastiaens & S. Carliner
(Eds.), Proceedings of World Conference on E‐Learning in
Corporate, Government, Healthcare, and Higher Education
2007 (pp. 7252‐7259). Chesapeake, VA: Association for
theAdvancementofComputinginEducation(AACE).
Nazir,S.,Colombo,S.,&Manca,D.(2013).Minimizingthe
riskin
theprocessindustrybyusingaplantsimulator:a
novel approach. Chemical Engineering Transactions, 32,
109‐114.doi:10.3303/ACOS1311028
Nazir, S., & Manca, D. (2015). How a plant simulator can
improveindustrialsafety.ProcessSafetyProgress,34(3),
237‐243.doi:10.1002/prs.11714
Nazir,S.,Sorensen, L.J.,Øvergård,K.I.&Manca,
D.(2015).
Impact of training methods on distributed situation
awarenessofindustrialoperators.SafetyScience,73,136‐
145.doi:10.1016/j.ssci.2014.11.015
Petersen,J.(2004).Controlsituationsinsupervisorycontrol.
Cognition, Technology, and Work, 6, 266‐274. doi:
10.1007/s10111‐004‐0164‐0
Saaty, T. L. (1980). The analytic hierarchy process: planning,
priority setting, resources allocation. New York, NY:
McGraw‐Hill.
Saaty, T. L. (2007). Time dependent decision‐making;
dynamicprioritiesintheAHP/ANP:Generalizingfrom
points to functions and from real to complex
variables.Mathematical and Computer Modelling,46(7),
860‐891.
Øvergård, K. I. (2012). Absolute constraints, situation
awareness and
modelling of socio‐technical systems.
Presentation at the Scandinavian Maritime Conference
2012, 28‐29 November at Vestfold University College,
Horten,Norway.
Øvergård,K.I.,Bjørkli,C.A.,Røed,B.K.&Hoff,T.(2010).
Control strategies used by experienced marine
navigators:observationsofverbalconversationsduring
navigation training. Cognition, Technology,
and Work,
12(3),163‐179.doi:10.1007/s10111‐009‐0132‐9
Øvergård,K. I.,Nielsen,A.R., Nazir,S.,&Sorensen,L.J.
(2015). Assessing navigational teamwork through the
situational correctness and relevance of
communication.Procedia Manufacturing,3, 2589‐2596.
doi:10.1016/j.promfg.2015.07.579
.
