International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 3
Number 3
September 2009
Simulators are widely used for different educational
purposes, such as learning, training, research, forma-
tive and summative assessment and evaluation, not
only in maritime education and training, but also in
aviation, nuclear power plants (Ham et al., 2008),
medicine: surgery (Cosman et al., 2002; Maithel et
al, 2006), nursing (Decker et al., 2008). Simulations
are example of technology enabling learning to be
done more easily and effectively (Biggs & Tang,
2008) in a safe and controlled environment. As tech-
nology is developing rapidly the simulators become
more popular educational mean.
Over last few decades there was a huge growth of
automated navigation systems. More and more sys-
tems making navigator’s job easier came on the sce-
ne. It has begun with global positioning systems
(GPS) and nowadays grown into electronic chart
display and information systems (ECDIS). There is
no doubt that new sophisticated systems facilitate
navigator‘s job and ensure safer navigation. For ex-
ample, the study of Gould et al. (2009) proved that
ECDIS appeared to improve navigation performance
compared to conventional navigation based on paper
charts. Although this fact was proved under normal
conditions and more investigations should be under
high-workload conditions and in the presence of
other stressors. Habit to use sophisticated (automat-
ed) systems sometimes have negative impact to the
qualification of the navigator, as captain can lose his
proficiency in using usual navigational means be-
cause of permanent usage of automated navigation
systems. It is obvious for the experienced navigators
that modern navigation equipment today still is not
Refusing the traditional means of navigation,
question of navigational safety may arise in case of
the failure or malfunctions of automated navigation
system, because during initial study stage attention
to the traditional navigational tasks slacks as student
contemplates that use of traditional navigational
tasks is not relevant, but this can be designated as
lack of navigators qualification, where qualifications
are well-defined in STCW (Standards of Training,
Certification and Watch keeping) convention.
Improvement of navigational systems tended to
facilitate safer navigation and load of the additional
New Capabilities of the NTPRO 4000 Full
Mission Ship Handling Simulator in the
Assessment and Evaluation Processes at
Lithuanian Maritime Academy
R. Zazeckis, I. Bartuseviciene & R. Maksimavicius
Lithuanian Maritime Academy, Klaipeda, Lithuania
ABSTRACT: Nowadays the equipment used on ships is becoming more and more sophisticated, safety of
shipping depends heavily not only on trustworthiness of high-tech products, but also on mariners’ compe-
tence. It is necessary to find methods to enable seafarers to keep track with technical developments. Simula-
tors as a tool combined with a properly developed course curriculum and qualified instructors provide an ap-
propriate method not only for training, but also for measuring, assessing and evaluating individual mariner
performance in order to test levels of competency and proficiency. According to Nieri (1995, p. 1/6), “The
development of performance-based assessment tools would utilize the widely-recognized advantages of simu-
lators and part-task trainers to generate a scenario in which the license candidate must demonstrate his
knowledge of system relationships, knowledge of operational procedures, monitor systems and situations, re-
spond to unexpected occurrences, identify and diagnose problems, and manage personnel, in a dynamic
world“. This paper discusses the usage of new capabilities of the NTPRO 4000 full mission ship handling
simulator in the assessment and evaluation processes at Lithuanian Maritime Academy.
tasks to the navigator increased requirements to the
qualification of seafarers. As a consequence teaching
process and teaching aids had to be improved. So
navigational simulators able to simulate different
navigational situations and to ensure that student is
able to clarify different solutions of navigational task
and implement the very best solution in certain cir-
cumstances were implemented to education and
training process of navigators.
Usage of simulators is the best solution for the
analysis of the actions taken by the students during
the performed tasks and for the understanding of the
competence of students. The diagnostics and analy-
sis of mistakes will allow avoiding possible mis-
takes, which may arise in different navigational situ-
During the education and training process most of
the mistakes are analyzed theoretically before the
usage of the simulator. However nowadays the in-
structors are practically able to allow students to
make mistakes using technologies of the modern
simulators and show the possible consequences of
the mistakes to students without real damage to the
environment. Possibility to observe mistakes during
performance of special navigational tasks helps to
make appropriate decision about the competence of
the student not only in routine, but also in emergen-
cy situations.
NTPRO 4000 full mission ship handling simula-
tor is used continuously in the process of education
and training of navigators in Lithuanian Maritime
Academy. Two years ago the new capability in
Evaluation and Assessment System (TEAS) was im-
plemented. The new capability allows assessing ob-
jectively the correctness of an exercise performance
by a trainee on NTPRO 4000 navigation simulator.
This opportunity is used continuously during all
learning process for the formative assessment. Ac-
cording to Brown & Glasner (1999), good assess-
ment (formative or summative) has to be valid, reli-
able, practical, developmental, manageable, cost-
effective, fit for purpose, relevant, authentic, closely
linked to learning outcomes and fair. In formative
assessment, the results are used to improve learning:
students can make mistakes: “the error detection is
the basis for error correction” (Biggs & Tang, 2007,
p. 164). It’s completely different in summative as-
sessment and evaluation: the results of such an as-
sessment are used to grade or certificate students at
the end of a course or program. According to Biggs
& Tang (2007), error no longer is there to instruct,
as in formative assessment: error now signals pun-
ishment. That’s why much more attention should be
paid for the final examination, where complex as-
sessment evaluates not only how each part of
knowledge is soaked up, but also how all knowledge
and skills are implemented in close to real situation
in real time. The TEAS gives this possibility.
For two years the TEAS of NTPRO 4000 full
mission ship handling simulator has been used for
summative assessment and evaluation during final
examination at Lithuanian Maritime Academy. The
authors are still searching the ways to improve ob-
jectivity, validity and reliability of the final evalua-
tion of the students. Some results of this experience
are presented in the article.
It is well-know that mandatory minimum require-
ments for certification of officers in charge of navi-
gational watch are well-defined in STCW code. The
code clearly states the standards of competence that
every candidate for certification shall be required to
demonstrate: the competence to undertake at opera-
tional level, the tasks, duties and responsibilities are
listed in column 1 of STCW code table A-II/1; the
minimum knowledge, understanding and proficiency
required for certification is listed in column 2 of
STCW code table A-II/1; the level of knowledge of
subjects listed in column 2 of STCW code table A-
II/1 shall be sufficient for officers of the watch to
carry out their watch keeping duties; every candidate
for certification shall be required to prove evidence
of having achieved the required standard of compe-
tence in accordance with methods for demonstrating
competence tabulated in columns 3 and 4 of STCW
code table A-II/1.
With reference to above mentioned TRANSAS
NTPRO 4000 simulator (especially the TEAS) is an
appropriate solution solving the student’s certifica-
tion problems in LMA.
2.1 Possible competency assessment using NTPRO
Most of the educational programs during the process
of education and training of navigators are accom-
plished with reference to STCW code. The same re-
quirement corresponds to assessment and evaluation
process. Assessment and evaluation in accordance to
STCW code requirements can be improved using the
NTPRO 4000 TEAS because this system allows as-
sessing the navigator’s competences stated in STCW
code (table 1).
Table 1. The list of competences, which could be assessed us-
ing NTPRO 4000 TEAS.
1. Plan and conduct a passage and determine position:
1.1. Ability to determine the ship‘s position by use of:
1.2. Ability to use navigational charts and publications,
such as sailing directions, tide tables, notices to mari-
ners, radio navigational warnings and ship’s routing
information.(ECDIS systems are considered to be in-
cluded under the term “charts”);
1.3. Ability to determine the ship‘s position by use of
electronic navigational aids;
1.4. Ability to operate echo sounders and apply the in-
1.5. Ability to determine error of the magnetic and giro
compasses and to allow for such errors;
1.6. Knowledge of steering control systems, operational
procedures and change-over from manual to automat-
ic control and vice-versa. Adjustment of controls for
optimum performance;
1.7. Ability to use and interpret information obtained
from shipborn meteorological instruments;
1.8. Knowledge of the characteristics of the various
weather systems, reporting procedures and recording
1.9. Ability to apply the meteorological information
2. Maintain a safe navigational watch:
2.1. Thorough knowledge of the content, application and
intent of the International Regulations for Preventing
Collisions at Sea;
2.2. Thorough knowledge of the basic principles of keep-
ing a navigational watch;
2.3. Thorough knowledge of effective bridge team work
2.4. The use of routeing in accordance with the General
provisions on Ship‘s Routeing.
3. Use of Radar and ARPA to maintain safety of navigation:
3.1. Performance including:
3.1.1. Factors effecting performance and accuracy;
3.1.2. Setting up and maintaining displays;
3.1.3. Detection of misrepresentation of information,
false echoes, sea return, etc., racons and
3.2. Use including:
3.2.1. Range and bearing; course and speed of other
ships; time and distance of closest approach of
crossing, meeting overtaking ships;
3.2.2. Identification of critical echoes; detecting
course and speed of other ships; effect of
changes in own ship‘s course or speed or both;
3.2.3. Application of the International Regulations for
Preventing Collisions at Sea;
3.2.4. Plotting techniques and relative and true motion
3.2.5. Parallel indexing;
3.3. Principal types of ARPA , their display characteris-
tics, performance standards and the dangers of over
reliance on ARPA;
3.4. Ability to operate and to interpret and analyze in-
formation obtained from ARPA, including:
3.4.1. System performance and accuracy, tracking ca-
pabilities and limitations, and proceeding de-
3.4.2. Use of operational warnings and system test;
3.4.3. Methods of target acquisition and their limita-
3.4.4. True and relative vectors, graphic representation
of target information and danger areas;
3.4.5. Deriving and analyzing information, critical
echoes, exclusion areas and trial manoeuvres.
4. Respond to emergences
4.1. Emergency procedures:
4.1.1. Precautions for the protection and safety of pas-
sengers in emergency situations;
4.1.2. Initial action to be taken following a collision or
a grounding; initial damage assessment and
4.1.3. Appreciation the procedures to be followed for
rescuing persons from sea, assisting a ship in
distress, responding to emergencies which arise
in port.
5. Respond to a distress signal at sea;
5.1. Knowledge of the contents of the IMO Merchant
Ship Search and Rescue Manual (MERSAR).
6. Use the IMO Standard Marine Communication Phrases
and use English in written and oral form;
6.1. Adequate knowledge of the English language to ena-
ble officer:
6.1.1. To use charts and other nautical publications;
6.1.2. To understand meteorological information and
messages concerning ship‘s safety and opera-
6.1.3. To communicate with other ships and coast
6.1.4. To perform the officer’s duties with a multilin-
gual crew;
6.1.5. To use and understand the IMO Standard Ma-
rine Communication Phrases.
7. Transmit and receive information by visual signalling;
7.1. Ability to transmit and receive signals by Morse
7.2. Ability to use the international Code of Signals.
8. Manoeuvre the ship;
8.1. Knowledge of ship manoeuvring and handling:
8.1.1. The effects of dead-weight, draught, trim, speed
and under keel clearance on turning circles and
stopping distances;
8.1.2. The effects of wind and current on ship han-
8.1.3. Manoeuvres and procedures for the rescue of
person overboard;
8.1.4. Squat, shallow water and similar effects;
8.1.5. Proper procedures for anchoring and mooring
In most cases the basic ordinary training and as-
sessment systems are applied. The process of basic
training and assessment system is presented in fig. 1.
Figure 1. Basic training and assessment system in LMA.
The more detailed description of the certain exer-
cise using NTPRO 4000 will be more complicated
and will look like in fig. 2.
Figure 2. Detailed exercise.
Several exercises according to the number of
competences intended to assess can be included to
overall assessment (e.g. final examination). Evalua-
tion of each exercise has its own particular weight
and influences as a part of the overall assessment
and evaluation the final decision about competency
of the trainee.
The thorough explanation of typical example of
above-mentioned detailed exercise is presented in
fig. 3:
Figure 3. Example of detailed exercise.
It is important to mention that only above men-
tioned competencies (table 1) can be assessed and
evaluated using NTPRO 4000 TEAS in present time,
the other competencies defined in STCW code are
evaluated in particular training during common ex-
Taking into consideration all the mentioned above,
final decision for issuing the Certificate of Compe-
tency is made using the complex assessment.
Complex assessment involves competences men-
tioned in 2.1 (table 1) and additional competencies
which were evaluated in particular training. The ad-
ditional competences of the students, such as use of
ARPA, ECDIS, usually are evaluated before the fi-
nal examination using NTPRO 4000 by Lithuanian
Maritime Safety Administration in accordance with
IMO Model Course 1.07 for ARPA, and IMO Model
Course 1.27 for ECDIS; and the results are the basis
for the diploma of the competency. Although the as-
sessment of mentioned competences is a part of
complex assessment as the student must show his
ability to implement all of competences required by
STCW code, and especially use not only of automat-
ed navigation systems, but also the solutions of tra-
ditional navigational tasks in real time simulation.
The overall complex assessment and weight of
assessed tasks can be presented in fig. 4.
Assessment and evaluation
Assessment and evaluation of target task Overall assessment and evaluation
Practical implementation
Practical study of the task
Practical implementaion of theoretical
Theoetical basics
Studing of the suject Analysis of the practical examples
Evaluation and assessment of the task
Correct operaor response
Performance measure
Overall assessment and evaluation
Evaluation and assessment of the actions taken and response in
the particular task(s)
Correct operator response
Position observed in accordance with instructions
Acceptable ranges for position observation: BRG +/- 0.5°;
RNG +/- 0.05nm; Speed +/- 1 knot; Course +/- 4°.
Performance measure
Position observed Ploted and recorded vessel data
Operator is instructed to observe vessel position using
bearing and range
Operator is instructed to plot and record vessel data when
Figure 4. Overall complex assessment.
During all complex assessment process correct
and incorrect actions and responses are electronical-
ly logged additionally and established paper log
book is filled, as not all competences can be logged
It is important that employing TEAS the percent-
age system of evaluation should be applied; it means
that student comes to assessment with the score of
all competencies 100%, and later he gets penalty
points for any failed navigational task. Penalty
points are multiplied by weight of the task evaluated,
so final evaluation result E can be expressed by for-
wPwPwPwPE 100)...(100
Where: P
penalty point for appropriate task; w
weight of the penalty point.
If there is necessity in more accurate results of
competence evaluation each task can be evaluated
separately with its penalty points (applying the same
formula), influencing final evaluation.
Detailed assessment chart is presented in fig. 5,
extraction of the competence evaluation in overall
complex assessment is presented in fig. 6.
Figure 5. Detailed chart of complex assessment.
Figure 6. Extraction from overall complex assessment for
“Plan and conduct a passage and determine position” compe-
During evaluation and assessment process all
mentioned competencies in 2.1 (table1) are thor-
oughly checked, especially competencies used to
solve traditional navigational tasks. As TEAS sys-
tem electronic logbook, paper logbook is employed,
the subjective assessors’ opinion is minimized, and
it allows making the very correct and objective deci-
sion about the competency and suitability for appro-
priate position of the student.
In other words the assessment using NTPRO
4000 is based on the on the check of the exercise ful-
filment correctness with regard to the selected set of
criteria. The criteria check consists in comparing the
exercise “assessment parameters” to the set limit
values according to the set “rule”. At each moment
of time a relative error is recorded (relative deviation
of the “assessment parameter” from the limit values)
and penalty points are calculated as a function of the
relative error and the “error weight”.
The overall sum of penalty points in the observa-
tion interval is calculated as a sum of penalties at
each moment of time by each “assessment parame-
The trainee competency is assessed in points
(Score %) starting from 100% minus penalty score.
The correctness of the exercise fulfilment can be as-
sessed after the end of the exercise during the play-
back of the exercise log. The Training Report is cre-
ated automatically at the moment when the log file is
loaded. Competency assessment can be obtained in
the process of the exercise and after its end as well.
For the final assessment of the trainee competency,
the “Passing score” is entered.
This process allows making objective judgement
about student’s competency and enable saving pub-
lished the results for longer time.
Extraction from practical evaluation and assessment
of Use of Radar and ARPA to maintain safety of
navigation is presented below. Ability to operate and
to interpret and analyse information obtained from
the radar, including factors affecting performance
and accuracy of the trainee is tested.
The exercise is set in adverse visual and radar ob-
servation conditions (fog, rough se, rain). The train-
ee is assigned with a task to proceed in restricted
waters with narrow passage making a 90° turn round
the buoy (which becomes a reference point) at a set
distance with a margin equal to the possible radar
range measurement error. Penalty Charge value is
The trainee must adjust the optimum radar picture
quality, identify the buoy echo among the clutter and
perform the required manoeuvre.
Evaluation is made through the exercise. Perfor-
mance criteria actual distance to reference point
must be near the limit defined (if the actual distance
exceeds defined limits, penalty charge are imposed,
here in this example-30% if grounding occurs-
Calculations of the assessment for this particular
task will look as follows:
0)7,0100130(%100)(%100 =+=+=
70)7,00130(%100)(%100 =+=+=
100)7,0010(%100)(%100 =+=+=
-penalty for exceeded limit,
if the grounding occurs,
-weight of the penalty in
particular exercise.
As seen above, depending on the trainee’s ability
to accomplish exercise, he gets final evaluation for
the particular task, which is a part of the complex as-
sessment during final examination.
During initial stage of study attention to the tradi-
tional navigational tasks slacks as student con-
templates that use of traditional navigational tasks
is not relevant.
Use of navigation simulators enables to test all
competencies in accordance with STCW code es-
pecially solving traditional navigational tasks, fi-
nal examination using NTPRO 4000 TEAS based
on methods described in the paper may be basis
for certification of competency.
Using TEAS the percentage system of evaluation
is applied, it is predicted that student gets the
score of 100% at the beginning of the assessment,
and later points for any failed navigational task
multiplied by weight are subtracted from initial
100 % in order to get final evaluation.
Lithuanian Maritime Academy uses new capabil-
ity of NTPRO 4000 not only during training pro-
cess by performing formative assessment but also
for the summative assessment and evaluation of
students’ competency during final examination.
Assessors are representatives and persons ap-
proved by Lithuanian Maritime Safety Admin-
istration; this enables getting the Certificate of
Competence after final complex assessment in
Lithuanian Maritime Academy.
Biggs, J., Tang, C. 2007. Teaching for Quality Learning at
University. Berkshire: Society for Research into Higher
Education & Open University press.
Brown, S., Glasner, A. 1999. Assessment Matters in Higher
Education: Choosing and Using Diverse Approaches.
Buckingham: Open University Press.
Cosman, P.H., Cregan, P.C., Martin, C.J., Cartmill, J.A. 2002.
Virtual Reality Simulators: Current Status in Acquisition
and Assessment of Surgical Skills. ANZ Journal of Surgery
72: 30-34.
Decker, S., Sportsman, S., Puetz, L., Billings, L. 2008. The
Evolution of Simulation and its Contribution to Competen-
cy. The Journal of Continuing Education in Nursing, 39(2):
Ham, D., Park, J., Jung, W. 2008. Evaluation of Human Inter-
action with Complex Systems Using a Full-Scope Simula-
tor: Lessons Learned and Methodological Issues. Interna-
tional Journal of Human-Computer Interaction, 24 (4):
Gould, K.S., Roed, B.K, Saus, E.R., Koefoed, E.F., Bridger,
R.S., Moen, B.E. 2009. Effects of navigation method on
workload and performance in simulated high-speed ship
navigation. Applied ergonomics 40 (2009): 103-114.
International Maritime Organisation. 2001. International Con-
vention on Standards of Training, Certification and Watch-
keeping for Seafarers, 1978, as amended in 1995, (STCW
95). London: Author.
Maithel, S., Sierra, R., Korndorffer, J., Meumann, P., Dawson,
S., Cellery, M., Jones, D., Scott, D. 2006. Construct and
face validity of MIST-VR, Endotower, and CELTS. Are we
ready for skills assessment using simulators? Surgical En-
doscopy (2006) 20: 104-112.
Matveevskii, A.S., Gravenstein, N. 2008. Role of Simulators,
educational programs, and nontechnical skills in anesthesia
resident selection, education and competency assessment.
Journal of Critical Care (2008) 23: 167-172.
Nieri, D. S. 1995. Certification and licensing of mariners based
on performance assessment. Proceedings of the second in-
ternational conference on engine room simulators ICERS 2,
05-09 June 1995: 1-9. Rimouski: Canada.
Transas. 2005. Navi-Trainer 4000 (v.4.5.1) Instructor manual.
Transas Ltd.