International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 5
Number 4
December 2011
The effectiveness evaluation of the Information &
Communication Technology (ICT) in the didactics
was mainly based on the experience & analysis (pos-
itivistic) methods, which accept that knowledge may
be attributed only to the objective reality existing re-
gardless of the values and beliefs the ones seeking to
discover her. Methods of the physics and behavioral
sciences are adopted, as well as objective forms of
knowledge and deterministic acknowledgements for
the human nature. As it shown in the international
bibliography, the use of multiple methods of evalua-
tion is more effective and the combinatorial use of
quantitative and qualitative approaches confines
their weaknesses (Brannen, 1995, Bryman, 1995,
Patton, 1990, Retalis et al., 2005). That combination
may bring out multiple applications of the ICT in the
educational sector, thus contributing in a more “suf-
ficient” evaluation of an application. According to
Hubermas (1971), the final target of the ICT effec-
tivity evaluation on the educational procedure must
call for the examination and the evaluation of the
three interests on knowledge: the technical (suggests
the scientific opinion experience & analysis exam-
ple), the practical (the interpretation methods that of-
fer knowledge that serves the “practical” interest on
the understanding and interest) and “manumission”
(offers the necessary critical and dialectical basis for
the substantial connection between theory and ac-
tion). Knowledge is the nucleus and around should
orbit the evaluation procedure as well as the result of
the human action that is defined by the physical
needs and interests that lead and at the same time
shape the way the knowledge is structured in several
human activities. It is more than obvious that all the
above conclude to the fact that the evaluation targets
and the interest for knowledge define the evaluation
mode and its results (Carr and Kemmis, 1997, Hu-
bermas, 1971, Retalis et al., 2005).
In the evaluation of the tutorial systems, the sys-
tem’s term utility is analyzed in two supplemen-
tary components: the utility offered to the final re-
garding the system’s efficiency and the usability
regarding the facility of the users to comprehend or
use that usability. These two senses are bound to-
gether but it’s not necessary one to exist without the
other (Grudin, 1992). In the evaluation, in particular,
is widely used the term manageabilitywhich is a
self-evident requirement for all the systems and tools
managed by men (Avouris, 2000, Avouris et al.,
2001, Nielsen, 1993, Papachristos et al., 2010).
One of the problems emerging by the evaluation
techniques is the fact that they are based on the ob-
servation and on the users’ answers. Today a great
interest is imposed on the application of the objec-
tive usability testing. This testing mainly concerns
the observation of the eyes and the measurement of
physical data related to several part of the human
physiology (heart activities, activity of perspiration
glens, electric activity of the muscles and brain). Re-
cording the measurements requires the application of
several organs and sensors on the users. The exist-
ence and correspondence of physiological measure-
ments patterns in specific emotional situations and in
Evaluation of Educational Software for Marine
Training with the Aid of Neuroscience
Methods and Tools
D. Papachristos & N. Nikitakos
Dept. Shipping, Trade and Transport, University of Aegean, Greece
ABSTRACT: The evaluation with the use of neuroscience methods and tools of a student’s satisfaction
happiness from using the e-learning system (e-learning platforms, e-games, simulators) poses an important re-
search subject matter. In the present project it is presented a research on course conducted in the Faculty of
Merchants Officers, Marine Academy of Aspropyrgos. In particular, this research with the use of a neurosci-
ence tool-gaze trucker, investigates the amount of satisfaction of the students using a simulator by monitoring
the users’ eye movement in combination with the use of qualitative and quantitative methods.
general the determination of a theoretical frame of
interpretation that defines a user’s emotional reac-
tion in an interface is a state of art research field
(Dix et al., 2004, Picard, 1997, Retalis et al., 2005).
Today the use of the consolidated strategic re-
search that is called cognitive neuroscience and in-
cludes the study of the behavior and the external sit-
uations related to it, as well as the expansion of the
nervous system mechanisms that intervene in this re-
lationship, leads to a better expansion of the user’s
physiological reactions. In the marine education and
training, in particular, the modelization of the stu-
dent’s satisfaction based on objective standards (bi-
ometrical measurements) that present a better frame
of interpretation is an important research subject.
Via cognitive neuroscience is possible to determine
a background that explains the satisfaction effect and
at the same time presents new considerations that
will expand the existing so far educational conclu-
sions regarding the adults’ education. The coexist-
ence of the obtainment and understanding of
knowledge and the acquisition of skills (according to
specific standards) necessary to practice the marine
profession is a crucial element for the marine profes-
sionals. In addition it will promote the design im-
provement of the e-learning software programs and
will help the e-learning software develop-
ers/manufacturers to improve their products on the
terms of the learning success and effectiveness
(Goswami, 2007, IMO, 2003, Papachristos and Ni-
kitakos, 2010, Kluj, 2002).
The recommended research procedure aims at the
modelization of the (subjected) satisfaction of the
students users in marine education via the user in-
terface evaluation of the MATLAB simulator. The
experiment is conducted in the Faculty of Merchants
Officers, Marine Academy of Aspropyrgos with a
random sampling among the students. The experi-
ment’s researching purposes concern the evaluation
of the students’ satisfaction from the MATLAB
simulator use and the educational evaluation of
MATLAB from the user’s point of view.
The research that will be conducted is a combina-
tion of a qualitative quantitative methodology on
one hand and a use of neuroscience tools (gaze
trucker use) on the other hand. The purpose of that
is to combine the positive elements of the corre-
sponding methodologies: targeting at measurable re-
sults & variable testing (quantitative), interpretative,
explanatory (qualitative) and more objective meas-
urements by “observation” of physiological data of
the user (neuroscience tools).
For the quantitative research, the questionnaire
method will be applied in order to extract measura-
ble results regarding the educational evaluation of
the MATLAB by the users in combination to the el-
ements that satisfy the user by the simulator’s opera-
tion. The use of the qualitative research via semi-
structured interviews was chosen in order to exam-
ine the deeper reasons of the users’ satisfaction of
the simulator and verify the neuroscience tool meas-
urements and the quantitative research results. The
use of the gaze trucker was chosen because it pre-
sents more “accurate” measurements. The course of
the researching procedure in the research conduction
is shown in the following figure (Fig.1).
Figure 1. Research steps.
The determination of the satisfaction level con-
cerns the following parameters: system’s usability
regarding the whole system (total usability), as well
as the corresponding learning scenario, the stimulant
for the active participation of the users (from the
system and the learning scenario) and the user’s
friendliness regarding interface. The emerging of
conclusions from the research results concludes the
evaluation procedure that concerns the identifica-
tionof the measurements from the gaze trucker in
relation with firstly the questionnaire results (educa-
tional evaluation) and secondly and most important-
ly the interview results (educational evaluation &
determination of the user’s satisfaction data) (Fig.
2). The evaluation’s intention is to create measure-
ment patterns that correspond to psychological char-
acteristics. The learning scenario applied was devel-
oped according to the STCW’ 95 corresponding
educational specifications on the Merchant Officers
education in collaboration with the staff of the Ma-
rine Academy of Aspropyrgos.
Figure 2. Verification procedure.
The human vision/eyesight is an extremely compli-
cated activity with physical restrictions as well as re-
strictions concerning perception. The optical percep-
tion is distinguished in the physical reception of a
stimulant from the outer world and the pro-
cessing/interpretation of the certain stimulant. The
observation of the gaze trucking as well as the pupil
is a possibility existing in many years now but the
technological developments in both material equip-
ment and software made it more viable, mostly as an
approach to measure usability (Duchowski, 2003,
Dix et al., 2004). The eye movements are supposed
to illustrate the amount of cognitive elaboration a
screen demands and therefore the difficulties and fa-
cilities in its processing. In general the optical meas-
urement focuses on the following (Goldberg and
Kotval, 1999, Dix et al., 2004): the focus points of
the eyes, the eye movement patterns and the varia-
tions of the eye pupil. The methodology of effectua-
tion of the optical measurement that was chosen on
the present experiment aims at the eye movement
observation and the user’s head movement regarding
with time. For carrying out of the experiment the
“Face Analysis” software developed by the Image,
Video and Multimedia Systems Lab (IVML) του
National Technical University of Athens (NTUA) is
applied in collaboration with a Web camera con-
nected to the computer hosting the research subject
(MATLAB) (Asteriadis et al., 2009). That software
records (parameters): (a) Eye gaze vector: vertical
and horizontal movements (2 floats), (b) Head Pose
Vector: pitch, yaw (2 floats), pitch and yaw come in
normalized floats, (c) Dist_monitor: Float indicating
distance of the user from the monitor (~1: fixed dis-
tance, <1 goes far, >1 comes close)(1 float) and (d)
Head roll: roll comes in degrees (1 float). The whole
data recording and analysis procedure is shown in
the following figure (Fig.3).
Figure 3. Optical Data Recording and Analysis procedure.
The first (random) sampling was carried out on the
January 2011, in the Information Technologies La-
boratory of the Marine Academy of Aspropyrgos.
The samples consisted of 11 students (9 Male, 2
Female) that were subjected to a specific experi-
mental procedure (research methodology) (Table 1).
The sample’s learning medical profile as shown in
Table 2 presents homogeneity, whereas from the
first processing of the Matlab gaze trucker data
emerge useful information (Table 3, 4).
Table 1. Structure of Sample
Variables Male (2)
Female (2)
Semester E (9) E (2)
Pass “Control System & - -
Matlab” course
Study “Control System & (9) (2)
Matlab” course
No pass ”Control System & - -
Matlab” course
Level computer using
(No Use, Basic, Medium, M(6),B(2)A(1) B(1), M(1)
Table 2. Structure of medical-learning profile
Question theme Male (2)
Female (2)
trabismus N*(9) N(2)
Monochromatism N (9) N (2)
Eye disease N (8), N,
Y(1-myopia,astigmatism) Y*(1-myopia)
Eye operation N(9) N(2)
Dyslexia N(8), Y(1) N(2)
ADHD N(9) N(2)
N: No,
Y: Yes,
Table 3. 1
Data Set of Sample.
Time (min) Eye gaze vector Eye gaze vector
(aprox.) vertical horizontal
Average F*(13.5), F*(0), F*(-.16),
M*(8.5), M*(0), M*(7.04),
T*(9.4) T*( 0) Τ
Standard F*(2.12), F*(0), F*(0.54),
Dev M*(3.3), M*(0), M*(31.81),
T*(3.67) T*(0) T*(29.7)
F*:Female, M*:Male, T*:Total
Table 4. 2
Data Set of Sample.
Dist_monitor Head roll Head pose Head pose
vector pitch vector yaw
Average F*( 1.2), F*(-1.2), F*( 0.04), F*(-0.16 ),
M*(1.23), M*( 2.03), M*( 0.04), M*( -0.04),
T*(1.22) T*( 1.14) T*(0.04 ) T*( -0.07)
Standard Dev F*(0.29), F*(7.90), F*( 0.31), F*( 0.54),
M*(1.07), M*(10.12), M*(0.25 ), M*( 0.35),
T*(0.92) T*(9.67 ) T*( 0.21) T*( 0.41)
F*:Female, M*:Male, T*:Total
The first measurements show that men needed
less time to execute the scenario than women (the in-
terview showed that women are more informed
about Matlab and the control systems and answered
in more questions than men). In the distance from
screen observation (dist_mon variable) it was rec-
orded that both men and women approach the screen
(>1), which means that they both fully watching the
scenario and have difficulty in using Matlab. As the
screen distance increased in men, they seemed to
have a more attention of the scenario and the Matlab
use, even though they don’t seem to achieve the
same success in the scenario solving case as the
women. In the next variable, we observe a non-
balance in the inclination of the head (~0) in both
sexes, which probably suggests a deeper inspection
of the Matlab interface (that is probably ought to the
lack of more practice time). The eye gaze vector ap-
proximately shows (the females marginally watch
inside the screen although the sample is small), the
students watch inside the screen and not outside of it
(there is an interest on the scenario execution with
the help of Matlab software or search about that).
The interview shows that 7 people from a total of
11 consider that Matlab software improves teaching
the control systems (6 male, 1 female), all of them
consider the Matlab interface to be friendly to the
user and that it is necessary to have a basic mathe-
matics background in order to use it. Without a
mathematics background, its operation is difficult.
During the execution of the learning scenario (closed
control system solving exercise and calculation of
time response with a paced entrance) none of the 11
students could get to a point of solution, rendering it
to the lack of necessary time practice of Matlab in
the School’s educational program and 4 students (4
male) to the fact that they are not interested in the
control systems & Matlab material. Furthermore 2
people from 11 (1 man, 1 woman) thought that the
scenario solution was difficult due to the difficulty
in Matlab use. In total 7 people from 11 agreed that
the test (scenario) conducted in Matlab in the control
systems course is satisfying, whereas 4 of them
thought otherwise (3 male, 1 female).
The scientific approach regarding the study of psy-
chological phenomena has wider application in the
fields of education, since a basic objective of the ed-
ucational process is to facilitate the process of learn-
ing and of Neuroscience, is the study of the nervous
system as a mediator of behaviour. The evaluation
using neuroscience methods and tools of happiness,
satisfaction of the trainee after the use of electronic
learning systems (e-learning platforms, e-games,
simulators) is an important issue of research. Fur-
thermore in the marine education, the use of more
objective evaluation methods of the modern educa-
tional environments will offer a qualitative im-
provement of the educational programs and at the
same time the re-development of the e-learning
methods that will benefit both the students and the
educational process (Blakemore and Frith, 2005,
Goswami, 2007, PareBlagoev, 2007).
Qualitative upgrading of the educational process
at university level marine education depends largely
on the instructive value of the trainers’ educational
software. Marine education software has developed
out of a specific initial implementation in an equiva-
lent manner to which the programs of practice and
training have been applied in cases of simulations
and programming environments. In most cases, edu-
cational software categories have developed without
taking into consideration any special pedagogical
theory (Tsoukalas et al., 2008).
In the present study, the evaluation of educational
software (simulator type) is conducted, in particular
MATLAB, which is used in the laboratory of control
systems of the engineers of the Merchant Navy from
the learnersperspective. The research takes place in
the School of Engineering of the Merchant Marine
Academy of Aspropyrgos in a first sample of 11 stu-
dents (target:21 students). Specifically, a biometric
tool of visual recording (a neuroscience tool) records
the trainees’ satisfaction from the educational use of
the simulator, by watching the users eye move-
ments in conjunction with questionnaires & inter-
views (Photo 1, 2).