International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 5

Number 4

December 2011

541

1 INTRODUCTION

The effectiveness evaluation of the Information &

Communication Technology (ICT) in the didactics

was mainly based on the experience & analysis (pos-

itivistic) methods, which accept that knowledge may

be attributed only to the objective reality existing re-

gardless of the values and beliefs the ones seeking to

discover her. Methods of the physics and behavioral

sciences are adopted, as well as objective forms of

knowledge and deterministic acknowledgements for

the human nature. As it shown in the international

bibliography, the use of multiple methods of evalua-

tion is more effective and the combinatorial use of

quantitative and qualitative approaches confines

their weaknesses (Brannen, 1995, Bryman, 1995,

Patton, 1990, Retalis et al., 2005). That combination

may bring out multiple applications of the ICT in the

educational sector, thus contributing in a more “suf-

ficient” evaluation of an application. According to

Hubermas (1971), the final target of the ICT effec-

tivity evaluation on the educational procedure must

call for the examination and the evaluation of the

three interests on knowledge: the technical (suggests

the scientific opinion – experience & analysis exam-

ple), the practical (the interpretation methods that of-

fer knowledge that serves the “practical” interest on

the understanding and interest) and “manumission”

(offers the necessary critical and dialectical basis for

the substantial connection between theory and ac-

tion). Knowledge is the nucleus and around should

orbit the evaluation procedure as well as the result of

the human action that is defined by the physical

needs and interests that lead and at the same time

shape the way the knowledge is structured in several

human activities. It is more than obvious that all the

above conclude to the fact that the evaluation targets

and the interest for knowledge define the evaluation

mode and its results (Carr and Kemmis, 1997, Hu-

bermas, 1971, Retalis et al., 2005).

In the evaluation of the tutorial systems, the sys-

tem’s term “utility” is analyzed in two supplemen-

tary components: the utility offered to the final re-

garding the system’s efficiency and the usability

regarding the facility of the users to comprehend or

use that usability. These two senses are bound to-

gether but it’s not necessary one to exist without the

other (Grudin, 1992). In the evaluation, in particular,

is widely used the term “manageability” which is a

self-evident requirement for all the systems and tools

managed by men (Avouris, 2000, Avouris et al.,

2001, Nielsen, 1993, Papachristos et al., 2010).

One of the problems emerging by the evaluation

techniques is the fact that they are based on the ob-

servation and on the users’ answers. Today a great

interest is imposed on the application of the objec-

tive usability testing. This testing mainly concerns

the observation of the eyes and the measurement of

physical data related to several part of the human

physiology (heart activities, activity of perspiration

glens, electric activity of the muscles and brain). Re-

cording the measurements requires the application of

several organs and sensors on the users. The exist-

ence and correspondence of physiological measure-

ments patterns in specific emotional situations and in

Evaluation of Educational Software for Marine

Training with the Aid of Neuroscience

Methods and Tools

D. Papachristos & N. Nikitakos

Dept. Shipping, Trade and Transport, University of Aegean, Greece

ABSTRACT: The evaluation with the use of neuroscience methods and tools of a student’s satisfaction –

happiness from using the e-learning system (e-learning platforms, e-games, simulators) poses an important re-

search subject matter. In the present project it is presented a research on course conducted in the Faculty of

Merchants Officers, Marine Academy of Aspropyrgos. In particular, this research with the use of a neurosci-

ence tool-gaze trucker, investigates the amount of satisfaction of the students using a simulator by monitoring

the users’ eye movement in combination with the use of qualitative and quantitative methods.

542

general the determination of a theoretical frame of

interpretation that defines a user’s emotional reac-

tion in an interface is a state of art research field

(Dix et al., 2004, Picard, 1997, Retalis et al., 2005).

Today the use of the consolidated strategic re-

search that is called cognitive neuroscience and in-

cludes the study of the behavior and the external sit-

uations related to it, as well as the expansion of the

nervous system mechanisms that intervene in this re-

lationship, leads to a better expansion of the user’s

physiological reactions. In the marine education and

training, in particular, the modelization of the stu-

dent’s satisfaction based on objective standards (bi-

ometrical measurements) that present a better frame

of interpretation is an important research subject.

Via cognitive neuroscience is possible to determine

a background that explains the satisfaction effect and

at the same time presents new considerations that

will expand the existing so far educational conclu-

sions regarding the adults’ education. The coexist-

ence of the obtainment and understanding of

knowledge and the acquisition of skills (according to

specific standards) necessary to practice the marine

profession is a crucial element for the marine profes-

sionals. In addition it will promote the design im-

provement of the e-learning software programs and

will help the e-learning software develop-

ers/manufacturers to improve their products on the

terms of the learning success and effectiveness

(Goswami, 2007, IMO, 2003, Papachristos and Ni-

kitakos, 2010, Kluj, 2002).

2 RESEARCH METHODOLOGY

The recommended research procedure aims at the

modelization of the (subjected) satisfaction of the

students – users in marine education via the user in-

terface evaluation of the MATLAB simulator. The

experiment is conducted in the Faculty of Merchants

Officers, Marine Academy of Aspropyrgos with a

random sampling among the students. The experi-

ment’s researching purposes concern the evaluation

of the students’ satisfaction from the MATLAB

simulator use and the educational evaluation of

MATLAB from the user’s point of view.

The research that will be conducted is a combina-

tion of a qualitative – quantitative methodology on

one hand and a use of neuroscience tools (gaze

trucker use) on the other hand. The purpose of that

is to combine the positive elements of the corre-

sponding methodologies: targeting at measurable re-

sults & variable testing (quantitative), interpretative,

explanatory (qualitative) and more objective meas-

urements by “observation” of physiological data of

the user (neuroscience tools).

For the quantitative research, the questionnaire

method will be applied in order to extract measura-

ble results regarding the educational evaluation of

the MATLAB by the users in combination to the el-

ements that satisfy the user by the simulator’s opera-

tion. The use of the qualitative research via semi-

structured interviews was chosen in order to exam-

ine the deeper reasons of the users’ satisfaction of

the simulator and verify the neuroscience tool meas-

urements and the quantitative research results. The

use of the gaze trucker was chosen because it pre-

sents more “accurate” measurements. The course of

the researching procedure in the research conduction

is shown in the following figure (Fig.1).

Figure 1. Research steps.

The determination of the satisfaction level con-

cerns the following parameters: system’s usability

regarding the whole system (total usability), as well

as the corresponding learning scenario, the stimulant

for the active participation of the users (from the

system and the learning scenario) and the user’s

friendliness regarding interface. The emerging of

conclusions from the research results concludes the

evaluation procedure that concerns the “identifica-

tion” of the measurements from the gaze trucker in

relation with firstly the questionnaire results (educa-

tional evaluation) and secondly and most important-

ly the interview results (educational evaluation &

determination of the user’s satisfaction data) (Fig.

2). The evaluation’s intention is to create measure-

ment patterns that correspond to psychological char-

acteristics. The learning scenario applied was devel-

oped according to the STCW’ 95 corresponding

educational specifications on the Merchant Officers

education in collaboration with the staff of the Ma-

rine Academy of Aspropyrgos.

543

Figure 2. Verification procedure.

3 OPTICAL MEASUREMENT TECHNIQUE

The human vision/eyesight is an extremely compli-

cated activity with physical restrictions as well as re-

strictions concerning perception. The optical percep-

tion is distinguished in the physical reception of a

stimulant from the outer world and the pro-

cessing/interpretation of the certain stimulant. The

observation of the gaze trucking as well as the pupil

is a possibility existing in many years now but the

technological developments in both material equip-

ment and software made it more viable, mostly as an

approach to measure usability (Duchowski, 2003,

Dix et al., 2004). The eye movements are supposed

to illustrate the amount of cognitive elaboration a

screen demands and therefore the difficulties and fa-

cilities in its processing. In general the optical meas-

urement focuses on the following (Goldberg and

Kotval, 1999, Dix et al., 2004): the focus points of

the eyes, the eye movement patterns and the varia-

tions of the eye pupil. The methodology of effectua-

tion of the optical measurement that was chosen on

the present experiment aims at the eye movement

observation and the user’s head movement regarding

with time. For carrying out of the experiment the

“Face Analysis” software developed by the Image,

Video and Multimedia Systems Lab (IVML) του

National Technical University of Athens (NTUA) is

applied in collaboration with a Web camera con-

nected to the computer hosting the research subject

(MATLAB) (Asteriadis et al., 2009). That software

records (parameters): (a) Eye gaze vector: vertical

and horizontal movements (2 floats), (b) Head Pose

Vector: pitch, yaw (2 floats), pitch and yaw come in

normalized floats, (c) Dist_monitor: Float indicating

distance of the user from the monitor (~1: fixed dis-

tance, <1 goes far, >1 comes close)(1 float) and (d)

Head roll: roll comes in degrees (1 float). The whole

data recording and analysis procedure is shown in

the following figure (Fig.3).

Figure 3. Optical Data Recording and Analysis procedure.

4 FIRST RESULTS

The first (random) sampling was carried out on the

January 2011, in the Information Technologies La-

boratory of the Marine Academy of Aspropyrgos.

The samples consisted of 11 students (9 Male, 2

Female) that were subjected to a specific experi-

mental procedure (research methodology) (Table 1).

The sample’s learning – medical profile as shown in

Table 2 presents homogeneity, whereas from the

first processing of the Matlab gaze trucker data

emerge useful information (Table 3, 4).

Table 1. Structure of Sample

___________________________________________________

Variables Male (2)

Female (2)

___________________________________________________

Semester E (9) E (2)

Pass “Control System & - -

Matlab” course

Study “Control System & (9) (2)

Matlab” course

No pass ”Control System & - -

Matlab” course

Level computer using

(No Use, Basic, Medium, M(6),B(2)A(1) B(1), M(1)

Advance)

___________________________________________________

(frequency)

Table 2. Structure of medical-learning profile

___________________________________________________

Question theme Male (2)

Female (2)

___________________________________________________

trabismus N*(9) N(2)

Monochromatism N (9) N (2)

Eye disease N (8), N,

Y(1-myopia,astigmatism) Y*(1-myopia)

Eye operation N(9) N(2)

Dyslexia N(8), Y(1) N(2)

ADHD N(9) N(2)

___________________________________________________

N: No,

Y: Yes,

(frequency)

Table 3. 1

Data Set of Sample.

___________________________________________________

Time (min) Eye gaze vector Eye gaze vector

(aprox.) vertical horizontal

___________________________________________________

Average F*(13.5), F*(0), F*(-.16),

M*(8.5), M*(0), M*(7.04),

T*(9.4) T*( 0) Τ

(6.16)

___________________________________________________

Standard F*(2.12), F*(0), F*(0.54),

Dev M*(3.3), M*(0), M*(31.81),

T*(3.67) T*(0) T*(29.7)

___________________________________________________

F*:Female, M*:Male, T*:Total

544

Table 4. 2

Data Set of Sample.

__________________________________________________________________________________________________________

Dist_monitor Head roll Head pose Head pose

vector pitch vector yaw

__________________________________________________________________________________________________________

Average F*( 1.2), F*(-1.2), F*( 0.04), F*(-0.16 ),

M*(1.23), M*( 2.03), M*( 0.04), M*( -0.04),

T*(1.22) T*( 1.14) T*(0.04 ) T*( -0.07)

Standard Dev F*(0.29), F*(7.90), F*( 0.31), F*( 0.54),

M*(1.07), M*(10.12), M*(0.25 ), M*( 0.35),

T*(0.92) T*(9.67 ) T*( 0.21) T*( 0.41)

__________________________________________________________________________________________________________

F*:Female, M*:Male, T*:Total

The first measurements show that men needed

less time to execute the scenario than women (the in-

terview showed that women are more informed

about Matlab and the control systems and answered

in more questions than men). In the distance from

screen observation (dist_mon variable) it was rec-

orded that both men and women approach the screen

(>1), which means that they both fully watching the

scenario and have difficulty in using Matlab. As the

screen distance increased in men, they seemed to

have a more attention of the scenario and the Matlab

use, even though they don’t seem to achieve the

same success in the scenario solving case as the

women. In the next variable, we observe a non-

balance in the inclination of the head (~0) in both

sexes, which probably suggests a deeper inspection

of the Matlab interface (that is probably ought to the

lack of more practice time). The eye gaze vector ap-

proximately shows (the females marginally watch

inside the screen although the sample is small), the

students watch inside the screen and not outside of it

(there is an interest on the scenario execution with

the help of Matlab software or search about that).

The interview shows that 7 people from a total of

11 consider that Matlab software improves teaching

the control systems (6 male, 1 female), all of them

consider the Matlab interface to be friendly to the

user and that it is necessary to have a basic mathe-

matics background in order to use it. Without a

mathematics background, its operation is difficult.

During the execution of the learning scenario (closed

control system solving exercise and calculation of

time response with a paced entrance) none of the 11

students could get to a point of solution, rendering it

to the lack of necessary time practice of Matlab in

the School’s educational program and 4 students (4

male) to the fact that they are not interested in the

control systems & Matlab material. Furthermore 2

people from 11 (1 man, 1 woman) thought that the

scenario solution was difficult due to the difficulty

in Matlab use. In total 7 people from 11 agreed that

the test (scenario) conducted in Matlab in the control

systems course is satisfying, whereas 4 of them

thought otherwise (3 male, 1 female).

5 DISCUSSION

The scientific approach regarding the study of psy-

chological phenomena has wider application in the

fields of education, since a basic objective of the ed-

ucational process is to facilitate the process of learn-

ing and of Neuroscience, is the study of the nervous

system as a mediator of behaviour. The evaluation

using neuroscience methods and tools of happiness,

satisfaction of the trainee after the use of electronic

learning systems (e-learning platforms, e-games,

simulators) is an important issue of research. Fur-

thermore in the marine education, the use of more

objective evaluation methods of the modern educa-

tional environments will offer a qualitative im-

provement of the educational programs and at the

same time the re-development of the e-learning

methods that will benefit both the students and the

educational process (Blakemore and Frith, 2005,

Goswami, 2007, Pare – Blagoev, 2007).

Qualitative upgrading of the educational process

at university level marine education depends largely

on the instructive value of the trainers’ educational

software. Marine education software has developed

out of a specific initial implementation in an equiva-

lent manner to which the programs of practice and

training have been applied in cases of simulations

and programming environments. In most cases, edu-

cational software categories have developed without

taking into consideration any special pedagogical

theory (Tsoukalas et al., 2008).

In the present study, the evaluation of educational

software (simulator type) is conducted, in particular

MATLAB, which is used in the laboratory of control

systems of the engineers of the Merchant Navy from

the learners’ perspective. The research takes place in

the School of Engineering of the Merchant Marine

Academy of Aspropyrgos in a first sample of 11 stu-

dents (target:21 students). Specifically, a biometric

tool of visual recording (a neuroscience tool) records

the trainees’ satisfaction from the educational use of

the simulator, by watching the users’ eye move-

ments in conjunction with questionnaires & inter-

views (Photo 1, 2).

545

6 CONCLUSIONS

A research that offers an objective data recording

concerning the emotional state of a user that affects

his/her problem solving ability and carrying out of

projects poses an important challenge. Overall the

first results indicate the following so far: (a) possible

relationship between the scenario comprehension

and the Matlab use with the head being in a distance

from the screen, (b) the head’s inclination shows the

expansion of the Matlab use among the users, indi-

cating an increase in training time of the Matlab use

(modification of the educational program and the

corresponding standard STCW'96) and (c) possible

relationship between the eye gaze vector (screen

monitoring) and the users’ satisfaction (interest =

satisfaction).

The research continues with the numeral increase

of the sample and the total processing and evaluation

of the research findings (qualitative and quantitative

data).

REFERENCES

Asteriadis, S. Tzouveli, P. Karpouzis, K. Kollias, S. 2009. Es-

timation of behavioral user state based on eye gaze and

head pose—application in an e-learning environment, Mul-

timedia Tools and Applications, Springer, Volume 41,

Number 3 / February, pp. 469-493.

Avouris, N. Tselios, N. Tatakis, E. C. 2001. Development and

Evaluation of a Computer-based Laboratory teaching tool.

Journal Computer Applications in Engineering Education,

9(1), 8-19.

Avouris, N. 2000. Human-Computer Interaction:An Introduc-

tion. Athens:Diavlos (in greek).

Blakemore, S. J. Frith. U. 2005. The Learning Brain: Lessons

for Educations. Oxford:Blackwell.

Brannen, J. 1995. Combining qualitative and quantitative ap-

proaches: An overview, J. Brannen (ed.), Mixing Methods:

Qualitative and Quantitative Research. UK:Avebury, 3-38.

Bryman, J. 1995. Quantitative and qualitative research:further

reflections on their integration, Mixing Methods: Qualita-

tive and Quantitative Research. UK:Avebury, 57-80.

Dix, A. Finlay, J. Abowd, G. D. Beale, R. 2004. Human-

Computer Interaction, UK:Pearson Education Limited.

Duchowski, A. T. 2003. Eye Tracking Methodology:Theory

and Practice, London: Springer-Verlag.

Habermas, J. 1971. Knowledge and Human Interest.

USA:Beacon Press.

Goldberg, J. H. & Kotval, X. P. 1999. Computer interface

evaluation using eye movements:methods and constructs.

International Journal of Industrial Economics, 24:631-45.

Goswami. U. 2007. Neuroscience and education:from research

to practice? Nature Review Neuroscience, 7:406-413.

Grudin, J. 1992. Utility and usability: Research issues and de-

velopment contexts. Interacting with Computers, 4(2), 209-

217.

IMO-International, Maritime Organization, 2003. Issues for

training seafarers resulting from the implementation

onboard technology, STW 34/INF.6.

Kluj, S. 2002. Relationship between learning goals and proper

simulator, ICERRS5 Paper.

Nielse, J. 1993. Usability Engineering. UK:Academic Press.

Papachristos, D., Alafodimos, N., Zafeiri, E., Sigalas, I. 2010.

Educational evaluation of Matlab simulation environment

in teaching technological courses:The example of Digital

Control Systems, Proceedings of the Hsci2010 - 7th Inter-

national Conference on Hands-on Science, The University

of Crete campus at Rethymno, July 25-31, 2010, pp.168-

174, http://www.clab.edc.uoc.gr/HSci2010/.

Papachristos, D. Nikitakos, N. 2010. Application Methods and

Tools of Neuroscience, in Marine Education, Conference

Proceedings of Marine Education & Marine Technology,

30 November – 1 December, Athens, pp.177-190,

www.elint.org.gr.

Pare-Blagoev, J. 2007. The neural correlates of reading disor-

der: functional magnetic resonance imaging. In K.W.

Fischer, J. H. Bernstein and M. h. Immordino-Yang (Eds),

Mind, Brain, and Education in Reading Disorders. Cam-

bridge:Cambridge University Press, 148-167.

Patton, M. Q. 1990. Qualitative Evaluation and Research

Methods. CA:Sage Publications.

Picard, Ρ. 1997. Affective Computing, MIΤ Press,

USA:Cambridge MA.

Retalis, S. (eds.), 2005. Educational Technology. The advanced

internet technologies in learning service, Ath-

ens:Kastaniotis Editions (in greek).

Tsoukalas, V. Papachristos, D. Mattheu, E. Tsoumas, N. 2008.

Marine Engineers’ Training: Educational Assessment of

Engine Room Simulators, WMU Journal of Maritime Af-

fairs, Vol.7, No.2, pp.429-448, ISSN 1651-436X, Current

Awareness Bulletin, Vol. XX-No.10, Dec. 2008, IMO Mar-

itime Knowledge Centre, pp.7.

Photo 1. Biometric tool (web camera & software ‘Face Analy-

sis’) in action (back side).

Photo 2. Biometric tool (web camera & software ‘Face Analy-

sis’) in action (side face)