127
1 INTRODUCTION
Ships have a significant environmental impact on
seaportareas.
Ports serve as entrance points for worldwide
maritime trade, however despite the sectorʹs
significant economic and social benefits, there are a
number of cases where it severely impacts the
environment and the health of EU inhabitants
[Passerini
etal2019]:
Ports and shipping can contribute to global
warmingbyemittinggreenhousegases.
Because 40% of Europeans live within 50
kilometersofthesea,airpollutionfromshipscan
harm both the marine ecosystem and human
health.
Pollution incidents, such as oil spills, can have
serious
consequences for the marine life and
inhabitantsoftheaffectedareas.
Accordingtostudy,underwater noise from ships
sailing over the ocean can cause hearing loss,
increasedstress,andbehavioralchangesinmarine
creatures.
Untreated ballast water aids in the spread of
speciesfromonemaritimeareato
another,altering
ecosystemsandthreateningnativemarinelife.
Ballast water is necessary for ships to operate
correctly. The International Maritime Organization
(IMO) has worked hard to eliminateboth accidental
andpurposeful shippollution.MARPOL, one of the
most important international agreements for the
maritime environment (International Convention for
the Prevention of
Pollution from Ships, 1973, as
revisedbytheProtocolof1978),hasbeenoneofthe
The Application of Mixed Reality and UAS Technology
in Port Decision-Making Process Based on PASSport
Project
L.Gucma
1
,B.Muczyński
1
,M.Bilewski
1
,M.Gucma
1
&M.Nisi
2
1
MaritimeUniversityofSzczecin,Szczecin,Poland
2
SistematicaS.p.A,Terni,Italy
ABSTRACT:Itisbecomingincreasinglyimportanttoprovideasolutionformanagingthepresentmissionand
status of autonomousand semi‐autonomous vehicles operatingon site in large and complex port locations.
Moreportsareimplementingdigitaltwinsystems,whichprovideacomplete3Dreconstructionofthe
portas
wellasreal‐timedataonallobjectsandactionsinprogress.Itischallengingtoprovideallrelevantdataina
waythatimprovessituationalawarenessanddecision‐making,resultinginimprovedmanagementandafaster,
moreeffectiveresponseduringanemergency.Thisisduetogreaterport
areasanddronesoperatingintheair,
onthewaterʹssurface,andunderneath.Tosolvethisissue,thePASSportinitiative,aprojectfinancedbythe
European Space Agency (EUSPA), is developing an innovative solution based on Mixed Reality (MR)
technology.Thesolutioncombinesreal‐timegeo‐taggedandEarth
Observationdatatoprovideenduserswith
enhanced3DrepresentationoftheportareaviaadedicatedHeadMountedDisplay(HMD)thatrecordsuser
locationandmovement.
http://www.transnav.eu
the International Journal
on Marine Navigation
and Safety of Sea Transportation
Volume 18
Number 1
March 2024
DOI:10.12716/1001.18.01.11
128
fundamentalendeavorsinthisarea.Itwasestablished
bytheInternationalMaritimeOrganization(IMO)to
reducepollutionoftheseasandoceans,includingthe
marine environment and adjacent air pollutants
included in Annex I, II, III, IV, V, and VI. Pollution
fromshipsinportsisclassifiedasfollows:
Particulate matter (PM), SOx (sulphur oxides),
NOx (nitrogen oxides), and greenhouse gases
(particulatematter)areexamplesofairpollutants.
Spills caused by ships, such as (accidental and
intentional) oil, chemical, and dry‐bulk leaks
duringloadingandunloadingprocedures.
Waste‐relatedwatercontamination.
Gray and black water
bilge water contamination
(e.g., sanitary facilities) (sinks, showers, kitchens,
andlaundries).
Contamination from water used for cargo hold
cleaning.
Pollutioncausedbysolidwaste(litter).
Invasivespecies,aswellasotherbiologicalmatter,
such as bacteria or viruses, contaminating ballast
water.
Noiseintheairand
beneaththewater.
Several studies have shown that more
internationallegislationareneededtoreviewvessel‐
relatedairpollutioncaused byshiptrafficemissions
[Lindgren,2021].
2 STATEOFTHEART
Both MR and UAV technologies have been used in
various scenarios in port environments. Drones,
includingUAS,arebeinginclude
asapartofso‐called
Industry4.0inportsandmaritimeindustrybringing
improvements in areas like ship‐shore packages
delivery, pollution monitoring and surveillance
[Zarzuelo,etal.2020].MRhasbeenusedmostlyasa
simulation technology that can improve immersion
andlowercostduringtrainingsandalso
asadesign
toolforshipbuilding[Sanchez‐Gonzalez,et al.2019].
DirectlyinportareasMRhasbeenimplementedasan
informationsharingplatform[Shuetal.2007]andfor
visualizationofaport’sdigitaltwin[Yaoetal.2021].
While there are notable examples of ports that
implementbothof
thosetechnologies,likeHamburg,
Rotterdam, or Shanghai, not much can be found in
scientific literature that would present a
comprehensivesystemutilizingUASandMRforport
operations,includingairpollutionmonitoring.
3 PASSPORTSOLUTION
The PASSport programʹs goal is to develop and
validate a method for expanding situational
awareness using fixed‐wing, rotary‐wing, and
underwaterdronesinportenvironments.
The EU Directive 2005/65/CE on Enhancing Port
Security calls for the establishment of surveillance
systems across the port area to considerably boost
securityandsafetyfordailyoperationsengagedinthe
port region. The directive applies to approximately
1000
Europeanports.Theproposedmethodintendsto
enhancepresentlyinuseplatformsbyextendingthe
surveillance perimeter with a fleet of drones. It
provides innovation and operational assistance for
identifying, managing, and analyzing safety and
securityaspectsofdailyoperations,withafocuson:
Pollutionmonitoring(environmentalprotection).
Supporttoe‐navigation(safety).
Critical buildings/ Infrastructures protection
(security).
Protection against non‐cooperative small craft
approachingtheportareas(security).
Underwaterthreatsmonitoring(security).
Theuniqueaspectoftheprojectistheuseofafleet
of partiallyautomated dronesthatintegratesGalileo
services (and other sensors)
for safe and effective
guidance,navigation,andcontrol(GNC)eveninthe
presenceofobstacleslikebuildingsandotherground
assets as well as potentially unfavourable weather
conditions.ThePASSportsystemdesignisshownin
Figure1,whichliststhesegmentsshownbelow:
PASSportAerialSegment(PAS),
PASSport
Ground Segment (PGS), composed by
mission(PMS)andControl(PCS)elements.
The purpose of PASSport is to ensure the
followingkeyfeatures:
Measuring player awareness of dangers and
increasing player awareness (security). The
PASSport platform suggests the design and
implementationofsuitableproceduresthatcanbe
used to fight threats
after the assets and
infrastructurewhichneedtobesecuredagainstthe
threats and risks of purposeful illegal activity
facing port activities are recognized. This is
accomplished after determining the risk level
(normal,increased,or high), and it is done so by
followingspecifiedmethodsandutilizingtechnical
toolsdesigned
withportsinmind.Thisenablesthe
proper response to be given to infrastructureʹs
potentialvulnerability.
Control and oversight of port areas (security and
safety). The PASSport platform offers a suitable
Human‐Machine Interface (HMI) to apply
pertinent procedures and monitor port security
andsafetyonaregularbasis.
Figure1.PASSportarchitectureforthepollutionmonitoring
(source:ownwork)
Data possessed by the Remotely Piloted Aircraft
Systems(RPAS)areanalysedatthePGSlevelinreal
timebyalocalcomputer.Inordertoprovideproper
positioning, RPASs are outfitted with high accuracy
129
GNSS receivers that leverage Galileo differentiators
suchasOSNMA(forpositionreliabilityandsecurity),
HAS/PPP (for positioning accuracy), and multi‐
frequency (for robustness and accuracy). These
technologies are combined with contemporary
robotics (vision‐based navigation, AI, and Deep
Learning algorithms) to ensure automated, secure,
andcontinuousoperations.
4 THE
VALIDATIONCAMPAIGNIN
KOLOBRZEG
Maintainingairandwaterqualityisadifficultyforall
ports and harbors. Mitigating difficulties entails
caring for the entire marine ecology as well as the
surrounding land. Ports and harbors are densely
populated industrial locations near the water. Many
activities, including boat repair, transportation,
terminal operations,
cargo handling, and storage,
have the potential to harm air/water quality,
especially if an incident occurs. In order to validate
thewaterandairpollutionmonitoringsystem,aswell
as the augmented reality system interface as a
situational awareness solution, the PASSPort team
conductedtestinginthePortofKoł
obrzeg(Figure2).
TheKołobrzegPortislocatedontheBalticSea,atthe
mouth of the Parsęta River. It performs a merchant
ship loading/discharging, fishing and passenger
function. The port has a several loading quay, two
shipyards, fishing harbour and two marinas. As a
result, the port of
Kołobrzeg recognized a pollution
monitoringconcernandviewsPASSportasasuitable
solutionfor:
Airqualitysurveillancemission.
Waterpollutionsurveillancemission
A3Dpointmapisgeneratedwitheachdatapoint
describingairandwaterqualityasmeasuredwithall
installed sensors. Colour coding and simple
alert
systemshouldbeimplemented.Themissionassumes
the surveillance flight aerial and marine
complementedwithadditionalEOdatafromsatellite.
Figure2. Port of Kołobrzeg. Layout. (1) – cargo handling
and administration area, (2) – fishing port and shipyard
area,(3)–yachtmarina
Theaimofthevalidationistoshowthesolutionto
the end user‐in this case the Kolobrzeg Sea Port
Authority. The system consists of the following
modules(Figure3):
A drone module consisting of flying drones
(UAVs)andfloatingdrones(USVs)withoperators
controllingtheiroperation.
Communication
systembasedontheInternetand
GSMcommunication(4G/5G).
Aserverresponsibleforprocessingdataincluding
alsodatafromCopernicusandpresentingittoend
usersanddecision‐makers.
A mixed reality system, which is an operator
decision support system enabling two‐way
communication between decision makers and
droneoperators.
Figure3. Architecture of the PASSport solution (1‐ flying
drone with operator, 2‐floating drone with operator, 3‐
server, 4‐decision‐making part with interface in mixed
realitytechnology)
5 PASSPORTPLATFORMVALIDATIONRESULTS
INKOOBRZEGPORT
The following equipment and services were used
duringthevalidationcampaign:
1. A rotary wings drone, i.e. DJI Matrice 300 RTK
(Figure 4). It provides a 30‐minute flight with a
maximum load of 3kg and the ability to operate
three sensors. Hot
‐swapping the battery, i.e.
withoutturningthedroneoff,allowsallintended
taskstobepracticallyaccomplished.Thedroneis
abletodefywindspeedsofupto15m/s.Itsspeed
duringflightisapproximately20m/s.Thedroneis
equippedwithanReal‐timekinematicpositioning
system (RTK)
to achieve in‐flight positioning
accuracy of up to 10 cm. A Sniffer 4D was used
[Kimet al. 2021] as themainair pollutionsensor
onthedrone,allowingthedetectionofparticulate
matter with different particle diameters, sulphur
oxides,nitrogenoxidesandozone(Figure6):
Particulatematter:
PM1(0.3‐10μm),
PM2.5(0.3‐10μm),
PM10(0.3‐10μm),
O3+NO2(0‐10ppm),
SO2(0‐10ppm).
The drone was additionally equipped with a
specialised Zenmuse H20t camera, which is an
integratedvisionandthermalimagingcamerathat
alsohasa
laserrangefinder.NoUVcameras(PCO‐
130
UVinthe190nm‐1100nmband)ortheMicaSense
multispectral camera (supporting as many as 10
bands)wereusedinthestudyduetotheinability
tosimulatespillage.
2. Adouble‐hulledUSVʹSharkyʹ(Figure5)designed
for hydrographic surveys in sheltered waters
(rivers, lakes, harbour basins,
lagoons) was used
duringthevalidation.The1mx0.85mvehiclehas
a laminated hull design allowing it to operate in
water temperatures in the full encountered range
of 1‐30 deg C. The freeboard height is 0.6 m and
theminimumdraughtof0.3mallowsitto
beused
intheshallowestareas.Thedisplacementofmax.
25kgallowsupto10kgofapparatustobefitted.
The electric propulsion motor (brushless direct‐
current motor‐BLDC) allows infinitely variable
speed control from 0 to 6 knots. The drone is
supplied with a single‐beam probe
and water
samplingkit.
Figure4. DJI Matrice 300 RTK drone with Sniffer 4D air
pollutionsensorandthermalimagingcamerainstalled
Figure5. The USVʺSharkyʺ floating drone in preparation
fortestingduringthevalidationcampaign
3. EO data from Copernicus. The scope of the
activities carried out within the PASSport project
was to evaluate and assess the usability and
applicability of Copernicus services to the
PASSport scenarios and to deduce attainable
performances applicable to PASSport‐related
applications. The great advantage of the
Copernicusprogramisrepresented
bythesynergic
useof different satelliteplatforms (ESAʹs families
ofdedicatedSentinelandContributingMissions),
hosting either active or passive sensors, and
observingdifferentportionsoftheelectromagnetic
spectrum. Each region of the electromagnetic
spectrum has its unique applicability for
observations. In particular, quality parameters
assessmentandmonitoring
usingSentinel‐5Phave
beenanalysedforthecampaignnKolobrzeg.
4. A dedicated application was created for the first‐
generation Microsoft HoloLens device, which is
oneofthemostadvancedmixedrealitytechnology
devices availableon the market, allowingfor full
3D holographic projection. Through its use, it is
possible
topresentathree‐dimensionalmapofthe
area with clean air measurement points, the
current position of the drone along with its
parameters.Configurablevaluesforstandardsand
alerts allow quick and accurate identification of
areaswherethereisariskofincreasedemissions.
Finally,aMixedRealitysolution
[Radanovicetal.
2022]hasbeenproposedasanexperimentalsolution
where data from multiple drones as well as any
external data source like Automatic identification
system (AIS) or Copernicus, can be integrated and
visualizedtogether.Forthepurposeofthecampaign,
a Microsoft HoloLens gen. 1 device has been used.
The application has been developed using following
softwareandSDK:
UnityEngine2019.4,
MicrosoftMixedRealityToolkit,
MicrosoftMapsSDKforUnity.
This particular set of tools makes it possible to
develop a solution that can be build for range of
hardware platforms in both AR and
VR technology,
including Microsoft Hololens gen. 1 and 2, Meta
Quest and SteamVR devices. Since the data is taken
directlyfromtheserveritispossibletofeedprocessed
andhistoricaldatainreal‐time(Figure7).
Figure6. Visualization of data from Kołobrzeg campaign
usingSniffer4DMapper™AnalyticSoftware
131
Figure7. Dedicated MR application. View from the Unity
EngineEditor
The presented architecture makes it possible to
presentthedatawitha1Hzsamplingrateinrealtime,
in 3D environment to any end‐used with an
authenticatedaccesstotheserver(Figure8).
Figure8.DataaspresentedintheMRapp.Screenshottaked
fromdesktopversionoftheapp(source:ownwork)
Due to the massive traffic involving the coastal
areaofKołobrzeg,themonitoringofairpollutionisa
paramount task in order to reduce the risk of
environmental health problems. Data has been
acquiredlocallyusingthedrones fleetequippedwith
theSniffer4Dsensors(Figure9).
Figure9. In situ data using the fleet of drones equipped
withSniffer4Dsensors.
6 CONCLUSIONSANDFUTUREWORK
ThisarticlepresentsPASSportʹsvalidationcampaign
intheportofKolobrzeginordertoevaluatethewater
and air pollution monitoring system and the
augmented reality system interface as a situational
awareness solution. This was the first in a series of
validation initiatives for the
PASSport project. The
following validation campaigns will take place in
Hamburg(forcriticalinfrastructure),Valencia(fore‐
Navigation),LeHavre(forsmall,recalcitrantvessels),
andRavenna(forunderwaterrisks).
In‐situ data from several GPS‐referenced drones,
as well as any external data sources,such as AIS or
Copernicus, are
merged and presented in a Mixed
Realityapplication,whichhasbeenmadeavailableas
anexperimentalsolution.
Future work will address a variety of other
research,operational,andengineeringissuesinorder
tofullyutilizeandimplementthedevelopedsystem
and drones for port surveillance and pollution
management.Amongtheseissues
are:
Automated monitoring‐Drones can perform
monitoring tasks autonomously, but safety is
crucialinthiscircumstance,thusdronesmustfly
over water while keeping a safe distance from
ships and other objects to avoid endangering
peopleofsurroundingstructures.
Developmentofamulti‐sensorheadforscanning
port
environmentalcontaminants(combinationof
opticsandothersensorheads).
Such a multisensor platform can be built using
OEMcomponents.
Aproposedoilspilltrackingandforecastinitiative
includes the development of a multisensor
platformwithintegratedoptics,laserfluorescence,
and a thermal imaging camera for tracking
already‐spilledoil.

Monitoringtriggeredbythe operator.Performing
certain tasks as directed by the operator.
Circumnavigatinganobject,stationarysuspension,
etc.
Physicalsamplingofwater.Landingandtakingoff
fromwater.
Assisting in firefighting operations and dealing
withchemicalandoilaccidents.Informationonan
oilspillisprovided.
Monitoringofoilspotsorfires
online.
Giving testimony in court about air and water
pollution.
Drone approach to a moving vessel collecting
exhaust samples utilizing AIS data and visual
observation by the operator. It is also in great
demandintermsofsafety.
Keepaneyeonthe
yachtasitapproachesthequay
forsignsofexcessivemooringenergyanddamage
tothefendersandquay.
ACKNOWLEDGEMENTS
The PASSport project, Operational Platform managing a
fleet of semi‐autonomous drones exploiting GNSS high
AccuracyandAuthenticationtoimproveSecurity&Safety
in port areas, has received funding from the European
UnionAgencyfortheSpaceProgramme(EUSPA)underthe
European Union’s Horizon 2020 research and innovation
programmeundergrant
agreementNo101004234.
REFERENCES
KimM.,JangY.,HeoJandParkD.2021,AUAV‐BASED
AIR QUALITY EVALUATION METHOD FOR
DETERMINING FUGITIVE EMISSIONS FROM A
132
QUARRY DURING THE RAILROAD LIFE CYCLE.
Sensors,https://doi.org/10.3390/s21093206
Lindgren S. 2021, THE COAST IS CLEAR: SHIPPING
EMISSION STANDARDS, AIR QUALITY AND
INFANT HEALTH, Transportation Research Part D:
TransportandEnvironment,
https://doi.org/10.1016/j.trd.2021.103067
Radanovic M., Khoshelham K. and Fraser C.  2022,
VIRTUALELEMENTRETRIEVALINMIXEDREALITY
ISPRS Annals of the Photogrammetry, Remote
Sensing
andSpatialInformationSciences,
Passerini, G., Rodriguez G. and Ricci, S. 2019, Maritime
Transport,WitPress.Southampton‐Boston.
Sanchez‐Gonzalez,P‐L,Díaz‐Gutiérrez,D.,LeoT.J.,Núñez‐
Rivas L.R. 2019, TOWARD DIGITALIZATION OF
MARITIME TRANSPORT? Sensors 19, no. 4: 926.
https://doi.org/10.3390/s19040926
Shu, F., Mi, W., Xu, Z. 2007 THE INFORMATION
SHARING PLATFORM FOR PORT CONTAINER
TERMINAL LOGISTICS USING VIRTUAL REALITY,
IEEE International Conference on Automation and
Logistics,Jinan,China,pp.2570‐2575,doi:
10.1109/ICAL.2007.4339013
Yao, H., Dachuan W., Mengchao S., Yue Q. 2021,
APPLICATION OF DIGITAL TWINS IN PORT
SYSTEM,JournalofPhysics:ConferenceSeries1846(1):
012008.https://doi.org/10.1088/1742
‐6596/1846/1/012008.
Zarzuelo, I. de la Peña, Soeane, M. J. F., Bermúdez, B. L.
2020, INDUSTRY 4.0 IN THE PORT AND MARITIME
INDUSTRY: A LITERATURE REVIEW, Journal of
Industrial Information Integration, Volume 20, 100173,
ISSN2452‐414X,https://doi.org/10.1016/j.jii.2020.100173