This document describes the folder structure to be used to store BEAST ROV data in such a way, that it can be directly ingested by the AWI Raw Data Ingestion Framework. All Path examples are given for the Polarstern expedition PS101. Data structure will adapt to changes in the vehicle composition

Katlein, Christian

Electronic Publication Information Center

Data description of ROV BEAST Last updated: 4. Nov 2016 Author: Christian Katlein, AWI-Sea-Ice-Physics, ckatlein@awi.de  This document describes the folder structure to be used to store BEAST ROV data in such a way, that it can be directly ingested by the AWI Raw Data Ingestion Framework. All Path examples are given for the Polarstern expedition PS101. Data structure will adapt to changes in the vehicle composition. Planned changes for PS106 in 2017 with unclear dataflow: Additional/Alternative LBL positioning, Macro Camera, 3rd RAMSES … . Updated data description will only be available after each survey. Document history: Date Changes Author/Editor 15. Nov 2016 Initial version Christian Katlein        First Level: Expedition/Campaign \PSXXX {alphanumeric expedition/campaign identifier}  \PS101 (Path examples in grey font)In this folder all data from the respective campaign is saved. It contains the following subfolders: \PSXXX\PSXXX_all_plots a general folder containing a variety of image files with quick view plots (no particular structure or format, all images in standard fileformats like .jpg or .png)  PSXXX\PSXXX_analysis a general folder containing a variety of image files with quick view plots as well as quick look processing code that is not part of the standard data processor (no particular structure or format, all images in standard fileformats like .jpg or .png) PSXXX\PSXXX-XXX_YYYYMMDD  PS101\PS101-057_20160915arbitrary amount of folders, one for each station containing the data sorted by station; data structure is kept consistent  Second Level: Station \PSXXX\PSXXX-XXX_YYYYMMDD  {alphanumeric  station number _ station date}  \PS101\PS101-057_20160915In this folder all data from one station are stored. It always contains the following files \PSXXX\PSXXX-XXX_YYYYMMDD\ROVSettings_PSXXX_XXX_YYYYMMDD_1.m  this is the settings file containing all path-, calibration and offset information needed by the Processor. Multiple files may exist if the station contains more than one survey. In that case, files are numbered with the numeric value after the last underscore according to the survey numbering. File is only present when data are processed. \PSXXX\PSXXX-XXX_YYYYMMDD\ PSXXX_XXX_ 1.mat  this is the MATLAB  file containing all (preprocessed) data. Multiple files may exist if the station contains more than one survey. In that case, files are numbered with the numeric value after the last underscore according to the survey numbering. File is only present when data are processed. The following subfolders can exist (only the first one – 1 survey folder – is mandatory): \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_1  The Spot.On survey folder generated by the Spot.On survey software containing all survey data. Multiple folders may exist if the station contains more than one survey. In that case, files are numbered with the numeric value after the last underscore according to the survey numbering. data structure is kept consistent \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_multibeam  The folder contains all multibeam data from this station. Data is saved in .D1R raw files, as well as .D1P point files. Both data formats are binary manufacturer formats that can be opened with the DT101.exe software The .D1P format is documented in the „IMAGENEX DT100 - D1P PROFILE POINT DATA FORMAT“ Document provided by the manufacturer Imagenex. Filenames and sizes vary according to survey settings. Folder only exists where multibeam system was operated. \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_still_images  The folder contains all upward looking still images from the TigerShark Camera taken during the respective station. Data is saved in standard .JPG files, with the filename ‘IMG_XXXX.JPG‘. Filenames and sizes vary according to camera settings. Folder only exists where still camera system was operated. \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_plots  The folder contains various quicklook plots. Images are saved in standard .png files Filenames and sizes vary according to processing state and settings. \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_auxiliary  The folder contains arbitrary additional information that does not fit into the standard data format and is not consistently occurring but deemed essential for the dataset. This might be further sensor raw data, videos, documentation etc. that should be archived together with the main ROV data. no particular structure or format  Third Level: Spot.On Survey \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_1  {numeric date _ survey number} \PS101\PS101-057_20160915\20160915_2 This folder contains all data from one survey that is saved by the Spot.On survey system. Folder structure is given by the Spot.On software and can not be changed. This folder can be opened directly in the Spot.On survey software to view the data. Surveys have to be named in Spot.On accordingly: YYYYDDMM_survey# e.g.  20160915_2 The folder can contain the following log files from the SeaNet pro Survey system containing position and sonar raw data: \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_#\ddd_dd_mmm_HH_MM.V4LOG   \PS101\PS101-238_20161015\20161015_1\Sat_15_Oct_05_18.V4LOGThis is the raw recording of the SeaNet-Pro survey software in a proprietary SeaNet Pro (Tritech, UK) data format. This data is only recorded if recording is specifically selected in SeaNet Pro which will mostly be related to fault searching at some of the ROV-Dives The folder can contain the following folders \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_#\Actors  The folder contains all data logged by Spot.On. Filenames and sizes vary according to stream settings and vehicle configuration \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_#\SurveyReport  The folder contains data generated for the automated survey report. Folder only exists if Report was generated in Spot.On \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_#\ViewerData  The folder contains data used for the Spot.On Viewer. Folder only exists if survey was exported and viewed in viewer mode.  4th Level: Actors \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_1\Actors   This folder contains no file but two folders “Rov” and “Ship”. In most cases only “Rov” will contain data if the ROV is used in a “fixed platform mode” on drifting sea-ice. If the System is used as classical ROV e.g. on Heincke, then the “Ship” folder will contain Ship navigation data.  5th Level: ROV Data \PSXXX\PSXXX-XXX_YYYYMMDD\YYYMMDD_1\Actors\ROV   This folder contains no file but two folders “Rov” and “Ship”. In most cases only “Rov” will contain data if the ROV is used in a “fixed platform mode” on drifting sea-ice. If the System is used as classical ROV e.g. on Heincke, then the “Ship” folder will contain Ship navigation data. This is the main Datafolder of the ROV system. The content varies dependent on connected sensors. Filenames should be consistent as far as possible. However as streamnames are partly entered manually they can vary and additional sensors will appear in the future! All data in this folder needs to be archived!  In general the filenames are named following the scheme: TypoOfData-StreamName-StartTimestamp.so*.  Most data files are ASCII datafiles of open format logging serial streams. The Video files are saved by Spot.On as .mpg (MP4 using the h.264 codec, encoded by x264vfw which can be found under http://sourceforge.net/projects/x264vfw/). Each file can be present several times if the survey recording was paused during operation. Then the files of each stream need to be concatenated for processing. The following will explain each file format: …\YYYMMDD_#\Actors\Rov\actorinfo-Rov-*.soai2  Spot.On Actor-Info file, an XML file containing some information about the ROV. Filename does not represent the survey time. …\YYYMMDD_#\Actors\Rov\attentionpoints-Rov-YYYY-MM-DDTHH-MM-SS-FFF.soap1  XML-file containing all the information on Attention points as marked during the survey. Each Attention Point item contains TimeStamp, Note and Category Tags.  Filename is given by Spot.On and always stays consistent. …\YYYMMDD_#\Actors\Rov\generalserial-ALTIMETER-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with Data from the upward looking altimeter measuring the distance to the ice of the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.48.916 | 010.000m] [Timestamp | Altimeter reading in m]; Erroneous readings are 10.0 and 9.999 m  Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. …\YYYMMDD_#\Actors\Rov\generalserial-ECO-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with Data from the WetLabs ECO-Triplet Fluorometer of the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.49.112 | 99/99/99 99:99:99 700 4130 695 54 460 50 584] [Timestamp | 99/999/99 99:99:99 BackscatterWavelength BackscatterCounts ChlorophyllWavelength ChlorophyllCounts FDOMWavelength FDOMCounts UndocumentedCounts] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. …\YYYMMDD_#\Actors\Rov\generalserial-GPCTD-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with Data from the SeaBird Glider-Payload CTD (GPCTD) of the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.49.317 |     1.54, -2.2081,-0.00001, 3199.84] [Timestamp |     Pressure(db), Temperature(°C), Conductivity(S/m), DissolvedOxygenFrequency(Hz)] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. …\YYYMMDD_#\Actors\Rov\generalserial-pH-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with Voltage Data from the SeaBird pH sensor (needs post survey conversion from mV to pH) of the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.49.612 | 2611mV] [Timestamp |   SensorOutput(mV)] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. …\YYYMMDD_#\Actors\Rov\nitrate-SUNA-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with Data from the Satlantic SUNA nitrate sensor. The Sensor can be configured to send either a short (SATSSLC) or long (SATSLF) data message of the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.51.108 | SATSLC0847,2016275,8.378725,17.46,0.2445,1.9988,2.1251,0.00,0.000872] [Timestamp | SATSLCnnnn, YearDayOfYear, HoursOfDay, NitrateConcentration(μM), NitrogenInNitrate(mg/l), Absorbance@254nm, Absorbance@350nm, BromideTrace(mg/l), FitRMSE] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. File Format depends on Sensor setting …\YYYMMDD_#\Actors\Rov\position-filtered-YYYY-MM-DDTHH-MM-SS-FFF.sofp4 ASCII file filtered and interpolated (smoothed) USBL vehicle positions (virtual geographic coordinates) in the following format: 1 Header line: [SPOT.ON fixed size position format version 4         ] followed by datalines: [2016.10.01T08.28.50.293     6.145493601E+001     8.686840133E+001     0.000000000E+000] [Timestamp    Longitude     Latitude      Depth(unused)] Filename is given by Spot.On and is always consistent . …\YYYMMDD_#\Actors\Rov\position-main-YYYY-MM-DDTHH-MM-SS-FFF.sofp4 ASCII file of unfiltered raw USBL fix vehicle positions (virtual geographic coordinates) in the following format: 1 Header line: [SPOT.ON fixed size position format version 4         ] followed by datalines: [2016.10.01T08.28.50.293     6.145493601E+001     8.686840133E+001     0.000000000E+000] [Timestamp    Longitude     Latitude      Depth(unused)] Filename is given by Spot.On and is always consistent . …\YYYMMDD_#\Actors\Rov\PSXXX-XXX_renavigation.txt ASCII file with renavigated and interpolated Navigation Data from the Spot.On survey system (for use in Multibeam processing)in the following format: 1 Header line: SpotOnTime Latitude[°] Longitude[°] X_Position[m] Y_Position[m] Depth[m] Roll[°] Pitch[°] USBL-COG Mag-Head TimeToPing[s] followed by datalines: 2016-10-01T09:13:40:024 86.868399 61.458916 24.165494 -0.244622 2.700000 0.000000 -0.200000 67.062653 187.997071 -18.671001 [explanation see header line] Filename is given according to stream name by Spot.On and only stays consistent if processing sticks to the convention. File is only present when data are processed. …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_IRRAD-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with raw data from the RAMSES Irradiance sensor (onboard the ROV) in the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T10.03.57.240 | 2048 1239 1294 … 1097 1138] [TimeStamp | IntegrationTime(ms) RawCountsOfSpectrum] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention.  …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_RAD-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with raw data from the RAMSES Radiance sensor (onboard the ROV). File format identical to irradiance sensor above. …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_INCOM-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with raw data from the RAMSES Irradiance surface sensor (on the ice surface). File format identical to irradiance sensor above. File not present during PS101 but in future campaigns.  …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_IRRAD-YYYY-MM-DDTHH-MM-SS-FFF.spec ASCII file with raw data from the RAMSES Irradiance sensor (onboard the ROV). The data is stored in the .spec format, an AWI internal format for RAMSES data handling described in the following: Datalines: 2016-10-01 08:22:50 SAM_8519 1024 61.454936 86.868401 3.6 7.6 -0.60 1240 1270 1389 1588 … 1092 1125 [Date Time Sensor_ID IntegrationTime(ms) Longitude Latitude Roll Pitch Pressure(dB) followed by RAW-counts of Spectrum] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. File is only present when data are processed. …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_RAD-YYYY-MM-DDTHH-MM-SS-FFF.spec ASCII file with raw data from the RAMSES Radiance sensor (onboard the ROV). The data is stored in the .spec format, analog to above. File is only present when data are processed …\YYYMMDD_#\Actors\Rov\ramses-RAMSES_INCOM-YYYY-MM-DDTHH-MM-SS-FFF.spec ASCII file with raw data from the RAMSES Irradiance surface sensor (on the ice surface). File format identical to irradiance sensor above. The data is stored in the .spec format, analog to above. File is only present when data are processed …\YYYMMDD_#\Actors\Rov\telemetry-main-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with telemetry (attitude)  NMEA-data from the vehicle in the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.49.235 | $IIROV,118.1,7.7,3.6,-0.5*6F] [TimeStamp | $IIROV,Heading,Roll,Pitch,Depth*Checksum ] Filename is given according to stream name by Spot.On and is always consistent. …\YYYMMDD_#\Actors\Rov\viper-VIPER-YYYY-MM-DDTHH-MM-SS-FFF.sogs1 ASCII file with raw hyperspectral extinction data from the TriOs VIPER G2 sensor from the vehicle in the following format: 1 Header line: [SPOT.ON general serial format version 1] followed by datalines: [2016.10.01T08.22.57.357 | Viper_A044 1 1 5.456299 1 150 -6.5625 177 349.2714 351.43 353.5884 355.7465 357.9042 360.0617 362.2188 364.3756 366.5321 368.6883 370.8442 372.9997 375.1549 377.3097 … 3.258904] [TimeStamp | SerialNumber SpectrumType Averaging CalFactor FlashCount PathLength Temperature SpectrumLength WavelengthValues(SpectrumLength) ExtinctionValues(SpectrumLength)] Filename is given according to stream name by Spot.On and only stays consistent if operator sticks to the convention. …\YYYMMDD_#\Actors\Rov\video-Blackmagic_WDM_Capture-YYYY-MM-DDTHH-MM-SS-FFF.mpg MPG file (h.264 codec) of the Main HD Zoom Video Camera: …\YYYMMDD_#\Actors\Rov\video-Decklink_Video_Capture-YYYY-MM-DDTHH-MM-SS-FFF.mpg MPG file (h.264 codec) of the secondary SD video stream (either TigerShark camera or multi-view of all 3 Navigation Camera): …\YYYMMDD_#\Actors\Rov\video-DVI2PCI_D2P181781_VGA_-YYYY-MM-DDTHH-MM-SS-FFF.mpg MPG file (h.264 codec) of the screen grab of the SeaNet Pro sonar and USBL display. 

Data description of ROV BEAST

Abstract

Similar works

Full text

Available Versions

Electronic Publication Information Center