{ "cells": [ { "cell_type": "markdown", "id": "8e79ad85-1172-4002-9f52-97c1e85cde44", "metadata": {}, "source": [ "# Processing raw RINEX data\n", "In this example, we process some example RINEX files to demonstrate gnssvod. \n", "\n", "## Running the tutorials\n", "Once gnssvod is installed in your environment, you can download the tutorials using the command line\n", "```bash\n", "git clone --depth 1 --filter=blob:none --sparse https://github.com/vincenthumphrey/gnssvod.git\n", "cd gnssvod\n", "git sparse-checkout set docs/source/examples\n", "```\n", "\n", "Alternatively, [download the repository as a ZIP](https://github.com/vincenthumphrey/gnssvod/archive/refs/heads/main.zip), unzip, and navigate to docs/source/examples" ] }, { "cell_type": "markdown", "id": "1a548c0a-44f2-4548-8bfb-fc9348b49fa7", "metadata": {}, "source": [ "## Preprocessing\n", "The main pre-processing function is {py:func}`gnssvod.preprocess`. This function can do several things:\n", "- It will read RINEX observation files as pandas data frames\n", "- It can aggregate the raw data to a lower temporal rate if specified.\n", "- It will automatically download orbit and clock files for the corresponding days from the GSSC ESA server\n", "- From the orbit and clock files, it will calculate azimuth and elevation for each measurement\n", "- It can save each processed file as a netcdf file in the outputdir folder or return the results as a dictionary\n", "\n", "### specifying input files\n", "The function exclusively reads RINEX observation files. Such files typically end with the extension '.yyO' where yy is the last two digit of the year. The function can be used to process a single file, a group of files, or several groups of files corresponding to several receivers, as shown in the examples below. All of this is done by specifying a pattern as the first argument to the function.\n", "\n", "### specifying output destinations\n", "Results are saved to a NetCDF file when an output directory is specified and/or returned as a dictionary when \"outputresult=True\" is passed.\n", "\n", "Let's read a single file using the example data to begin with" ] }, { "cell_type": "code", "execution_count": 8, "id": "c06a8784-1265-4ef9-8973-9455df3edd2d", "metadata": {}, "outputs": [], "source": [ "import gnssvod as gv" ] }, { "cell_type": "code", "execution_count": 9, "id": "6c8e3815-c887-48d4-8fdb-ab81701c29ff", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created a temporary directory at /tmp/tmpzf3jqmyn\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O is read in 4.06 seconds.\n", "Processing 112382 individual observations\n", "Calculating Azimuth and Elevation\n", "This file does not exist: /tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz: 0.98MB [00:01, 521kB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz\n", "/tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 2.49 seconds\n", "This file does not exist: /tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz: 0.98MB [00:00, 2.00MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz\n", "/tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 1.10 seconds\n", "This file does not exist: /tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz: 4.54MB [00:00, 8.26MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz\n", "/tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 3.08 seconds\n", "This file does not exist: /tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz: 4.55MB [00:01, 4.53MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz\n", "/tmp/tmpzf3jqmyn/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 3.21 seconds\n", "SP3 interpolation is done in 8.28 seconds\n", "Removed the temporary directory at /tmp/tmpzf3jqmyn\n" ] } ], "source": [ "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O'}\n", "result = gv.preprocess(pattern, outputresult=True)" ] }, { "cell_type": "markdown", "id": "f3d9f478-a625-4dd7-8f5d-71786face67f", "metadata": {}, "source": [ "The logs should indicate how many observations were read in the file.\n", "\n", "The logs also show some orbit files were downloaded. Orbit files are necessary to calculate the azimuth and elevation of the satellites. A temporary folder is automatically created to store those orbit files and process them, then the temporary folder is deleted.\n", "\n", "If you process very recent data (less than 3 days old), it could be that the orbit and clock files are not available on the ESA server yet and there would then be an error.\n", "\n", "The result returned by the function is a dictionary that maps keys to lists of Observation objects." ] }, { "cell_type": "code", "execution_count": 10, "id": "fc4fe50a-c999-48ef-a385-2e6efd785c56", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Dav2_Twr': []}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result" ] }, { "cell_type": "markdown", "id": "f652b71f-aec9-427e-8e48-4c39479d3f19", "metadata": {}, "source": [ "Since we processed one file, there is only one Observation object in the list. Let us access this first and unique item." ] }, { "cell_type": "code", "execution_count": 11, "id": "a4cd0168-3c4e-4450-a082-35646c4a42d5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "obs = result['Dav2_Twr'][0]\n", "obs" ] }, { "cell_type": "markdown", "id": "f858f1f5-a2d4-47fe-831a-146a755864ab", "metadata": {}, "source": [ "Observation objects are custom classes introduced in the [`gnsspy`](https://github.com/GNSSpy-Project/gnsspy) package by Mustafa Serkan Işık and Volkan Özbey. A significant number of base functions in `gnssvod` are based on gnsspy.\n", "\n", "\n", "| Attribute | Description |\n", "|-----------|-------------|\n", "| `obs.filename` | Name of the source file |\n", "| `obs.epoch` | Datetime indicating the day at the start of the record |\n", "| `obs.observation` | Pandas DataFrame containing all measurements |\n", "| `obs.approx_position` | Approximate receiver position from the RINEX file `[X, Y, Z]` |\n", "| `obs.receiver_type` | Receiver type (if provided in the RINEX file) |\n", "| `obs.antenna_type` | Antenna type (if provided in the RINEX file) |\n", "| `obs.interval` | Measurement frequency in seconds |\n", "| `obs.receiver_clock` | Receiver clock information (if provided) |\n", "| `obs.version` | RINEX file version |\n", "| `obs.observation_types` | Observation types reported as columns in `obs.observation` |\n", "\n", "---\n", "\n", "Let's just look at the data.." ] }, { "cell_type": "code", "execution_count": 12, "id": "7903891f-7535-4a13-b306-71da222c78d5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
C1L1D1S1C2L2D2S2C7L7D7S7AzimuthElevation
EpochSV
2021-04-28 21:07:08G012.300005e+071.208661e+08-2545.05047.02.300006e+079.418140e+07-1983.22344.0NaNNaNNaNNaN132.37286459.902810
G032.257959e+071.186567e+08-528.92147.02.257959e+079.245973e+07-412.19742.0NaNNaNNaNNaN-16.74417679.170432
G042.373431e+071.247247e+081971.15745.02.373431e+079.718807e+071535.98342.0NaNNaNNaNNaN-169.82243250.693499
G092.620530e+071.377098e+083155.79840.02.620531e+071.073063e+082459.03340.0NaNNaNNaNNaN-150.32851019.288991
G172.423057e+071.273325e+08-403.68449.02.423058e+079.922014e+07-314.54340.0NaNNaNNaNNaN-81.69603743.789922
................................................
2021-04-28 22:07:07C16NaN2.282536e+08-1695.97236.04.383370e+072.282536e+08-1695.97236.04.383372e+07NaN-1311.19731.041.1725587.343129
C27NaN1.432542e+081694.59644.02.751047e+071.432542e+081694.59644.0NaNNaNNaNNaN-42.39574620.938009
C28NaN1.326840e+08-1191.32749.02.548056e+071.326840e+08-1191.32749.0NaNNaNNaNNaN-99.55178049.140759
S234.111082e+072.160388e+08-516.36145.0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
S364.078565e+072.143301e+08-517.12042.0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

112382 rows × 14 columns

\n", "
" ], "text/plain": [ " C1 L1 D1 S1 \\\n", "Epoch SV \n", "2021-04-28 21:07:08 G01 2.300005e+07 1.208661e+08 -2545.050 47.0 \n", " G03 2.257959e+07 1.186567e+08 -528.921 47.0 \n", " G04 2.373431e+07 1.247247e+08 1971.157 45.0 \n", " G09 2.620530e+07 1.377098e+08 3155.798 40.0 \n", " G17 2.423057e+07 1.273325e+08 -403.684 49.0 \n", "... ... ... ... ... \n", "2021-04-28 22:07:07 C16 NaN 2.282536e+08 -1695.972 36.0 \n", " C27 NaN 1.432542e+08 1694.596 44.0 \n", " C28 NaN 1.326840e+08 -1191.327 49.0 \n", " S23 4.111082e+07 2.160388e+08 -516.361 45.0 \n", " S36 4.078565e+07 2.143301e+08 -517.120 42.0 \n", "\n", " C2 L2 D2 S2 \\\n", "Epoch SV \n", "2021-04-28 21:07:08 G01 2.300006e+07 9.418140e+07 -1983.223 44.0 \n", " G03 2.257959e+07 9.245973e+07 -412.197 42.0 \n", " G04 2.373431e+07 9.718807e+07 1535.983 42.0 \n", " G09 2.620531e+07 1.073063e+08 2459.033 40.0 \n", " G17 2.423058e+07 9.922014e+07 -314.543 40.0 \n", "... ... ... ... ... \n", "2021-04-28 22:07:07 C16 4.383370e+07 2.282536e+08 -1695.972 36.0 \n", " C27 2.751047e+07 1.432542e+08 1694.596 44.0 \n", " C28 2.548056e+07 1.326840e+08 -1191.327 49.0 \n", " S23 NaN NaN NaN NaN \n", " S36 NaN NaN NaN NaN \n", "\n", " C7 L7 D7 S7 Azimuth \\\n", "Epoch SV \n", "2021-04-28 21:07:08 G01 NaN NaN NaN NaN 132.372864 \n", " G03 NaN NaN NaN NaN -16.744176 \n", " G04 NaN NaN NaN NaN -169.822432 \n", " G09 NaN NaN NaN NaN -150.328510 \n", " G17 NaN NaN NaN NaN -81.696037 \n", "... ... .. ... ... ... \n", "2021-04-28 22:07:07 C16 4.383372e+07 NaN -1311.197 31.0 41.172558 \n", " C27 NaN NaN NaN NaN -42.395746 \n", " C28 NaN NaN NaN NaN -99.551780 \n", " S23 NaN NaN NaN NaN NaN \n", " S36 NaN NaN NaN NaN NaN \n", "\n", " Elevation \n", "Epoch SV \n", "2021-04-28 21:07:08 G01 59.902810 \n", " G03 79.170432 \n", " G04 50.693499 \n", " G09 19.288991 \n", " G17 43.789922 \n", "... ... \n", "2021-04-28 22:07:07 C16 7.343129 \n", " C27 20.938009 \n", " C28 49.140759 \n", " S23 NaN \n", " S36 NaN \n", "\n", "[112382 rows x 14 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "obs.observation" ] }, { "cell_type": "markdown", "id": "76f816b1-6353-48af-9aa5-5e621720a4fe", "metadata": {}, "source": [ "The pandas data frame has a MultIndex that contains both Epoch and SV as indices. The Epoch is the local time of the measurement and the SV is a satellite identification number (also called PRN).\n", "\n", "The columns correspond to:\n", "- C# = Pseudorange from the receiver to the satellite, in meters\n", "- L# = Carrier phase, in cycles\n", "- D# = Doppler, in Hz\n", "- S# = Carrier to noise density C/N$_0$, in dB (receiver-dependent)\n", "\n", "And the numbers (S1, S2, etc. ) indicate the corresponding GNSS frequency\n", "\n", "The azimuth and elevation of the satellite with respect to the receiver are expressed in degrees. Computation speed for the azimuth and elevation can vary according to your hardware. Most of the time is spent interpolating the orbit parameters to the time stamps of each measurement. This is why it is sometimes useful to resample high frequency data (here one measurement per second) to for instance one measurement each 15 seconds.\n", "\n", "### Resampling\n", "\n", "We can pass \"interval='15s'\" to resample the data during the preprocessing. The returned data will be smaller and the calculation of the azimuths and elevations (reported as \"SP3 interpolation\") will be faster." ] }, { "cell_type": "code", "execution_count": 13, "id": "d98edc38-4750-480b-9098-d7eb27b9a3df", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created a temporary directory at /tmp/tmp2y9_xgyk\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O is read in 4.06 seconds.\n", "Processing 112382 individual observations\n", "Calculating Azimuth and Elevation\n", "This file does not exist: /tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz: 0.98MB [00:01, 608kB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz\n", "/tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 2.34 seconds\n", "This file does not exist: /tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz: 0.98MB [00:00, 2.12MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz\n", "/tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 1.01 seconds\n", "This file does not exist: /tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz: 4.54MB [00:00, 8.07MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz\n", "/tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 3.40 seconds\n", "This file does not exist: /tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz: 4.55MB [00:00, 7.95MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz\n", "/tmp/tmp2y9_xgyk/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 3.03 seconds\n", "SP3 interpolation is done in 2.08 seconds\n", "Removed the temporary directory at /tmp/tmp2y9_xgyk\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
C1C2C7D1D2D7L1L2L7S1S2S7AzimuthElevation
EpochSV
2021-04-28 21:07:00C06NaN4.307927e+074.307927e+07-1539.026429-1539.026429-1189.8415712.243250e+082.243250e+081.734648e+0838.00000038.00000031.00000036.61449510.132689
C09NaN4.080972e+074.080973e+07-1076.832857-1076.832857-832.7465712.125070e+082.125070e+081.643239e+0841.00000041.00000036.00000049.03447232.742503
C11NaN2.596665e+072.596666e+07-3555.085714-3555.085714-2748.9675711.352152e+081.352152e+081.045570e+0843.42857143.42857141.000000177.23454935.079888
C14NaN2.382462e+072.382462e+0732.76042932.76042925.2824291.240611e+081.240611e+089.593200e+0745.00000045.00000042.285714-96.37335376.785189
C16NaN4.268828e+074.268830e+07-1595.855857-1595.855857-1234.1170002.222891e+082.222891e+081.718881e+0838.00000038.00000033.00000038.26660315.249211
................................................
2021-04-28 22:07:00R182.326705e+072.326705e+07NaN-3848.505125-2993.250750NaN1.242018e+089.660093e+07NaN46.37500042.000000NaN-167.16071845.315939
R192.237872e+072.237872e+07NaN-52.467125-40.688500NaN1.197111e+089.310873e+07NaN37.37500038.875000NaN-74.71156163.385367
R202.596147e+072.596148e+07NaN3565.5648752773.281125NaN1.388277e+081.079770e+08NaN37.87500034.750000NaN-26.14666013.545577
S234.111048e+07NaNNaN-515.803000NaNNaN2.160370e+08NaNNaN44.750000NaNNaNNaNNaN
S364.078531e+07NaNNaN-516.305375NaNNaN2.143283e+08NaNNaN41.750000NaNNaNNaNNaN
\n", "

7550 rows × 14 columns

\n", "
" ], "text/plain": [ " C1 C2 C7 \\\n", "Epoch SV \n", "2021-04-28 21:07:00 C06 NaN 4.307927e+07 4.307927e+07 \n", " C09 NaN 4.080972e+07 4.080973e+07 \n", " C11 NaN 2.596665e+07 2.596666e+07 \n", " C14 NaN 2.382462e+07 2.382462e+07 \n", " C16 NaN 4.268828e+07 4.268830e+07 \n", "... ... ... ... \n", "2021-04-28 22:07:00 R18 2.326705e+07 2.326705e+07 NaN \n", " R19 2.237872e+07 2.237872e+07 NaN \n", " R20 2.596147e+07 2.596148e+07 NaN \n", " S23 4.111048e+07 NaN NaN \n", " S36 4.078531e+07 NaN NaN \n", "\n", " D1 D2 D7 L1 \\\n", "Epoch SV \n", "2021-04-28 21:07:00 C06 -1539.026429 -1539.026429 -1189.841571 2.243250e+08 \n", " C09 -1076.832857 -1076.832857 -832.746571 2.125070e+08 \n", " C11 -3555.085714 -3555.085714 -2748.967571 1.352152e+08 \n", " C14 32.760429 32.760429 25.282429 1.240611e+08 \n", " C16 -1595.855857 -1595.855857 -1234.117000 2.222891e+08 \n", "... ... ... ... ... \n", "2021-04-28 22:07:00 R18 -3848.505125 -2993.250750 NaN 1.242018e+08 \n", " R19 -52.467125 -40.688500 NaN 1.197111e+08 \n", " R20 3565.564875 2773.281125 NaN 1.388277e+08 \n", " S23 -515.803000 NaN NaN 2.160370e+08 \n", " S36 -516.305375 NaN NaN 2.143283e+08 \n", "\n", " L2 L7 S1 S2 \\\n", "Epoch SV \n", "2021-04-28 21:07:00 C06 2.243250e+08 1.734648e+08 38.000000 38.000000 \n", " C09 2.125070e+08 1.643239e+08 41.000000 41.000000 \n", " C11 1.352152e+08 1.045570e+08 43.428571 43.428571 \n", " C14 1.240611e+08 9.593200e+07 45.000000 45.000000 \n", " C16 2.222891e+08 1.718881e+08 38.000000 38.000000 \n", "... ... ... ... ... \n", "2021-04-28 22:07:00 R18 9.660093e+07 NaN 46.375000 42.000000 \n", " R19 9.310873e+07 NaN 37.375000 38.875000 \n", " R20 1.079770e+08 NaN 37.875000 34.750000 \n", " S23 NaN NaN 44.750000 NaN \n", " S36 NaN NaN 41.750000 NaN \n", "\n", " S7 Azimuth Elevation \n", "Epoch SV \n", "2021-04-28 21:07:00 C06 31.000000 36.614495 10.132689 \n", " C09 36.000000 49.034472 32.742503 \n", " C11 41.000000 177.234549 35.079888 \n", " C14 42.285714 -96.373353 76.785189 \n", " C16 33.000000 38.266603 15.249211 \n", "... ... ... ... \n", "2021-04-28 22:07:00 R18 NaN -167.160718 45.315939 \n", " R19 NaN -74.711561 63.385367 \n", " R20 NaN -26.146660 13.545577 \n", " S23 NaN NaN NaN \n", " S36 NaN NaN NaN \n", "\n", "[7550 rows x 14 columns]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O'}\n", "result = gv.preprocess(pattern,interval='15s',outputresult=True)\n", "# and show data frame\n", "result['Dav2_Twr'][0].observation" ] }, { "cell_type": "markdown", "id": "3bb50205-b28e-4d3f-a037-f9e9d8159c67", "metadata": {}, "source": [ "There are now less rows in the data frame since we resampled data to one value every 15 seconds." ] }, { "cell_type": "markdown", "id": "d478c7b1-74ce-4da7-89b7-d8690011c37c", "metadata": {}, "source": [ "## Batch processing\n", "We now demonstrate how to use the preprocessing function to process not just one but many files and save the outputs as NetCDF files (instead of returning the results as objects). If we were processing several hundreds of files, your computer may not have sufficient memory to hold all of the outputs, so it makes sense to save processed data as a NetCDF file.\n", "\n", "### Specifying several groups of files\n", "Instead of specifying just one file, like `data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O`, we specify a UNIX-style pattern like `data_RINEX2.11/Dav2_Twr/rinex/*.*O`. All files matching that pattern (relying on function :func:`glob.glob`) will be processed. We can process several groups files by specifying different matching patterns (see below).\n", "\n", "### Specifying where to save data\n", "Same as for specifying the inputs, we use a dictionary to indicate where to save data (see the example call below). The destination folder will be created if it does not exist.\n", "\n", "### Specifying a list of variables to save\n", "For calculating GNSS-VOD, we only need the \"S\" variables. We can reduce the size of the saved NetCDF files by discarding the other variables, this is done with the 'keepvars' argument, which will only keep the variables present in the passed list. This argument supports UNIX-style pattern matching (e.g. 'S*' will match all variables starting with 'S')\n", "\n", "### Compression\n", "Unless `encoding=None` is passed as argument, `gv.preprocess()` will compress all S* variables, as well as Azimuth and Elevation when saving to NetCDF. These variables are encoded as Int16 with a scale factor of 0.1. The decoding is automatically applied when reading the data with xarray." ] }, { "cell_type": "code", "execution_count": 14, "id": "0e7e75d9-c77a-4b5c-9752-2317ac852696", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created a temporary directory at /tmp/tmpfk6r0oua\n", "Could not find any files matching the pattern data_RINEX2.11/Dav2_Twr/nc/*.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O is read in 3.90 seconds.\n", "Processing 112382 individual observations\n", "Calculating Azimuth and Elevation\n", "This file does not exist: /tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz: 0.98MB [00:00, 2.11MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 1.10 seconds\n", "This file does not exist: /tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3\n", "Downloading: GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz: 0.98MB [00:00, 2.15MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 1.01 seconds\n", "This file does not exist: /tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz: 4.54MB [00:00, 7.65MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 3.22 seconds\n", "This file does not exist: /tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK\n", "Downloading: GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz" ] }, { "name": "stderr", "output_type": "stream", "text": [ "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz: 4.55MB [00:00, 8.36MB/s] \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | Download completed for GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 2.86 seconds\n", "SP3 interpolation is done in 2.22 seconds\n", "Saved 7550 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104282106.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282206.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282206.21O is read in 3.90 seconds.\n", "Processing 112113 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7533 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104282206.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282306.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282306.21O is read in 3.94 seconds.\n", "Processing 115429 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7756 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104282306.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290006.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290006.21O is read in 3.95 seconds.\n", "Processing 109702 individual observations\n", "Calculating Azimuth and Elevation\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.50 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.62 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 3.09 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 3.23 seconds\n", "SP3 interpolation is done in 2.26 seconds\n", "Saved 7370 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104290006.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290106.21O is read in 4.27 seconds.\n", "Processing 117216 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7871 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104290106.nc\n", "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290206.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290206.21O is read in 3.81 seconds.\n", "Processing 108484 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7331 individual observations in data_RINEX2.11/Dav2_Twr/nc/Reach_Dav2_Twr-raw_202104290206.nc\n", "Could not find any files matching the pattern data_RINEX2.11/Dav1_Grnd/nc/*.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282106.21O is read in 3.15 seconds.\n", "Processing 111162 individual observations\n", "Calculating Azimuth and Elevation\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.27 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.27 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 2.13 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.93 seconds\n", "SP3 interpolation is done in 1.51 seconds\n", "Saved 7481 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104282106.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282206.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282206.21O is read in 3.74 seconds.\n", "Processing 107018 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7231 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104282206.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282306.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282306.21O is read in 3.98 seconds.\n", "Processing 113394 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7670 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104282306.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290006.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290006.21O is read in 3.91 seconds.\n", "Processing 110793 individual observations\n", "Calculating Azimuth and Elevation\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.48 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.48 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 2.53 seconds\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n", "/tmp/tmpfk6r0oua/GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 2.99 seconds\n", "SP3 interpolation is done in 2.20 seconds\n", "Saved 7498 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104290006.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290106.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290106.21O is read in 3.80 seconds.\n", "Processing 109171 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 7414 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104290106.nc\n", "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290206.21O exists | Reading...\n", "Observation file data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290206.21O is read in 3.63 seconds.\n", "Processing 102591 individual observations\n", "Calculating Azimuth and Elevation\n", "Saved 6961 individual observations in data_RINEX2.11/Dav1_Grnd/nc/Reach_Dav1_Grnd-raw_202104290206.nc\n", "Removed the temporary directory at /tmp/tmpfk6r0oua\n" ] } ], "source": [ "# use gnssvod to batch process the observation RINEX files \n", "# (files with extension .yyO for each station)\n", "# pattern = {'choice_of_name_for_station1':'pattern to match (UNIX-style)',\n", "# 'choice_of_name_for_station2':'pattern to match (UNIX-style)',\n", "# ...}\n", "#\n", "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/*.*O',\n", " 'Dav1_Grnd':'data_RINEX2.11/Dav1_Grnd/rinex/*.*O'}\n", "outputdir = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/nc/',\n", " 'Dav1_Grnd':'data_RINEX2.11/Dav1_Grnd/nc/'}\n", "# what variables should be kept\n", "keepvars = ['S?','S??']\n", "\n", "gv.preprocess(pattern,interval='15s',keepvars=keepvars,outputdir=outputdir)" ] }, { "cell_type": "markdown", "id": "ea7754bc-2319-4102-ae32-4c195724a89d", "metadata": {}, "source": [ "### Skipping existing files by default\n", "The preprocess function will scan the destination folder for existing NetCDF files. If some files are found that have already been processed, these files will be skipped unless overwrite=True has been passed.\n", "\n", "Here because the destination folder was empty, a user warning appears in the log above but can be ignored (\"Could not find any files matching the pattern data_RINEX2.11/Dav2_Twr/nc/*.nc\")" ] } ], "metadata": { "kernelspec": { "display_name": "gnssvod-j8VmKh7q-py3.9", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10" } }, "nbformat": 4, "nbformat_minor": 5 }