Ctrl+K

Hint

You can run this notebook in a live session with Binder Helmholtz.

  • it is important to acknowledge that the utilization of the Helmholtz is restricted to individuals who possess the necessary credentials as Helmholtz users. It is highly recommended to utilize the Python scipy option when constructing the environment in order to mitigate the occurrence of a 404 Bad request error.

TDS2STACIntegrator#

In this section, we present a set of illustrative examples that demonstrate the process of working with TDS2STAC.

Harvesting and creating without any additional extension or asset:#

In order to achieve this objective, it is necessary to complete two distinct arguments while the remaining arguments will be automatically populated.

In the present example, we intend to extract data from the dataset located at IMK-IFU Thredds and thereafter save the generated STAC-metadata in the “/stac” directory. We have incorporated a limited_number parameter to choose extract a single data point from the collection. It should be mentioned the primary objective of limited_number is to facilitate unit-testing and development activities.

[ ]:

from tds2stac import TDS2STACIntegrator

TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/climate/raster/global/chelsa/v1.2/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    logger_properties={"logger_handler":"StreamHandler"},
)

Output:

stac/
└── stac
    ├── catalog.json
    └── catalog_climate_raster_global_chelsa_v1_2_catalog_html_collection
        ├── chelsa_v1_2_chelsa_prec_10_v1_2_land
        │   └── chelsa_v1_2_chelsa_prec_10_v1_2_land.json
        └── collection.json

4 directories, 3 files

Verify the presence of a catalog in the directory#

This functionality is intended for scenarios when a STAC-Collections or STAC-Item is available and requires updating. As demonstrated in the preceding example, a STAC-Catalog was generated and assigned to the /stac directory. We seek to modify the previously established catalog and collection in the following manner. Prior to making any updates, it is advisable to review the STAC-Catalog and STAC-Collection.

The JSON file provided exhibits the structure of STAC-Catalog and STAC-Collections. As an illustrative instance, we aim to initially examine the STAC-Metadata and thereafter employ the stac_existence and stac_existence_collection

STAC-Catalog:

{
  "type": "Catalog",
  "id": "TDS2STAC",
  "stac_version": "1.0.0",
  "description": "[This is a STAC catalog created by tds2stac](https://thredds.imk-ifu.kit.edu/thredds/catalog/climate/raster/global/chelsa/v1.2/catalog.html)",
  "links": [
    {
      "rel": "root",
      "href": "./catalog.json",
      "type": "application/json",
      "title": "TDS2STAC"
    },
    {
      "rel": "child",
      "href": "./catalog_climate_raster_global_chelsa_v1_2_catalog_html_collection/collection.json",
      "type": "application/json",
      "title": "Catalog Climate Raster Global Chelsa V1 2 Catalog Html Collection"
    }
  ],
  "title": "TDS2STAC"
}

STAC-Collection:

{
  "type": "Collection",
  "id": "catalog_climate_raster_global_chelsa_v1_2_catalog_html_collection",
  "stac_version": "1.0.0",
  "description": "[Link to TDS](https://thredds.imk-ifu.kit.edu/thredds/catalog/climate/raster/global/chelsa/v1.2/catalog.html)",
  "links": [
    {
      "rel": "root",
      "href": "../catalog.json",
      "type": "application/json",
      "title": "TDS2STAC"
    },
    {
      "rel": "item",
      "href": "./chelsa_v1_2_chelsa_prec_10_v1_2_land/chelsa_v1_2_chelsa_prec_10_v1_2_land.json",
      "type": "application/json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json",
      "type": "application/json",
      "title": "TDS2STAC"
    }
  ],
  "title": "Catalog Climate Raster Global Chelsa V1 2 Catalog Html Collection",
  "extent": {
    "spatial": {
      "bbox": [
        [
          -179.99597,
          -89.99597,
          179.99569,
          83.99569
        ]
      ]
    },
    "temporal": {
      "interval": [
        [
          "1996-10-01T01:10:36Z",
          "1996-10-01T01:10:36Z"
        ]
      ]
    }
  },
  "license": "proprietary"
}

In the current setting, the utilization of stac_existence and stac_existence_collection is employed to facilitate the process of updating the catalog and collections. In this part, the value of limited_number is modified to 2 in order to see the disparities within the STAC-Collection. Additionally, another collection is harvested to assess the distinctions present within the STAC-Catalog.

[ ]:

from tds2stac import TDS2STACIntegrator

TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/climate/raster/global/chelsa/v1.2/catalog.html",
    stac_dir="stac/",
    limited_number=2,
    stac_existence=True,
    stac_existence_collection=True,
    logger_properties={"logger_handler":"StreamHandler"},
)

TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/regional/eobs/v23.1e/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    stac_existence=True,
    stac_existence_collection=True,
    logger_properties={"logger_handler":"StreamHandler"},
)

Output:

stac/
└── stac
    ├── catalog.json
    ├── catalog_climate_raster_global_chelsa_v1_2_catalog_html_collection
    │   ├── chelsa_v1_2_chelsa_prec_10_v1_2_land
    │   │   └── chelsa_v1_2_chelsa_prec_10_v1_2_land.json
    │   ├── chelsa_v1_2_chelsa_prec_10_v1_2_land_10min
    │   │   └── chelsa_v1_2_chelsa_prec_10_v1_2_land_10min.json
    │   └── collection.json
    └── catalog_regclim_raster_regional_eobs_v23_1e_catalog_html_collection
        ├── collection.json
        └── eobs_daily_0_1_v23_1e_pp_ens_mean_0_1deg_reg_v23_1e
            └── eobs_daily_0_1_v23_1e_pp_ens_mean_0_1deg_reg_v23_1e.json

7 directories, 6 files

Change the ID, name, and description of STAC-Collections#

To modify the information of STAC-Collections, the initial step is executing the NestedCollectionInspector to identify the automated Collection IDs. Subsequently, the auto-generated IDs serve as the basis for making the desired changes to all relevant collections. However, prior to utilizing the NestedCollectionInspector class, it is necessary to determine the depth of the dataset catalog in order to organize the collections into a given layer. In order to accomplish this objective, it is necessary to utilize the Recognizer to get the deepest layer number and subsequently employ the NestedCollectionInspector. Once the dataset depth has been determined and a specific layer has been selected, and in the case of having five distinct STAC-Collections in our STAC-Catalog harvesting process, if there is a need to modify the ID, name, and description of two of these collections, the following steps should be taken.

Firstly, we must index two auto-generated IDs as the first element of tuples within a list of tuples (collection_tuples). Subsequently, the desired ID should be added as the second element of each tuple, while the name and description should be added as the third and fourth elements, respectively.

To be exact, we can have a look at the following example.

[ ]:

from tds2stac import Recognizer, NestedCollectionInspector, TDS2STACIntegrator

print("Finding depth of the dataset:")
Recognizer(
    main_catalog_url="https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.html",
    nested_check=True,
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)

print("Zero depth of dataset:")
NestedCollectionInspector(
    main_catalog_url="https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.html",
    nested_number=0,  # by default the depth layer is zero. In these situation there is no need to write the zero down.
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)
print("First depth of dataset:")
NestedCollectionInspector(
    main_catalog_url="https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.html",
    nested_number=1,
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)
print("Second depth of dataset:")
NestedCollectionInspector(
    main_catalog_url="https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.html",
    nested_number=2,
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)
print("Third depth of dataset:")
NestedCollectionInspector(
    main_catalog_url="https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.html",
    nested_number=3,
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)

Output:

Finding depth of the dataset:

INFO 2023-11-15 12:10:56,495 logger.py @function init line 402 - (‘First Scenario’, 4)

Zero depth of dataset:

INFO 2023-11-15 12:12:55,736 logger.py @function init line 402 - [(’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/catalog.xml’, ‘catalog_icon_accmip’, ‘Catalog Icon Accmip’, […])]

First depth of dataset:

INFO 2023-11-15 12:15:56,435 logger.py @function init line 402 - [(’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/catalog.xml’, ‘catalog_icon_accmip_c2h6’, ‘Catalog Icon Accmip C2H6’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH3COCH3/catalog.xml’, ‘catalog_icon_accmip_ch3coch3’, ‘Catalog Icon Accmip Ch3Coch3’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/catalog.xml’, ‘catalog_icon_accmip_ch4’, ‘Catalog Icon Accmip Ch4’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/catalog.xml’, ‘catalog_icon_accmip_co’, ‘Catalog Icon Accmip Co’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/catalog.xml’, ‘catalog_icon_accmip_nox’, ‘Catalog Icon Accmip Nox’, […])]

Second depth of dataset:

INFO 2023-11-15 12:18:51,031 logger.py @function init line 402 - [(’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/ANT/catalog.xml’, ‘catalog_icon_accmip_c2h6_ant’, ‘Catalog Icon Accmip C2H6 Ant’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/BBE/catalog.xml’, ‘catalog_icon_accmip_c2h6_bbe’, ‘Catalog Icon Accmip C2H6 Bbe’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH3COCH3/BBE/catalog.xml’, ‘catalog_icon_accmip_ch3coch3_bbe’, ‘Catalog Icon Accmip Ch3Coch3 Bbe’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/ANT/catalog.xml’, ‘catalog_icon_accmip_ch4_ant’, ‘Catalog Icon Accmip Ch4 Ant’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/BBE/catalog.xml’, ‘catalog_icon_accmip_ch4_bbe’, ‘Catalog Icon Accmip Ch4 Bbe’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/ANT/catalog.xml’, ‘catalog_icon_accmip_co_ant’, ‘Catalog Icon Accmip Co Ant’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/BBE/catalog.xml’, ‘catalog_icon_accmip_co_bbe’, ‘Catalog Icon Accmip Co Bbe’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/catalog.xml’, ‘catalog_icon_accmip_nox_ant’, ‘Catalog Icon Accmip Nox Ant’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/BBE/catalog.xml’, ‘catalog_icon_accmip_nox_bbe’, ‘Catalog Icon Accmip Nox Bbe’, […])]

Third depth of dataset:

INFO 2023-11-15 12:21:43,703 logger.py @function init line 402 - [(’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/ANT/RCP26/catalog.xml’, ‘catalog_icon_accmip_c2h6_ant_rcp26’, ‘Catalog Icon Accmip C2H6 Ant Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/ANT/RCP26_ships/catalog.xml’, ‘catalog_icon_accmip_c2h6_ant_rcp26_ships’, ‘Catalog Icon Accmip C2H6 Ant Rcp26 Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/ANT/hist/catalog.xml’, ‘catalog_icon_accmip_c2h6_ant_hist’, ‘Catalog Icon Accmip C2H6 Ant Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/ANT/hist_ships/catalog.xml’, ‘catalog_icon_accmip_c2h6_ant_hist_ships’, ‘Catalog Icon Accmip C2H6 Ant Hist Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/BBE/RCP26/catalog.xml’, ‘catalog_icon_accmip_c2h6_bbe_rcp26’, ‘Catalog Icon Accmip C2H6 Bbe Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/C2H6/BBE/hist/catalog.xml’, ‘catalog_icon_accmip_c2h6_bbe_hist’, ‘Catalog Icon Accmip C2H6 Bbe Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH3COCH3/BBE/RCP26/catalog.xml’, ‘catalog_icon_accmip_ch3coch3_bbe_rcp26’, ‘Catalog Icon Accmip Ch3Coch3 Bbe Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH3COCH3/BBE/hist/catalog.xml’, ‘catalog_icon_accmip_ch3coch3_bbe_hist’, ‘Catalog Icon Accmip Ch3Coch3 Bbe Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/ANT/RCP26/catalog.xml’, ‘catalog_icon_accmip_ch4_ant_rcp26’, ‘Catalog Icon Accmip Ch4 Ant Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/ANT/RCP26_ships/catalog.xml’, ‘catalog_icon_accmip_ch4_ant_rcp26_ships’, ‘Catalog Icon Accmip Ch4 Ant Rcp26 Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/ANT/hist/catalog.xml’, ‘catalog_icon_accmip_ch4_ant_hist’, ‘Catalog Icon Accmip Ch4 Ant Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/ANT/hist_ships/catalog.xml’, ‘catalog_icon_accmip_ch4_ant_hist_ships’, ‘Catalog Icon Accmip Ch4 Ant Hist Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/BBE/RCP26/catalog.xml’, ‘catalog_icon_accmip_ch4_bbe_rcp26’, ‘Catalog Icon Accmip Ch4 Bbe Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CH4/BBE/hist/catalog.xml’, ‘catalog_icon_accmip_ch4_bbe_hist’, ‘Catalog Icon Accmip Ch4 Bbe Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/ANT/RCP26/catalog.xml’, ‘catalog_icon_accmip_co_ant_rcp26’, ‘Catalog Icon Accmip Co Ant Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/ANT/RCP26_ships/catalog.xml’, ‘catalog_icon_accmip_co_ant_rcp26_ships’, ‘Catalog Icon Accmip Co Ant Rcp26 Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/ANT/hist/catalog.xml’, ‘catalog_icon_accmip_co_ant_hist’, ‘Catalog Icon Accmip Co Ant Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/ANT/hist_ships/catalog.xml’, ‘catalog_icon_accmip_co_ant_hist_ships’, ‘Catalog Icon Accmip Co Ant Hist Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/BBE/RCP26/catalog.xml’, ‘catalog_icon_accmip_co_bbe_rcp26’, ‘Catalog Icon Accmip Co Bbe Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/CO/BBE/hist/catalog.xml’, ‘catalog_icon_accmip_co_bbe_hist’, ‘Catalog Icon Accmip Co Bbe Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/RCP26/catalog.xml’, ‘catalog_icon_accmip_nox_ant_rcp26’, ‘Catalog Icon Accmip Nox Ant Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/RCP26_aircraft/catalog.xml’, ‘catalog_icon_accmip_nox_ant_rcp26_aircraft’, ‘Catalog Icon Accmip Nox Ant Rcp26 Aircraft’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/RCP26_ships/catalog.xml’, ‘catalog_icon_accmip_nox_ant_rcp26_ships’, ‘Catalog Icon Accmip Nox Ant Rcp26 Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/hist/catalog.xml’, ‘catalog_icon_accmip_nox_ant_hist’, ‘Catalog Icon Accmip Nox Ant Hist’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/hist_aircraft/catalog.xml’, ‘catalog_icon_accmip_nox_ant_hist_aircraft’, ‘Catalog Icon Accmip Nox Ant Hist Aircraft’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/ANT/hist_ships/catalog.xml’, ‘catalog_icon_accmip_nox_ant_hist_ships’, ‘Catalog Icon Accmip Nox Ant Hist Ships’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/BBE/RCP26/catalog.xml’, ‘catalog_icon_accmip_nox_bbe_rcp26’, ‘Catalog Icon Accmip Nox Bbe Rcp26’, […]), (’https://thredds.atmohub.kit.edu/thredds/catalog/icon/ACCMIP/NOx/BBE/hist/catalog.xml’, ‘catalog_icon_accmip_nox_bbe_hist’, ‘Catalog Icon Accmip Nox Bbe Hist’, […])]

Now, we possess knowledge on the default ID and name of Collections, which is determined by the depth of the dataset. Consequently, we have the ability to modify either a single or all of these default values. This can be achieved by implementing the following changes:

[ ]:

print("Using `collection_tuples` to change the STAC-Collection details in zero dataset depth")
TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/regional/eobs/v23.1e/catalog.html",
    stac_dir="stac/",
    depth_number=0,
    limited_number=1,
    collection_tuples = [("catalog_icon_accmip",
                          "user-defined_catalog_icon_accmip",
                          "user-defined Catalog Icon Accmip",
                          "This is a description created by user")],
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)
print("Using `collection_tuples` to change the STAC-Collection details in second dataset depth")
TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/regional/eobs/v23.1e/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    depth_number=2,
    collection_tuples = [("catalog_icon_accmip_c2h6_ant",
                          "user-defined_catalog_icon_accmip_c2h6_ant",
                          "user-defined Catalog Icon Accmip C2H6 Ant",
                          "This is a description created by user"),
                        ("catalog_icon_accmip_co_bbe",
                          "user-defined_catalog_icon_accmip_co_bbe",
                          "user-defined Catalog Icon Accmip Co Bbe",
                          "This is a description created by user")],
    logger_properties={
        "logger_handler": "StreamHandler",
    },
)

Date Time Filter#

In this section, the objective is to apply a temporal filter (datetime_filter) to the dataset within a specified catalog. The outcome will be the creation of a new STAC-Catalog and STAC-Collections, or an update of existing ones. The purpose of this argument is to provide a strategy that can be employed in order to prevent the duplication of efforts in catalog harvesting, hence expediting the process of generating STAC-Metadata. By setting the variables stac_existence and stac_existence_collection to True, it becomes possible to verify the presence of the STAC-Catalog and STAC-Collections in the local storage and TDS catalog modified_datetime simultaneously.

Caution should be exercised while utilizing the limited_number parameter in order to prevent any conflicts in obtaining datetime-based data.

To illustrate this point, we have provided an example below:

[ ]:

from tds2stac import TDS2STACIntegrator
TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/global/era5_land/daily/catalog.html",
    stac_dir="stac/",
    datetime_filter=["2023-04-12T00:00:00.000Z","2023-04-12T23:59:59.000Z"],
    logger_properties = {"logger_handler": "StreamHandler"}
)

Aggregated Dataset’s URL#

Due to the malfunction of the HTTPServer web-service in Thredds for aggregated datasets, it is necessary to substitute the HTTPServer link with the URL of the dataset containing non-aggregated data (aggregated_dataset_url). This modification aims to address the issue of the inability to download the aggregated data.

[ ]:

TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/catalogues/climate_catalog_ext.html?dataset=eobs_daily_0.1_aggregated_v27",
    stac_dir="stac/",
    limited_number=1,
    aggregated_dataset_url="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/regional/eobs/v27.0e/catalog.html",
    logger_properties = {"logger_handler": "StreamHandler"}
)

Spatial information#

Certain datasets exhibit a deficiency in spatial coordination, which can be rectified by incorporating Latitude and Longitude information into the STAC-Metadata. (spatial_information)

[ ]:

from tds2stac import TDS2STACIntegrator
#Raster dataset, POLYGON geometry
TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/sawam_data/karun/frcst/regav/hydrol/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    spatial_information=[0,0,0,0],
    logger_properties = {"logger_handler": "StreamHandler"}
)
# Station-based dataset, POINT geometry
TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/sawam_data/karun/frcst/point/hydrol/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    spatial_information=[0,0],
    logger_properties = {"logger_handler": "StreamHandler"}
)

Temporal information with dataset name#

Certain datasets may lack temporal information, and to address this limitation, this approach aims to infer temporal information by using the dataset’s name. To obtain further information regarding the naming conventions for each variable, it is advisable to refer to the documentation on strftime() and strptime() Format Codes.(temporal_format_by_dataname)

https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes

[ ]:

from tds2stac import TDS2STACIntegrator

TDS2STACIntegrator(
    TDS_catalog="https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/atmos_obs/lidar/Data/20220714/catalog.html?dataset=lidar/Data/20220714/e2271407.590015",
    stac_dir="stac/",
    limited_number=2,
    spatial_information=[0,0,0,0],
    temporal_format_by_dataname="e%y%m%d%H.%M%S%f",
    logger_properties = {"logger_handler": "StreamHandler"}
)

Switch the geometry to LineString#

Certain datasets contain LineString geometry, which, similar to Polygon geometry, have attributes such as minimum latitude, maximum latitude, minimum longitude, and maximum longitude. However, these attributes may not be readily identifiable in the dataset’s webservice, making it difficult to distinguish them from Polygon geometry. Therefore, this argument is intended to modify the representation of Polygons in such cases.(item_geometry_linestring)

[ ]:

TDS2STACIntegrator(
    TDS_catalog="https://thredds.atmohub.kit.edu/thredds/catalog/caribic/IAGOS-CARIBIC_MS_files_collection_20231115/CARIBIC-1/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    item_geometry_linestring=True
)

Extension properties#

The extension_properties argument can be employed to specify extended properties for STAC-Collections, referred to as collection_extensions, and for STAC-Items, referred to as item_extensions. The structural framework employed for both entities is same. In the present iteration, the utilization of two predefined keywords, namely common_metadata and item_datacube_extension, enables the incorporation of common_metadata and datacube extension components into our items. To create custom extensions, it is possible to specify new ones as detailed in the provided documentation. Presented below is an illustrative example that warrants examination.

In the shown example, two distinct types of extensions were utilized. The custom extension, which was both user-defined and in existence. The common_metadata and item_datacube_extension have been preconfigured, while the user is responsible for specifying two more tuples. The scientific_extension is a STAC extension that builds upon the extension function of pySTAC. Similarly, the contacts_extension is a custom function created by us, which is derived from the definition of this extension found in its repository. The primary key of the JSON file corresponds to the first element in these tuples, which is determined by the definition of the key in the tag_file.json. The second parameter refers to the identifier of the function or class, while the third parameter pertains to the location of the script.

[ ]:

TDS2STACIntegrator(
    "https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/global/era5/sfc/single/daily/catalog.html",
    stac_dir="/Users/hadizadeh-m/stac/",
    limited_number=2,
    extension_properties={
        "item_extensions": [
            "common_metadata",
            "item_datacube_extension",
            (
                "scientific_extension",
                "item",
                "./custom_based_on_defined_extension_pystac.py",
            ),
            (
                "contact_extension",
                "item",
                "./custom_based_on_none_defined_extension_pystac.py",
            ),
        ],
    },

Webservice properties#

This dictionary serves the purpose of identifying the location of the self-created tag_file.json. If the tag file is not explicitly defined in this context, the default tag file stored in the main directory of TDS2STAC is utilized.

To obtain instructions on creating a tag_file.json, please refer to this documentation. In this particular instance, our objective is to designate our tag configuration file as tag_example.json. Subsequently, we aim to ascertain the value of web_service_config_file key by referring to the subsequent example.

[ ]:

from tds2stac import TDS2STACIntegrator

TDS2STACIntegrator(
    "https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/global/era5/sfc/single/daily/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    depth_number=1,
    webservice_properties={
        "web_service_config_file": "./tag_example.json",
    },
    logger_properties={"logger_handler":"StreamHandler"},
)

Asset properties#

By utilizing asset_properties argument, we are able to effectively administer our assets within each STAC-Item and STAC-Collection. To obtain further information regarding asset properties, go to the aforementioned webpage.

In the subsequent illustration, we employ all conceivable premises of this contention.

[ ]:

from tds2stac import TDS2STACIntegrator

TDS2STACIntegrator(
    "https://thredds.imk-ifu.kit.edu/thredds/catalog/regclim/raster/global/era5/sfc/single/daily/catalog.html",
    stac_dir="stac/",
    limited_number=1,
    asset_properties={
        "item_thumbnail": True,
        "item_getminmax_thumbnail": True,
        "explore_data": True,
        "verify_explore_data": True,
        "jupyter_notebook": True,
        "collection_thumbnail": "link",
        "collection_overview": "wms",
        "collection_link": "https://images.fineartamerica.com/images-medium-large-5/global-precipitation-william-putmannasa-goddard-space-flight-center.jpg",
        "assets_list_allowed": ["wms", "wcs", "wfs"],
        "assets_list_avoided": ["http"],
    },
   logger_properties={"logger_handler":"StreamHandler"},
)
On this page
Edit
Show Source