Skip to content

CDSE to EO guide #1967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Mar 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
f1ec40f
rm sentinel from mosaics text
samumantha Jan 22, 2024
5ab085d
general CDSE section added
samumantha Jan 22, 2024
c312091
fix link
samumantha Jan 22, 2024
e4a80f2
add landsat mosaics back to data page
samumantha Jan 29, 2024
3d89785
add draft download suggestion for CDSE to eoguide
samumantha Jan 29, 2024
055e438
fix internal docs link
samumantha Jan 29, 2024
579ead1
small fixes to text
samumantha Jan 29, 2024
85638eb
Remove SWIFT, reorganize all connection set up, add CDSE.
ktiits Jan 29, 2024
1436970
link fix?
samumantha Jan 29, 2024
dfc6dda
fix typos
samumantha Jan 29, 2024
7856701
minor text changes
ktiits Jan 30, 2024
5727378
Small fixes to CDSE instructions
samumantha Jan 30, 2024
d66a9d1
add geoconda mention to CDSE download section
samumantha Jan 31, 2024
b7b84dc
grammar fixes from code review
samumantha Jan 31, 2024
7ba59a5
eo guide link fixes
samumantha Feb 2, 2024
9dc3f5a
simplify CDSE download text
samumantha Feb 2, 2024
cd53d27
add note on more than data download
samumantha Feb 2, 2024
62b7ec5
GDAL cloud: add CDSE links for S3
ktiits Feb 5, 2024
e29a182
minor working
ktiits Feb 5, 2024
4dd8609
AWS_PROFILE to Linux style
ktiits Feb 5, 2024
7cea5ff
update gdal installation differences section
ktiits Feb 12, 2024
262bf02
update gdal versions
ktiits Feb 12, 2024
f3925d2
add Arttu, more STAC links
ktiits Feb 12, 2024
7dd3d37
styling fix
ktiits Feb 12, 2024
c8e54fe
Add Allas files being public
ktiits Feb 12, 2024
015a883
Merge branch 'master' into swi_eoguide
ktiits Feb 22, 2024
b98de5a
CDSE part bigger changes, less about openEO and SentinelHub
ktiits Feb 22, 2024
20444e7
style fixes
ktiits Feb 22, 2024
7a6df1f
Paituli STAC status update
ktiits Feb 22, 2024
52529f8
order change of CDSE limitations
ktiits Feb 22, 2024
a1fe6e8
typo
ktiits Feb 22, 2024
efeb95c
CDSE processing update
ktiits Feb 22, 2024
371ebf8
remove sentinelhub from alternative processing services, included in …
ktiits Feb 22, 2024
cf36894
Update eo_guide.md
ktiits Feb 23, 2024
74e5f5d
Update eo_guide.md
ktiits Feb 23, 2024
922ace9
Update eo_guide.md
ktiits Feb 23, 2024
eaa708e
Update docs/support/tutorials/gis/eo_guide.md
ktiits Feb 26, 2024
9537b69
Update docs/support/tutorials/gis/eo_guide.md
ktiits Feb 26, 2024
7a72e15
Merge branch 'master' into swi_eoguide
ktiits Mar 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/apps/gdal.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ tags:

GDAL is available with following versions:

* 3.5.0 - [geoconda-3.10.6](geoconda.md) in Puhti
* 3.8.3 - in the newest [QGIS](qgis.md) in Puhti and LUMI
* 3.6.2 - in the newest [geoconda](geoconda.md) in Puhti
* 3.4.3 stand-alone: `gdal` in Puhti
* 3.4.1 - [QGIS-3.31 module](qgis.md) in Puhti and LUMI
* Also in Puhti: [r-env](r-env-for-gis.md#gdal-and-saga-gis-support) and [OrfeoToolBox](otb.md)

!!! note
The stand-alone version doesn't have python bindings installed so e.g __gdal_calc__ works only in the geoconda installations. Also, the supported file formats vary slightly between the GDAL installations. For instance, the PostGIS driver is not available in stand-alone gdal but is included in the geoconda versions.
The stand-alone GDAL and R module don't have Python bindings installed so e.g `gdal_calc` works only in the geoconda and qgis modules. Also, the supported file formats vary between the modules. For example, the PostGIS driver is not available in stand-alone GDAL and r-env, but is included in the geoconda and qgis modules. Standalone GDAL also does not support virtual drivers to work with data from HTTPS or S3-links. To standalone and r-env GDAL installations it is possible to add more drivers, please ask. geoconda and qgis GDAL installation are impossible to change regarding drivers support.

## Usage

Expand Down
41 changes: 25 additions & 16 deletions docs/data/datasets/spatial-data-in-csc-computing-env.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,44 @@ Puhti has following datasets:
+ 2m and 10m DEM and infrared orthophotos have virtual rasters, see Puhti virtual rasters below.
+ Stereoclassified lidar data has been slightly modified. The original NLS data had mistakes in headers, these have been fixed. Additionally lax-index files have been added.
+ Automatically classified lidar data, only data of year 2019
- The easiest way to find Paituli raster data is with [Paituli STAC](https://paituli.csc.fi/stac.html), it has also links to Puhti local files.
* **LUKE, multi-source national forest inventory**, 2013, 2015, 2017, 2019 and 2021. LUKE license changed in Aug 2019 to CC BY 4.0.
* **SYKE, All open spatial datasets** available from [SYKE open data service](https://www.syke.fi/fi-FI/Avoin_tieto/Paikkatietoaineistot/Ladattavat_paikkatietoaineistot). Weekly updates.
* **Finnish Forest Centre**, CC BY 4.0 license, **updated in 8/2023**
* [Canopy height model](https://www.paikkatietohakemisto.fi/geonetwork/srv/fin/catalog.search#/metadata/0e7ad446-2999-4c94-ad0d-095991d8f80a)
* [Gridcells](http://www.paikkatietohakemisto.fi/geonetwork/srv/fin/catalog.search#/metadata/3fa1beeb-ea6b-42b1-8e76-eb2bc8ac6d24)
* [Forest mask](https://www.paikkatietohakemisto.fi/geonetwork/srv/fin/catalog.search#/metadata/df99fbd3-44b3-4ffc-b84a-9459f318d545)
* [Forest resource plots](http://www.paikkatietohakemisto.fi/geonetwork/srv/fin/catalog.search#/metadata/332e5abf-63c2-4723-9c2d-4a926bbe587a)
* **Landsat mosaics produced by SYKE and FMI** in Paikkatietoalusta project
- [Historical Landsat satellite image mosaics](https://ckan.ymparisto.fi/dataset/historical-landsat-satellite-image-mosaics-href-historialliset-landsat-kuvamosaiikit-href): 1985, 1990, 1995
- [Historical Landsat NDVI mosaics](https://ckan.ymparisto.fi/dataset/historical-landsat-image-index-mosaics-hind-historialliset-landsat-kuvaindeksimosaiikit-hind): 1984-2011

!!! warning "Satellite mosaics produced by SYKE and FMI in Paikkatietoalusta project were removed from Puhti on 21.11.2023"

The removed datasets were: [Sentinel1 SAR mosaics](https://ckan.ymparisto.fi/dataset/sentinel-1-sar-image-mosaic-s1sar-sentinel-1-sar-kuvamosaiikki-s1sar), [Sentinel2 index mosaics](https://ckan.ymparisto.fi/dataset/sentinel-2-image-index-mosaics-s2ind-sentinel-2-kuvamosaiikit-s2ind), [Historical Landsat satellite image mosaics](https://ckan.ymparisto.fi/dataset/historical-landsat-satellite-image-mosaics-href-historialliset-landsat-kuvamosaiikit-href) and [Historical Landsat NDVI mosaics: 1984-2011](https://ckan.ymparisto.fi/dataset/historical-landsat-image-index-mosaics-hind-historialliset-landsat-kuvaindeksimosaiikit-hind). They are available from FMI's own object storage which has more data than was stored to Puhti local disks. The easiest way to find PTA sentinel mosaics from FMI is with [Paituli STAC](https://paituli.csc.fi/stac.html). Paituli STAC page includes also usage examples for R and Python.

NLS 2m DEM, lidar, infrared ortophotos and all SYKE datasets are updated in Puhti automatically every Monday.

The open spatial data is stored in Puhti: **/appl/data/geo**

Open spatial data in Puhti is maintained by CSC personnel. If you notice any problems with data or wish some new dataset, contact CSC Servicedesk.
Accessing data in Puhti requires CSC user account with a project with Puhti service enabled. All Puhti users have **read** access to these datasets. You do not need to move the files: they can be used directly, unless you need to modify them, which requires you to make your own copy. Open spatial data in Puhti is maintained by CSC personnel. If you notice any problems with data or wish some new dataset, contact CSC Servicedesk.

!!! warning "Sentinel satellite mosaics produced by SYKE and FMI in Paikkatietoalusta project were removed from Puhti on 21.11.2023"

The removed datasets were: [Sentinel1 SAR mosaics](https://ckan.ymparisto.fi/dataset/sentinel-1-sar-image-mosaic-s1sar-sentinel-1-sar-kuvamosaiikki-s1sar), [Sentinel2 index mosaics](https://ckan.ymparisto.fi/dataset/sentinel-2-image-index-mosaics-s2ind-sentinel-2-kuvamosaiikit-s2ind). They are available from FMI's own object storage which has more data than was stored to Puhti local disks. The easiest way to find PTA sentinel mosaics from FMI is with [Paituli STAC](https://paituli.csc.fi/stac.html). Paituli STAC page includes also usage examples for R and Python.

## Spatial data in Allas

CSC computing services users are welcome to share spatial data in [Allas](../Allas/index.md) with other users, if the data license terms allow this. This is a community service, meaning that any CSC user is welcome to contribute and add data to Allas. The data buckets in Allas are owned by data collaborators. If you would like some share some data you have in Allas, and would like the dataset be added to this page, contact CSC Servicedesk

Currently available:

1. **[Sentinel2 2A level images](https://a3s.fi/sentinel-readme/README.txt)**. Maria Yli-Heikkilä (LUKE) and Arttu Kivimäki (NLS/FGI) have downloaded Finnihs data from vegetation period (ca 10.5.-1.9.) in 2016->. The easiest way to find Sentinel data stored in Allas is with [Paituli STAC](https://paituli.csc.fi/stac.html). These files are public, so anybody can download them, also from own computer or other services.

For using data in Allas, see [CSC webinar about Allas and geospatial data](https://youtu.be/mnFXe2-dJ_g) and [Using geospatial files directly from cloud, inc Allas tutorial](../../support/tutorials/gis/gdal_cloud.md).

## Puhti virtual rasters

### Puhti virtual rasters
Virtaul rasters is a very practical concept for working with bigger raster datasets, see CSC [Virtual rasters](../../support/tutorials/gis/virtual-rasters.md) page for longer explanation and how to create own virtual rasters, inc from STAC search results.

CSC has added [virtual rasters](../../support/tutorials/gis/virtual-rasters.md) to NLS 2m and 10m elevation models and infrared ortophotos in Puhti. There are two variants of virtual rasters for the elevation models:
### NLS DEM and orthophotos ready-made virtual rasters
CSC has added to NLS 2m and 10m elevation models and infrared ortophotos in Puhti. There are two variants of virtual rasters for the elevation models:

1. The **direct** virtual rasters contain directly the source tif images without any hierarchical structure, overviews or pre-calculated statistics. The direct virtual raster is meant for using **only in scripts**. It should **not** be opened in QGIS, unless zoomed in and need to open only a few files etc:
* 2m DEM: `/appl/data/geo/mml/dem2m/dem2m_direct.vrt`
Expand All @@ -43,7 +60,7 @@ CSC has added [virtual rasters](../../support/tutorials/gis/virtual-rasters.md)

* 2m DEM: `/appl/data/geo/mml/dem2m/dem2m_hierarchical.vrt`

#### Puhti: create virtual rasters of DEM for custom area
### Puhti: create virtual rasters of DEM for custom area

In some cases it might be useful to create virtual rasters that cover only your study area or some part of it. CSC has made a Python script for creating virtual rasters for custom area from NLS 2m and 10m DEM datasets in Puhti. It's used in the following way:

Expand All @@ -60,15 +77,7 @@ Optional arguments:
* -o: create overviews
* -p: output name prefix

## Spatial data in Allas

CSC computing services users are welcome to share spatial data in [Allas](../Allas/index.md) with other users, if the data license terms allow this. This is a community service, meaning that any CSC user is welcome to contribute and add data to Allas. The data buckets in Allas are owned by data collaborators. If you would like some share some data you have in Allas, and would like the dataset be added to this page, contact CSC Servicedesk

Currently available:

1. **[Sentinel2 2A level images](https://a3s.fi/sentinel-readme/README.txt)**. Maria Yli-Heikkilä (LUKE) has downloaded data of Finland from vegetation period (ca 10.5.-1.9.) in 2016-2020.

For using data in Allas, see [CSC webinar about Allas and geospatial data](https://youtu.be/mnFXe2-dJ_g) and [Using geospatial files directly from cloud, inc Allas tutorial](../../support/tutorials/gis/gdal_cloud.md). The easiest way to find Sentinel data stored in Allas is with [Paituli STAC](https://paituli.csc.fi/stac.html).

## License and acknowledgement

Expand Down
Loading