Wrap a Pattern 1 workflow¶
In this tutorial you will wrap the EOAP Pattern 1 workflow. The application workflow has:
- one input parameter of type
Directory - one output parameter of type
Directory
The wrapper exposes URI-compatible inputs and outputs by adding stage-in and stage-out steps around the application workflow.
Build the workflow¶
In [1]:
Copied!
from cwl_loader import dump_cwl
from eoap_cwlwrap import wrap_locations
base_url = "https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main"
workflow_id = "pattern-1"
stage_in_cwl = f"{base_url}/templates/stage-in.cwl"
workflows_cwl = f"{base_url}/cwl-workflow/{workflow_id}.cwl"
directory_stage_out_cwl = f"{base_url}/templates/stage-out.cwl"
main_workflow = wrap_locations(
directory_stage_in=stage_in_cwl,
workflows=f"{workflows_cwl}#{workflow_id}",
directory_stage_out=directory_stage_out_cwl,
)
from cwl_loader import dump_cwl
from eoap_cwlwrap import wrap_locations
base_url = "https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main"
workflow_id = "pattern-1"
stage_in_cwl = f"{base_url}/templates/stage-in.cwl"
workflows_cwl = f"{base_url}/cwl-workflow/{workflow_id}.cwl"
directory_stage_out_cwl = f"{base_url}/templates/stage-out.cwl"
main_workflow = wrap_locations(
directory_stage_in=stage_in_cwl,
workflows=f"{workflows_cwl}#{workflow_id}",
directory_stage_out=directory_stage_out_cwl,
)
2026-06-26 15:22:39.553 | DEBUG | cwl_loader:load_cwl_from_location:369 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-in.cwl...
2026-06-26 15:22:39.584 | DEBUG | cwl_loader:_load_cwl_from_stream:372 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-in.cwl...
2026-06-26 15:22:39.592 | DEBUG | cwl_loader:load_cwl_from_stream:339 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-06-26 15:22:39.592 | DEBUG | cwl_loader:load_cwl_from_yaml:253 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-06-26 15:22:39.592 | DEBUG | cwl_loader:load_cwl_from_yaml:268 - Raw CWL document successfully updated to v1.2!
2026-06-26 15:22:39.593 | DEBUG | cwl_loader:load_cwl_from_yaml:274 - Parsing the raw CWL document to the CWL Utils DOM...
2026-06-26 15:22:39.740 | DEBUG | cwl_loader:load_cwl_from_yaml:283 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-06-26 15:22:39.740 | DEBUG | cwl_loader:load_cwl_from_yaml:285 - Dereferencing the steps[].run...
2026-06-26 15:22:39.741 | DEBUG | cwl_loader:load_cwl_from_yaml:289 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-06-26 15:22:39.741 | DEBUG | cwl_loader:load_cwl_from_yaml:293 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-06-26 15:22:39.742 | DEBUG | cwl_loader:load_cwl_from_yaml:299 - All steps[].run link are resolvable!
2026-06-26 15:22:39.742 | DEBUG | cwl_loader:load_cwl_from_yaml:302 - Sorting Process instances by dependencies....
2026-06-26 15:22:39.743 | DEBUG | cwl_loader:load_cwl_from_yaml:304 - Sorting process is over.
2026-06-26 15:22:39.743 | DEBUG | cwl_loader:_load_cwl_from_stream:382 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-in.cwl successfully load!
2026-06-26 15:22:39.744 | DEBUG | eoap_cwlwrap:_load_process_from_location:655 - 'directory-stage-in' from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-in.cwl is a valid single 'Process'
2026-06-26 15:22:39.744 | DEBUG | cwl_loader:load_cwl_from_location:369 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-out.cwl...
2026-06-26 15:22:39.748 | DEBUG | cwl_loader:_load_cwl_from_stream:372 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-out.cwl...
2026-06-26 15:22:39.758 | DEBUG | cwl_loader:load_cwl_from_stream:339 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-06-26 15:22:39.758 | DEBUG | cwl_loader:load_cwl_from_yaml:253 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-06-26 15:22:39.759 | DEBUG | cwl_loader:load_cwl_from_yaml:268 - Raw CWL document successfully updated to v1.2!
2026-06-26 15:22:39.760 | DEBUG | cwl_loader:load_cwl_from_yaml:274 - Parsing the raw CWL document to the CWL Utils DOM...
2026-06-26 15:22:39.910 | DEBUG | cwl_loader:load_cwl_from_yaml:283 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-06-26 15:22:39.911 | DEBUG | cwl_loader:load_cwl_from_yaml:285 - Dereferencing the steps[].run...
2026-06-26 15:22:39.912 | DEBUG | cwl_loader:load_cwl_from_yaml:289 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-06-26 15:22:39.912 | DEBUG | cwl_loader:load_cwl_from_yaml:293 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-06-26 15:22:39.913 | DEBUG | cwl_loader:load_cwl_from_yaml:299 - All steps[].run link are resolvable!
2026-06-26 15:22:39.913 | DEBUG | cwl_loader:load_cwl_from_yaml:302 - Sorting Process instances by dependencies....
2026-06-26 15:22:39.914 | DEBUG | cwl_loader:load_cwl_from_yaml:304 - Sorting process is over.
2026-06-26 15:22:39.914 | DEBUG | cwl_loader:_load_cwl_from_stream:382 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-out.cwl successfully load!
2026-06-26 15:22:39.915 | DEBUG | eoap_cwlwrap:_load_process_from_location:655 - 'directory-stage-out' from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/templates/stage-out.cwl is a valid single 'Process'
2026-06-26 15:22:39.915 | DEBUG | cwl_loader:load_cwl_from_location:369 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl...
2026-06-26 15:22:39.919 | DEBUG | cwl_loader:_load_cwl_from_stream:372 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl...
2026-06-26 15:22:39.937 | DEBUG | cwl_loader:load_cwl_from_stream:339 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-06-26 15:22:39.937 | DEBUG | cwl_loader:load_cwl_from_yaml:253 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-06-26 15:22:39.938 | DEBUG | cwl_loader:load_cwl_from_yaml:268 - Raw CWL document successfully updated to v1.2!
2026-06-26 15:22:39.938 | DEBUG | cwl_loader:load_cwl_from_yaml:274 - Parsing the raw CWL document to the CWL Utils DOM...
2026-06-26 15:22:40.003 | DEBUG | cwl_loader:load_cwl_from_yaml:283 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-06-26 15:22:40.004 | DEBUG | cwl_loader:load_cwl_from_yaml:285 - Dereferencing the steps[].run...
2026-06-26 15:22:40.004 | DEBUG | cwl_loader:_on_process:175 - Checking if https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl#clt must be externally imported...
2026-06-26 15:22:40.005 | DEBUG | cwl_loader:_on_process:179 - run_url: https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl - uri: https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl
2026-06-26 15:22:40.006 | DEBUG | cwl_loader:load_cwl_from_yaml:289 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-06-26 15:22:40.006 | DEBUG | cwl_loader:load_cwl_from_yaml:293 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-06-26 15:22:40.007 | DEBUG | cwl_loader:load_cwl_from_yaml:299 - All steps[].run link are resolvable!
2026-06-26 15:22:40.008 | DEBUG | cwl_loader:load_cwl_from_yaml:302 - Sorting Process instances by dependencies....
2026-06-26 15:22:40.009 | DEBUG | cwl_loader:load_cwl_from_yaml:304 - Sorting process is over.
2026-06-26 15:22:40.009 | DEBUG | cwl_loader:_load_cwl_from_stream:382 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/main/cwl-workflow/pattern-1.cwl successfully load!
2026-06-26 15:22:40.010 | INFO | eoap_cwlwrap.types:_validate_stage_in:301 - Validating stage-in 'my-asthonishing-stage-in-directory'...
2026-06-26 15:22:40.011 | INFO | eoap_cwlwrap.types:_validate_stage_in:324 - stage-in 'my-asthonishing-stage-in-directory' is valid
2026-06-26 15:22:40.012 | INFO | eoap_cwlwrap.types:_validate_stage_out:354 - Validating stage-out 'my-super-stage-out'...
2026-06-26 15:22:40.012 | INFO | eoap_cwlwrap.types:_validate_stage_out:377 - file stage-out 'my-super-stage-out' is valid
2026-06-26 15:22:40.013 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:170 - Building the CWL Orchestrator Workflow...
2026-06-26 15:22:40.014 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:215 - Analyzing pattern-1 inputs...
2026-06-26 15:22:40.014 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:229 - * pattern-1/aoi: string
2026-06-26 15:22:40.015 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:229 - * pattern-1/epsg: string
2026-06-26 15:22:40.016 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:229 - * pattern-1/bands: string[]
2026-06-26 15:22:40.016 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:229 - * pattern-1/item: Directory
2026-06-26 15:22:40.017 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:246 - Directory type detected, creating a related 'directory_stage_in_0'...
2026-06-26 15:22:40.017 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:250 - Converting Directory to URL-compatible type...
2026-06-26 15:22:40.018 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:258 - Directory converted to https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
2026-06-26 15:22:40.019 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:309 - Connecting 'app/item' to 'directory_stage_in_0' output...
2026-06-26 15:22:40.019 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:353 - Analyzing pattern-1 outputs...
2026-06-26 15:22:40.020 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:358 - * pattern-1/water_bodies: Directory
2026-06-26 15:22:40.020 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:375 - Directory type detected, creating a related 'directory_stage_out_0'...
2026-06-26 15:22:40.021 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:383 - Directory converted to https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
2026-06-26 15:22:40.022 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:434 - Connecting 'app/water_bodies' to 'stage_out_0' output...
2026-06-26 15:22:40.023 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:523 - Orchestrator Workflow built in 0.0096 seconds
Check the generated CWL document¶
In [2]:
Copied!
import sys
dump_cwl(main_workflow, sys.stderr)
import sys
dump_cwl(main_workflow, sys.stderr)
cwlVersion: v1.2
$namespaces:
s: https://schema.org/
schemas:
- http://schema.org/version/9.0/schemaorg-current-http.rdf
s:softwareVersion: 1.0.0
s:applicationCategory: Earth Observation application package
s:keywords:
- '@type': s:DefinedTerm
s:name: application-type
s:description: delineation
- '@type': s:DefinedTerm
s:name: domain
s:description: hydrology
s:thumbnail:
'@type': s:ImageObject
s:contentUrl: https://s3.waw3-2.cloudferro.com/swift/v1/stac-png/S2_L2A.jpg
s:caption: Water bodies detected based on the NDWI and otsu threshold
s:encodingFormat: image/jpeg
s:height: '360'
s:width: '640'
s:license:
'@type': s:CreativeWork
s:name: License CC BY 4.0
s:url: https://creativecommons.org/licenses/by/4.0/
s:encodingFormat: text/html
s:softwareHelp:
- '@type': s:CreativeWork
s:name: User Manual
s:url: https://eoap.github.io/application-package-patterns/
s:encodingFormat: text/html
s:author:
'@type': s:Person
s:givenName: John
s:familyName: Doe
s:affiliation:
'@type': s:Organization
s:name: Make EO Great Again Platform
s:email: john.doe@meogap.org
$graph:
- id: my-asthonishing-stage-in-directory
class: CommandLineTool
inputs:
- id: reference
label: STAC Item URL
doc: A STAC Item to stage
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
- id: another_input
label: Another Input
doc: An additional input for demonstration purposes
type: string
outputs:
- id: staged
type: Directory
outputBinding:
glob: .
requirements:
- class: NetworkAccess
networkAccess: true
- class: SchemaDefRequirement
types:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time/value
type: string
type: record
- class: DockerRequirement
dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
listing:
- entryname: stage.py
entry: |-
import pystac
import stac_asset
import asyncio
import os
import sys
config = stac_asset.Config(warn=True)
async def main(href: str):
item = pystac.read_file(href)
os.makedirs(item.id, exist_ok=True)
cwd = os.getcwd()
os.chdir(item.id)
item = await stac_asset.download_item(item=item, directory=".", config=config)
os.chdir(cwd)
cat = pystac.Catalog(
id="catalog",
description=f"catalog with staged {item.id}",
title=f"catalog with staged {item.id}",
)
cat.add_item(item)
cat.normalize_hrefs("./")
cat.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
return cat
href = sys.argv[1]
empty_arg = sys.argv[2]
cat = asyncio.run(main(href))
cwlVersion: v1.2
baseCommand:
- python
- stage.py
arguments:
- $( inputs.reference.value )
- $( inputs.another_input )
- id: pattern-1
class: Workflow
label: Water bodies detection based on NDWI and the otsu threshold
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
inputs:
- id: aoi
label: area of interest
doc: area of interest as a bounding box
default: -118.985,38.432,-118.183,38.938
type: string
- id: epsg
label: EPSG code
doc: EPSG code
default: EPSG:4326
type: string
- id: bands
label: bands used for the NDWI
doc: bands used for the NDWI
default:
- green
- nir08
type:
name: _:2e0dcfb6-0e2a-4a9d-907f-a78e330d067e
items: string
type: array
- id: item
label: Landsat-8/9 acquisition reference
doc: Landsat-8/9 acquisition reference
type: Directory
outputs:
- id: water_bodies
label: Water bodies detected
doc: Water bodies detected based on the NDWI and otsu threshold
outputSource:
- step/stac-catalog
type: Directory
requirements: []
cwlVersion: v1.2
steps:
- id: step
label: Water bodies detection
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
in:
- id: item
source: item
- id: aoi
source: aoi
- id: epsg
source: epsg
- id: band
source: bands
out:
- stac-catalog
run: '#clt'
- id: main
class: Workflow
label: Workflow pattern-1 orchestrator
doc: This Workflow is used to orchestrate the Workflow pattern-1
inputs:
- id: aoi
label: area of interest - pattern-1/aoi
doc: area of interest as a bounding box - This parameter is derived from
pattern-1/aoi
default: -118.985,38.432,-118.183,38.938
type: string
- id: epsg
label: EPSG code - pattern-1/epsg
doc: EPSG code - This parameter is derived from pattern-1/epsg
default: EPSG:4326
type: string
- id: bands
label: bands used for the NDWI - pattern-1/bands
doc: bands used for the NDWI - This parameter is derived from
pattern-1/bands
default:
- green
- nir08
type:
name: _:2e0dcfb6-0e2a-4a9d-907f-a78e330d067e
items: string
type: array
- id: another_input
label: Another Input - my-asthonishing-stage-in-directory/another_input
doc: An additional input for demonstration purposes - This parameter is
derived from my-asthonishing-stage-in-directory/another_input
type: string
- id: item
label: Landsat-8/9 acquisition reference - pattern-1/item
doc: Landsat-8/9 acquisition reference - This parameter is derived from
pattern-1/item
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
- id: s3_bucket
label: my-super-stage-out/s3_bucket
doc: 'This parameter is derived from: my-super-stage-out/s3_bucket'
type: string
- id: sub_path
label: my-super-stage-out/sub_path
doc: 'This parameter is derived from: my-super-stage-out/sub_path'
type: string
- id: aws_access_key_id
label: my-super-stage-out/aws_access_key_id
doc: 'This parameter is derived from: my-super-stage-out/aws_access_key_id'
type: string
- id: aws_secret_access_key
label: my-super-stage-out/aws_secret_access_key
doc: 'This parameter is derived from: my-super-stage-out/aws_secret_access_key'
type: string
- id: region_name
label: my-super-stage-out/region_name
doc: 'This parameter is derived from: my-super-stage-out/region_name'
type: string
- id: endpoint_url
label: my-super-stage-out/endpoint_url
doc: 'This parameter is derived from: my-super-stage-out/endpoint_url'
type: string
outputs:
- id: water_bodies
label: Water bodies detected
doc: Water bodies detected based on the NDWI and otsu threshold
outputSource: stage_out_0/s3_catalog_output
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
requirements:
- class: SubworkflowFeatureRequirement
- class: SchemaDefRequirement
types:
- $import:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml
steps:
- id: directory_stage_in_0
label: Stage-in 0
doc: Stage-in Directory 0
in:
- id: reference
source: item
- id: another_input
source: another_input
out:
- staged
run: '#my-asthonishing-stage-in-directory'
- id: app
label: Water bodies detection based on NDWI and the otsu threshold
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
in:
- id: aoi
source: aoi
- id: epsg
source: epsg
- id: bands
source: bands
- id: item
source: directory_stage_in_0/staged
out:
- water_bodies
run: '#pattern-1'
- id: stage_out_0
label: Stage-out 0
doc: Stage-out Directory 0
in:
- id: s3_bucket
source: s3_bucket
- id: sub_path
source: sub_path
- id: aws_access_key_id
source: aws_access_key_id
- id: aws_secret_access_key
source: aws_secret_access_key
- id: region_name
source: region_name
- id: endpoint_url
source: endpoint_url
- id: stac_catalog
source: app/water_bodies
out:
- s3_catalog_output
run: '#my-super-stage-out'
- id: clt
class: CommandLineTool
inputs:
- id: item
type: Directory
inputBinding:
prefix: --input-item
- id: aoi
type: string
inputBinding:
prefix: --aoi
- id: epsg
type: string
inputBinding:
prefix: --epsg
- id: band
type:
- name: _:77e2c9f5-0035-4378-9575-c30603c35d59
items: string
type: array
inputBinding:
prefix: --band
outputs:
- id: stac-catalog
type: Directory
outputBinding:
glob: .
requirements:
- class: InlineJavascriptRequirement
- class: EnvVarRequirement
envDef:
- envName: PATH
envValue: /app/envs/runner/bin
- class: ResourceRequirement
coresMax: 1
ramMax: 512
hints:
- class: DockerRequirement
dockerPull: ghcr.io/eoap/application-package-patterns/runner:0.2.0
cwlVersion: v1.2
baseCommand:
- runner
arguments:
- pattern-1
- id: my-super-stage-out
class: CommandLineTool
doc: Stage-out the results to S3
inputs:
- id: s3_bucket
type: string
- id: sub_path
type: string
- id: aws_access_key_id
type: string
- id: aws_secret_access_key
type: string
- id: region_name
type: string
- id: endpoint_url
type: string
- id: stac_catalog
label: STAC Catalog folder
doc: The folder containing the STAC catalog to stage out
type: Directory
outputs:
- id: s3_catalog_output
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
outputBinding:
loadContents: true
glob: catalog-uri.txt
outputEval: |
${
return { "value": self[0].contents, "type": "https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI" };
}
requirements:
- class: NetworkAccess
networkAccess: true
- class: SchemaDefRequirement
types:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time/value
type: string
type: record
- class: DockerRequirement
dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0
- class: InlineJavascriptRequirement
- class: EnvVarRequirement
envDef:
- envName: aws_access_key_id
envValue: $( inputs.aws_access_key_id )
- envName: aws_secret_access_key
envValue: $( inputs.aws_secret_access_key )
- envName: aws_region_name
envValue: $( inputs.region_name )
- envName: aws_endpoint_url
envValue: $( inputs.endpoint_url )
- class: ResourceRequirement
- class: InitialWorkDirRequirement
listing:
- entryname: stage.py
entry: |-
import os
import sys
import pystac
import botocore
import boto3
import shutil
from pystac.stac_io import DefaultStacIO, StacIO
from urllib.parse import urlparse
cat_url = sys.argv[1]
bucket = sys.argv[2]
subfolder = sys.argv[3]
aws_access_key_id = os.environ["aws_access_key_id"]
aws_secret_access_key = os.environ["aws_secret_access_key"]
region_name = os.environ["aws_region_name"]
endpoint_url = os.environ["aws_endpoint_url"]
shutil.copytree(cat_url, "/tmp/catalog")
cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json"))
class CustomStacIO(DefaultStacIO):
"""Custom STAC IO class that uses boto3 to read from S3."""
def __init__(self):
self.session = botocore.session.Session()
self.s3_client = self.session.create_client(
service_name="s3",
use_ssl=True,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
endpoint_url=endpoint_url,
region_name=region_name,
)
def write_text(self, dest, txt, *args, **kwargs):
parsed = urlparse(dest)
if parsed.scheme == "s3":
self.s3_client.put_object(
Body=txt.encode("UTF-8"),
Bucket=parsed.netloc,
Key=parsed.path[1:],
ContentType="application/geo+json",
)
else:
super().write_text(dest, txt, *args, **kwargs)
# client = boto3.client(
# "s3",
# aws_access_key_id=aws_access_key_id,
# aws_secret_access_key=aws_secret_access_key,
# endpoint_url=endpoint_url,
# region_name=region_name,
# )
# StacIO.set_default(CustomStacIO)
# for item in cat.get_items():
# for key, asset in item.get_assets().items():
# s3_path = os.path.normpath(
# os.path.join(os.path.join(subfolder, item.id, asset.href))
# )
# print(f"upload {asset.href} to s3://{bucket}/{s3_path}",file=sys.stderr)
# client.upload_file(
# asset.get_absolute_href(),
# bucket,
# s3_path,
# )
# asset.href = f"s3://{bucket}/{s3_path}"
# item.add_asset(key, asset)
# cat.normalize_hrefs(f"s3://{bucket}/{subfolder}")
# for item in cat.get_items():
# # upload item to S3
# print(f"upload {item.id} to s3://{bucket}/{subfolder}", file=sys.stderr)
# pystac.write_file(item, item.get_self_href())
# # upload catalog to S3
# print(f"upload catalog.json to s3://{bucket}/{subfolder}", file=sys.stderr)
# pystac.write_file(cat, cat.get_self_href())
print(f"s3://{bucket}/{subfolder}/catalog.json", end="", file=sys.stdout)
cwlVersion: v1.2
baseCommand:
- python
- stage.py
arguments:
- $( inputs.stac_catalog.path )
- $( inputs.s3_bucket )
- ${ var firstPart = (Math.random() * 46656) | 0; var secondPart =
(Math.random() * 46656) | 0; firstPart = ("000" +
firstPart.toString(36)).slice(-3); secondPart = ("000" +
secondPart.toString(36)).slice(-3); return inputs.sub_path + "-" + firstPart
+ secondPart; }
stdout: catalog-uri.txt