Wrap a Pattern 1 workflow¶
In this tutorial you will wrap the EOAP Pattern 1 workflow. The application workflow has:
- one input parameter of type
Directory - one output parameter of type
Directory
The wrapper exposes URI-compatible inputs and outputs by adding stage-in and stage-out steps around the application workflow.
Install transpiler-mate-runtime and eoap-cwlwrap in the notebook environment before running these cells.
Build the workflow¶
In [1]:
Copied!
from pathlib import Path
from transpiler_mate.runtime.context_resolver import DefaultTranspilerContextResolver
from eoap_cwlwrap.plugin import CwlWrapOptions, cwlwrap
base_url = "https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop"
workflow_id = "pattern-1"
context = DefaultTranspilerContextResolver().resolve(
f"{base_url}/cwl-workflow/{workflow_id}.cwl#{workflow_id}"
)
options = CwlWrapOptions(
directory_stage_in=f"{base_url}/templates/stage-in.cwl",
directory_stage_out=f"{base_url}/templates/stage-out.cwl",
output=Path("wrapped.cwl"),
)
cwlwrap.execute(context, options)
from pathlib import Path
from transpiler_mate.runtime.context_resolver import DefaultTranspilerContextResolver
from eoap_cwlwrap.plugin import CwlWrapOptions, cwlwrap
base_url = "https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop"
workflow_id = "pattern-1"
context = DefaultTranspilerContextResolver().resolve(
f"{base_url}/cwl-workflow/{workflow_id}.cwl#{workflow_id}"
)
options = CwlWrapOptions(
directory_stage_in=f"{base_url}/templates/stage-in.cwl",
directory_stage_out=f"{base_url}/templates/stage-out.cwl",
output=Path("wrapped.cwl"),
)
cwlwrap.execute(context, options)
2026-09-27 21:36:31.813 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:67 - Mounting 'http://' scheme to 'HTTPAdapter'...
2026-09-27 21:36:31.813 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:69 - Scheme 'http://' successfully mount to 'HTTPAdapter'
2026-09-27 21:36:31.814 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:67 - Mounting 'https://' scheme to 'HTTPAdapter'...
2026-09-27 21:36:31.814 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:69 - Scheme 'https://' successfully mount to 'HTTPAdapter'
2026-09-27 21:36:31.815 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:67 - Mounting 'file://' scheme to 'FileAdapter'...
2026-09-27 21:36:31.816 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:69 - Scheme 'file://' successfully mount to 'FileAdapter'
2026-09-27 21:36:31.816 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:67 - Mounting 'oci://' scheme to 'OCIAdapter'...
2026-09-27 21:36:31.817 | DEBUG | transpiler_mate.runtime.context_resolver:_mount_session:69 - Scheme 'oci://' successfully mount to 'OCIAdapter'
2026-09-27 21:36:31.819 | DEBUG | cwl_loader:load_cwl_from_location:391 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl...
2026-09-27 21:36:31.993 | DEBUG | cwl_loader:_load_cwl_from_stream:401 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl...
2026-09-27 21:36:32.017 | DEBUG | cwl_loader:load_cwl_from_stream:336 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-09-27 21:36:32.017 | DEBUG | cwl_loader:load_cwl_from_yaml:254 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-09-27 21:36:32.020 | DEBUG | cwl_loader:load_cwl_from_yaml:264 - Raw CWL document successfully updated to v1.2!
2026-09-27 21:36:32.021 | DEBUG | cwl_loader:load_cwl_from_yaml:270 - Parsing the raw CWL document to the CWL Utils DOM...
2026-09-27 21:36:32.224 | DEBUG | cwl_loader:load_cwl_from_yaml:279 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-09-27 21:36:32.224 | DEBUG | cwl_loader:load_cwl_from_yaml:281 - Dereferencing the steps[].run...
2026-09-27 21:36:32.225 | DEBUG | cwl_loader._dereference:_dereference_step:148 - Checking if https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl#clt must be externally imported...
2026-09-27 21:36:32.225 | DEBUG | cwl_loader._dereference:_dereference_step:150 - run_url: https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl - uri: https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl
2026-09-27 21:36:32.226 | DEBUG | cwl_loader:load_cwl_from_yaml:290 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-09-27 21:36:32.226 | DEBUG | cwl_loader:load_cwl_from_yaml:294 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-09-27 21:36:32.227 | DEBUG | cwl_loader:load_cwl_from_yaml:298 - All steps[].run link are resolvable!
2026-09-27 21:36:32.227 | DEBUG | cwl_loader:load_cwl_from_yaml:301 - Sorting Process instances by dependencies....
2026-09-27 21:36:32.228 | DEBUG | cwl_loader:load_cwl_from_yaml:303 - Sorting process is over.
2026-09-27 21:36:32.229 | DEBUG | cwl_loader:_load_cwl_from_stream:411 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/cwl-workflow/pattern-1.cwl successfully load!
2026-09-27 21:36:32.299 | DEBUG | cwl_loader:load_cwl_from_location:391 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-in.cwl...
2026-09-27 21:36:32.345 | DEBUG | cwl_loader:_load_cwl_from_stream:401 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-in.cwl...
2026-09-27 21:36:32.363 | DEBUG | cwl_loader:load_cwl_from_stream:336 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-09-27 21:36:32.363 | DEBUG | cwl_loader:load_cwl_from_yaml:254 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-09-27 21:36:32.366 | DEBUG | cwl_loader:load_cwl_from_yaml:264 - Raw CWL document successfully updated to v1.2!
2026-09-27 21:36:32.367 | DEBUG | cwl_loader:load_cwl_from_yaml:270 - Parsing the raw CWL document to the CWL Utils DOM...
2026-09-27 21:36:32.793 | DEBUG | cwl_loader:load_cwl_from_yaml:279 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-09-27 21:36:32.794 | DEBUG | cwl_loader:load_cwl_from_yaml:281 - Dereferencing the steps[].run...
2026-09-27 21:36:32.794 | DEBUG | cwl_loader:load_cwl_from_yaml:290 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-09-27 21:36:32.795 | DEBUG | cwl_loader:load_cwl_from_yaml:294 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-09-27 21:36:32.796 | DEBUG | cwl_loader:load_cwl_from_yaml:298 - All steps[].run link are resolvable!
2026-09-27 21:36:32.796 | DEBUG | cwl_loader:load_cwl_from_yaml:301 - Sorting Process instances by dependencies....
2026-09-27 21:36:32.796 | DEBUG | cwl_loader:load_cwl_from_yaml:303 - Sorting process is over.
2026-09-27 21:36:32.797 | DEBUG | cwl_loader:_load_cwl_from_stream:411 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-in.cwl successfully load!
2026-09-27 21:36:32.804 | DEBUG | cwl_loader:load_cwl_from_location:391 - Loading CWL document from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-out.cwl...
2026-09-27 21:36:32.849 | DEBUG | cwl_loader:_load_cwl_from_stream:401 - Reading stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-out.cwl...
2026-09-27 21:36:32.870 | DEBUG | cwl_loader:load_cwl_from_stream:336 - CWL data of type <class 'ruamel.yaml.comments.CommentedMap'> successfully loaded from stream
2026-09-27 21:36:32.871 | DEBUG | cwl_loader:load_cwl_from_yaml:254 - Updating the model from version 'v1.0' to version 'v1.2'...
2026-09-27 21:36:32.874 | DEBUG | cwl_loader:load_cwl_from_yaml:264 - Raw CWL document successfully updated to v1.2!
2026-09-27 21:36:32.875 | DEBUG | cwl_loader:load_cwl_from_yaml:270 - Parsing the raw CWL document to the CWL Utils DOM...
2026-09-27 21:36:33.256 | DEBUG | cwl_loader:load_cwl_from_yaml:279 - Raw CWL document successfully parsed to the CWL Utils DOM!
2026-09-27 21:36:33.256 | DEBUG | cwl_loader:load_cwl_from_yaml:281 - Dereferencing the steps[].run...
2026-09-27 21:36:33.257 | DEBUG | cwl_loader:load_cwl_from_yaml:290 - steps[].run successfully dereferenced! Dereferencing the FQNs...
2026-09-27 21:36:33.257 | DEBUG | cwl_loader:load_cwl_from_yaml:294 - CWL document successfully dereferenced! Now verifying steps[].run integrity...
2026-09-27 21:36:33.258 | DEBUG | cwl_loader:load_cwl_from_yaml:298 - All steps[].run link are resolvable!
2026-09-27 21:36:33.259 | DEBUG | cwl_loader:load_cwl_from_yaml:301 - Sorting Process instances by dependencies....
2026-09-27 21:36:33.259 | DEBUG | cwl_loader:load_cwl_from_yaml:303 - Sorting process is over.
2026-09-27 21:36:33.260 | DEBUG | cwl_loader:_load_cwl_from_stream:411 - Stream from https://raw.githubusercontent.com/eoap/application-package-patterns/refs/heads/develop/templates/stage-out.cwl successfully load!
2026-09-27 21:36:33.267 | INFO | eoap_cwlwrap.types:_validate_stage_in:282 - Validating stage-in 'my-asthonishing-stage-in-directory'...
2026-09-27 21:36:33.267 | INFO | eoap_cwlwrap.types:_validate_stage_in:303 - stage-in 'my-asthonishing-stage-in-directory' is valid
2026-09-27 21:36:33.268 | INFO | eoap_cwlwrap.types:_validate_stage_out:333 - Validating stage-out 'my-super-stage-out'...
2026-09-27 21:36:33.268 | INFO | eoap_cwlwrap.types:_validate_stage_out:356 - file stage-out 'my-super-stage-out' is valid
2026-09-27 21:36:33.269 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:367 - Building the CWL Orchestrator Workflow...
2026-09-27 21:36:33.269 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:398 - Analyzing pattern-1 inputs...
2026-09-27 21:36:33.270 | INFO | eoap_cwlwrap:_connect_inputs:148 - * pattern-1/aoi: string
2026-09-27 21:36:33.271 | INFO | eoap_cwlwrap:_connect_inputs:148 - * pattern-1/epsg: string
2026-09-27 21:36:33.271 | INFO | eoap_cwlwrap:_connect_inputs:148 - * pattern-1/bands: string[]
2026-09-27 21:36:33.272 | INFO | eoap_cwlwrap:_connect_inputs:148 - * pattern-1/item: Directory
2026-09-27 21:36:33.272 | INFO | eoap_cwlwrap:_connect_inputs:163 - Directory type detected, creating a related 'directory_stage_in_0'...
2026-09-27 21:36:33.273 | INFO | eoap_cwlwrap:_connect_inputs:167 - Converting Directory to URL-compatible type...
2026-09-27 21:36:33.275 | INFO | eoap_cwlwrap:_connect_inputs:173 - Directory converted to https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
2026-09-27 21:36:33.275 | INFO | eoap_cwlwrap:_connect_inputs:197 - Connecting 'app/item' to 'directory_stage_in_0' output...
2026-09-27 21:36:33.276 | INFO | eoap_cwlwrap:_build_orchestrator_workflow:412 - Analyzing pattern-1 outputs...
2026-09-27 21:36:33.276 | INFO | eoap_cwlwrap:_connect_outputs:250 - * pattern-1/water_bodies: Directory
2026-09-27 21:36:33.277 | INFO | eoap_cwlwrap:_connect_outputs:265 - Directory type detected, creating a related 'directory_stage_out_0'...
2026-09-27 21:36:33.277 | INFO | eoap_cwlwrap:_connect_outputs:271 - Directory converted to https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
2026-09-27 21:36:33.279 | INFO | eoap_cwlwrap:_connect_outputs:293 - Connecting 'app/water_bodies' to 'stage_out_0' output...
2026-09-27 21:36:33.280 | SUCCESS | eoap_cwlwrap:_build_orchestrator_workflow:426 - Orchestrator Workflow built in 0.0109 seconds
2026-09-27 21:36:33.280 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:117 - - Checking ResourceRequirement in main...
2026-09-27 21:36:33.281 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:152 - main is not a CommandLineTool instance, skipping.
2026-09-27 21:36:33.281 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:117 - - Checking ResourceRequirement in my-asthonishing-stage-in-directory...
2026-09-27 21:36:33.282 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:148 - ResourceRequirement in my-asthonishing-stage-in-directory adjusted to {'extension_fields': {}, 'loadingOptions': <schema_salad.runtime.LoadingOptions object at 0x7fc3916fbaf0>, 'class_': 'ResourceRequirement', 'coresMin': 2, 'coresMax': 2, 'ramMin': 2000, 'ramMax': 2000, 'tmpdirMin': None, 'tmpdirMax': None, 'outdirMin': None, 'outdirMax': None}
2026-09-27 21:36:33.282 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:117 - - Checking ResourceRequirement in pattern-1...
2026-09-27 21:36:33.282 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:152 - pattern-1 is not a CommandLineTool instance, skipping.
2026-09-27 21:36:33.283 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:117 - - Checking ResourceRequirement in clt...
2026-09-27 21:36:33.284 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:148 - ResourceRequirement in clt adjusted to {'extension_fields': {}, 'loadingOptions': <schema_salad.runtime.LoadingOptions object at 0x7fc3917174d0>, 'class_': 'ResourceRequirement', 'coresMin': 1, 'coresMax': 1, 'ramMin': 512, 'ramMax': 512, 'tmpdirMin': None, 'tmpdirMax': None, 'outdirMin': None, 'outdirMax': None}
2026-09-27 21:36:33.284 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:117 - - Checking ResourceRequirement in my-super-stage-out...
2026-09-27 21:36:33.284 | DEBUG | eoap_cwlwrap.requirements:adjust_resource_requirements:148 - ResourceRequirement in my-super-stage-out adjusted to {'extension_fields': {}, 'loadingOptions': <schema_salad.runtime.LoadingOptions object at 0x7fc391e27ce0>, 'class_': 'ResourceRequirement', 'coresMin': None, 'coresMax': None, 'ramMin': None, 'ramMax': None, 'tmpdirMin': None, 'tmpdirMax': None, 'outdirMin': None, 'outdirMax': None}
2026-09-27 21:36:33.357 | INFO | eoap_cwlwrap.plugin:cwlwrap:160 - New Workflow successfully saved to /home/runner/work/eoap-cwlwrap/eoap-cwlwrap/docs/tutorials/wrapped.cwl!
Check the generated CWL document¶
In [2]:
Copied!
print(options.output.read_text())
print(options.output.read_text())
cwlVersion: v1.2
$graph:
- id: main
class: Workflow
label: Workflow pattern-1 orchestrator
doc: This Workflow is used to orchestrate the Workflow pattern-1
inputs:
- id: aoi
label: area of interest - pattern-1/aoi
doc: area of interest as a bounding box - This parameter is derived from
pattern-1/aoi
default: -118.985,38.432,-118.183,38.938
type: string
- id: epsg
label: EPSG code - pattern-1/epsg
doc: EPSG code - This parameter is derived from pattern-1/epsg
default: EPSG:4326
type: string
- id: bands
label: bands used for the NDWI - pattern-1/bands
doc: bands used for the NDWI - This parameter is derived from
pattern-1/bands
default:
- green
- nir08
type:
name: _:96bf41fc-8b5b-430c-a0f2-d52f59015018
items: string
type: array
- id: another_input
label: Another Input - my-asthonishing-stage-in-directory/another_input
doc: An additional input for demonstration purposes - This parameter is
derived from my-asthonishing-stage-in-directory/another_input
type: string
- id: item
label: Landsat-8/9 acquisition reference - pattern-1/item
doc: Landsat-8/9 acquisition reference - This parameter is derived from
pattern-1/item
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
- id: s3_bucket
label: S3 Bucket name - my-super-stage-out/s3_bucket
doc: The name of the S3 bucket to stage out to - This parameter is derived
from my-super-stage-out/s3_bucket
type: string
- id: sub_path
label: Sub-path in S3 bucket - my-super-stage-out/sub_path
doc: The sub-path within the S3 bucket to stage out to - This parameter is
derived from my-super-stage-out/sub_path
type: string
- id: aws_access_key_id
label: AWS Access Key ID - my-super-stage-out/aws_access_key_id
doc: The AWS Access Key ID - This parameter is derived from
my-super-stage-out/aws_access_key_id
type: string
- id: aws_secret_access_key
label: AWS Secret Access Key - my-super-stage-out/aws_secret_access_key
doc: The AWS Secret Access Key - This parameter is derived from
my-super-stage-out/aws_secret_access_key
type: string
- id: region_name
label: AWS Region Name - my-super-stage-out/region_name
doc: The AWS Region Name - This parameter is derived from
my-super-stage-out/region_name
type: string
- id: endpoint_url
label: AWS Endpoint URL - my-super-stage-out/endpoint_url
doc: The AWS Endpoint URL - This parameter is derived from
my-super-stage-out/endpoint_url
type: string
outputs:
- id: water_bodies
label: Water bodies detected
doc: Water bodies detected based on the NDWI and otsu threshold
outputSource: stage_out_0/s3_catalog_output
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
requirements:
- class: SubworkflowFeatureRequirement
- class: SchemaDefRequirement
types:
- $import:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml
steps:
- id: directory_stage_in_0
label: Stage-in 0
doc: Stage-in Directory 0
in:
- id: reference
source: item
- id: another_input
source: another_input
out:
- staged
run: '#my-asthonishing-stage-in-directory'
- id: app
label: Water bodies detection based on NDWI and the otsu threshold
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
in:
- id: aoi
source: aoi
- id: epsg
source: epsg
- id: bands
source: bands
- id: item
source: directory_stage_in_0/staged
out:
- water_bodies
run: '#pattern-1'
- id: stage_out_0
label: Stage-out 0
doc: Stage-out Directory 0
in:
- id: s3_bucket
source: s3_bucket
- id: sub_path
source: sub_path
- id: aws_access_key_id
source: aws_access_key_id
- id: aws_secret_access_key
source: aws_secret_access_key
- id: region_name
source: region_name
- id: endpoint_url
source: endpoint_url
- id: stac_catalog
source: app/water_bodies
out:
- s3_catalog_output
run: '#my-super-stage-out'
- https://schema.org/name: Astonishing Stage-in from STAC Item
https://schema.org/description: Stages in a STAC Item from a given URL
https://schema.org/dateCreated: '2025-01-01'
https://schema.org/license:
'@type': s:CreativeWork
s:identifier: Apache-2.0
s:name: Apache License 2.0
s:url: https://spdx.org/licenses/Apache-2.0.html
https://schema.org/keywords:
- CWL
- Workflow
- Earth Observation
https://schema.org/operatingSystem:
- Linux
- macOS
- macOS Server
https://schema.org/softwareRequirements:
- https://cwltool.readthedocs.io/en/latest/
- https://www.python.org/
https://schema.org/softwareVersion: 1.0.0
https://schema.org/softwareHelp:
- '@type': s:CreativeWork
s:name: User Manual
s:url: https://eoap.github.io/application-package-patterns/pattern-1/
https://schema.org/publisher:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
https://schema.org/author:
- '@type': s:Role
s:roleName: Project administration
s:startDate: '2025-01-01'
s:additionalType:
https://credit.niso.org/contributor-roles/project-administration/
s:author:
'@type': s:Person
s:givenName: Lois
s:familyName: Lane
s:email: lois.lane@meoga.com
s:identifier: https://orcid.org/0000-9999-0000-9999
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
- '@type': s:Role
s:roleName: Supervision
s:startDate: '2025-01-01'
s:additionalType: https://credit.niso.org/contributor-roles/supervision/
s:author:
'@type': s:Person
s:givenName: Clark
s:familyName: Kent
s:email: clark.kent@meoga.com
s:identifier: https://orcid.org/9999-0000-9999-0000
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
https://schema.org/contributor:
- '@type': s:Role
s:roleName: Software
s:startDate: '2025-01-01'
s:additionalType: https://credit.niso.org/contributor-roles/software/
s:contributor:
'@type': s:Person
s:givenName: Lex
s:familyName: Luthor
s:email: lex.luthor@meoga.com
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
id: my-asthonishing-stage-in-directory
class: CommandLineTool
label: Astonishing Stage-in from STAC Item
doc: Stages in a STAC Item from a given URL
inputs:
- id: reference
label: STAC Item URL
doc: A STAC Item to stage
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
- id: another_input
label: Another Input
doc: An additional input for demonstration purposes
type: string
outputs:
- id: staged
label: Staged Directory
doc: The directory containing the staged STAC Item as a STAC catalog
type: Directory
outputBinding:
glob: .
requirements:
- class: NetworkAccess
networkAccess: true
- class: SchemaDefRequirement
types:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time/value
type: string
type: record
- class: DockerRequirement
dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
listing:
- entryname: stage.py
entry: |-
import pystac
import stac_asset
import asyncio
import os
import sys
config = stac_asset.Config(warn=True)
async def main(href: str):
item = pystac.read_file(href)
os.makedirs(item.id, exist_ok=True)
cwd = os.getcwd()
os.chdir(item.id)
item = await stac_asset.download_item(item=item, directory=".", config=config)
os.chdir(cwd)
cat = pystac.Catalog(
id="catalog",
description=f"catalog with staged {item.id}",
title=f"catalog with staged {item.id}",
)
cat.add_item(item)
cat.normalize_hrefs("./")
cat.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
return cat
href = sys.argv[1]
empty_arg = sys.argv[2]
cat = asyncio.run(main(href))
- class: LoadListingRequirement
loadListing: deep_listing
- class: ResourceRequirement
coresMin: 2
coresMax: 2
ramMin: 2000
ramMax: 2000
cwlVersion: v1.2
baseCommand:
- python
- stage.py
arguments:
- $( inputs.reference.value )
- $( inputs.another_input )
$namespaces:
s: https://schema.org/
- id: pattern-1
class: Workflow
label: Water bodies detection based on NDWI and the otsu threshold
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
inputs:
- id: aoi
label: area of interest
doc: area of interest as a bounding box
default: -118.985,38.432,-118.183,38.938
type: string
- id: epsg
label: EPSG code
doc: EPSG code
default: EPSG:4326
type: string
- id: bands
label: bands used for the NDWI
doc: bands used for the NDWI
default:
- green
- nir08
type:
name: _:96bf41fc-8b5b-430c-a0f2-d52f59015018
items: string
type: array
- id: item
label: Landsat-8/9 acquisition reference
doc: Landsat-8/9 acquisition reference
type: Directory
outputs:
- id: water_bodies
label: Water bodies detected
doc: Water bodies detected based on the NDWI and otsu threshold
outputSource:
- step/stac-catalog
type: Directory
requirements: []
hints:
- class: eoap:JSONSchemaHint
cwlVersion: v1.2
steps:
- id: step
label: Water bodies detection
doc: Water bodies detection based on NDWI and otsu threshold applied to a
single Landsat-8/9 acquisition
in:
- id: item
source: item
- id: aoi
source: aoi
- id: epsg
source: epsg
- id: band
source: bands
out:
- stac-catalog
run: '#clt'
$namespaces: &id001
s: https://schema.org/
- id: clt
class: CommandLineTool
inputs:
- id: item
type: Directory
inputBinding:
prefix: --input-item
- id: aoi
type: string
inputBinding:
prefix: --aoi
- id: epsg
type: string
inputBinding:
prefix: --epsg
- id: band
type:
- name: _:e5a2b7d7-6ad4-4467-8f3c-506bba0c60dd
items: string
type: array
inputBinding:
prefix: --band
outputs:
- id: stac-catalog
type: Directory
outputBinding:
glob: .
requirements:
- class: InlineJavascriptRequirement
- class: EnvVarRequirement
envDef:
- envName: PATH
envValue: /app/envs/runner/bin
- class: ResourceRequirement
coresMin: 1
coresMax: 1
ramMin: 512
ramMax: 512
- class: NetworkAccess
networkAccess: true
- class: LoadListingRequirement
loadListing: deep_listing
hints:
- class: DockerRequirement
dockerPull: ghcr.io/eoap/application-package-patterns/runner:0.2.0
cwlVersion: v1.2
baseCommand:
- runner
arguments:
- pattern-1
$namespaces: *id001
- https://schema.org/name: Super Stage-out to S3
https://schema.org/description: Stage-out the results to S3
https://schema.org/dateCreated: '2025-01-01'
https://schema.org/license:
'@type': s:CreativeWork
s:identifier: Apache-2.0
s:name: Apache License 2.0
s:url: https://spdx.org/licenses/Apache-2.0.html
https://schema.org/keywords:
- CWL
- Workflow
- Earth Observation
https://schema.org/operatingSystem:
- Linux
- macOS
- macOS Server
https://schema.org/softwareRequirements:
- https://cwltool.readthedocs.io/en/latest/
- https://www.python.org/
https://schema.org/softwareVersion: 1.0.0
https://schema.org/softwareHelp:
- '@type': s:CreativeWork
s:name: User Manual
s:url: https://eoap.github.io/application-package-patterns/pattern-1/
https://schema.org/publisher:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
https://schema.org/author:
- '@type': s:Role
s:roleName: Project administration
s:startDate: '2025-01-01'
s:additionalType:
https://credit.niso.org/contributor-roles/project-administration/
s:author:
'@type': s:Person
s:givenName: Lois
s:familyName: Lane
s:email: lois.lane@meoga.com
s:identifier: https://orcid.org/0000-9999-0000-9999
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
- '@type': s:Role
s:roleName: Supervision
s:startDate: '2025-01-01'
s:additionalType: https://credit.niso.org/contributor-roles/supervision/
s:author:
'@type': s:Person
s:givenName: Clark
s:familyName: Kent
s:email: clark.kent@meoga.com
s:identifier: https://orcid.org/9999-0000-9999-0000
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
https://schema.org/contributor:
- '@type': s:Role
s:roleName: Software
s:startDate: '2025-01-01'
s:additionalType: https://credit.niso.org/contributor-roles/software/
s:contributor:
'@type': s:Person
s:givenName: Lex
s:familyName: Luthor
s:email: lex.luthor@meoga.com
s:affiliation:
'@type': s:Organization
s:name: Make Earth Observation Great Again
s:email: info@meoga.com
s:identifier: https://ror.org/9999cx000
id: my-super-stage-out
class: CommandLineTool
label: Super Stage-out to S3
doc: Stage-out the results to S3
inputs:
- id: s3_bucket
label: S3 Bucket name
doc: The name of the S3 bucket to stage out to
type: string
- id: sub_path
label: Sub-path in S3 bucket
doc: The sub-path within the S3 bucket to stage out to
type: string
- id: aws_access_key_id
label: AWS Access Key ID
doc: The AWS Access Key ID
type: string
- id: aws_secret_access_key
label: AWS Secret Access Key
doc: The AWS Secret Access Key
type: string
- id: region_name
label: AWS Region Name
doc: The AWS Region Name
type: string
- id: endpoint_url
label: AWS Endpoint URL
doc: The AWS Endpoint URL
type: string
- id: stac_catalog
label: STAC Catalog folder
doc: The folder containing the STAC catalog to stage out
type: Directory
outputs:
- id: s3_catalog_output
type:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
outputBinding:
loadContents: true
glob: catalog-uri.txt
outputEval: |
${
return { "value": self[0].contents, "type": "https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI" };
}
requirements:
- class: NetworkAccess
networkAccess: true
- class: SchemaDefRequirement
types:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Date/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#DateTime/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Duration/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Email/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Hostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNEmail/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IDNHostname/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv4/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IPv6/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#IRIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#JsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Password/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#RelativeJsonPointer/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#UUID/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URIReference/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URITemplate/value
type: string
type: record
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time
fields:
- name:
https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#Time/value
type: string
type: record
- class: DockerRequirement
dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0
- class: InlineJavascriptRequirement
- class: EnvVarRequirement
envDef:
- envName: aws_access_key_id
envValue: $( inputs.aws_access_key_id )
- envName: aws_secret_access_key
envValue: $( inputs.aws_secret_access_key )
- envName: aws_region_name
envValue: $( inputs.region_name )
- envName: aws_endpoint_url
envValue: $( inputs.endpoint_url )
- class: ResourceRequirement
- class: InitialWorkDirRequirement
listing:
- entryname: stage.py
entry: |-
import os
import sys
import pystac
import botocore
import boto3
import shutil
from pystac.stac_io import DefaultStacIO, StacIO
from urllib.parse import urlparse
cat_url = sys.argv[1]
bucket = sys.argv[2]
subfolder = sys.argv[3]
aws_access_key_id = os.environ["aws_access_key_id"]
aws_secret_access_key = os.environ["aws_secret_access_key"]
region_name = os.environ["aws_region_name"]
endpoint_url = os.environ["aws_endpoint_url"]
shutil.copytree(cat_url, "/tmp/catalog")
cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json"))
class CustomStacIO(DefaultStacIO):
"""Custom STAC IO class that uses boto3 to read from S3."""
def __init__(self):
self.session = botocore.session.Session()
self.s3_client = self.session.create_client(
service_name="s3",
use_ssl=True,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
endpoint_url=endpoint_url,
region_name=region_name,
)
def write_text(self, dest, txt, *args, **kwargs):
parsed = urlparse(dest)
if parsed.scheme == "s3":
self.s3_client.put_object(
Body=txt.encode("UTF-8"),
Bucket=parsed.netloc,
Key=parsed.path[1:],
ContentType="application/geo+json",
)
else:
super().write_text(dest, txt, *args, **kwargs)
# client = boto3.client(
# "s3",
# aws_access_key_id=aws_access_key_id,
# aws_secret_access_key=aws_secret_access_key,
# endpoint_url=endpoint_url,
# region_name=region_name,
# )
# StacIO.set_default(CustomStacIO)
# for item in cat.get_items():
# for key, asset in item.get_assets().items():
# s3_path = os.path.normpath(
# os.path.join(os.path.join(subfolder, item.id, asset.href))
# )
# print(f"upload {asset.href} to s3://{bucket}/{s3_path}",file=sys.stderr)
# client.upload_file(
# asset.get_absolute_href(),
# bucket,
# s3_path,
# )
# asset.href = f"s3://{bucket}/{s3_path}"
# item.add_asset(key, asset)
# cat.normalize_hrefs(f"s3://{bucket}/{subfolder}")
# for item in cat.get_items():
# # upload item to S3
# print(f"upload {item.id} to s3://{bucket}/{subfolder}", file=sys.stderr)
# pystac.write_file(item, item.get_self_href())
# # upload catalog to S3
# print(f"upload catalog.json to s3://{bucket}/{subfolder}", file=sys.stderr)
# pystac.write_file(cat, cat.get_self_href())
print(f"s3://{bucket}/{subfolder}/catalog.json", end="", file=sys.stdout)
- class: LoadListingRequirement
loadListing: deep_listing
cwlVersion: v1.2
baseCommand:
- python
- stage.py
arguments:
- $( inputs.stac_catalog.path )
- $( inputs.s3_bucket )
- ${ var firstPart = (Math.random() * 46656) | 0; var secondPart =
(Math.random() * 46656) | 0; firstPart = ("000" +
firstPart.toString(36)).slice(-3); secondPart = ("000" +
secondPart.toString(36)).slice(-3); return inputs.sub_path + "-" + firstPart
+ secondPart; }
stdout: catalog-uri.txt
$namespaces:
s: https://schema.org/