EO Exploitation Platform Common Architecture
Data Access Interface Control Document
EOEPCA.SDD.xxx
COMMENTS and ISSUES |
PDF |
EUROPEAN SPACE AGENCY CONTRACT REPORT |
TELESPAZIO VEGA UK Ltd |
- AMENDMENT HISTORY
-
This document shall be amended by releasing a new edition of the document in its entirety.
The Amendment Record Sheet below records the history and issue status of this document.Table 1. Amendment Record Sheet ISSUE DATE REASON 0.1
dd/mm/yyyy
Initial in-progress draft
1. Introduction
1.1. Purpose and Scope
This document presents the Data Access Interfaces for the Common Architecture.
1.2. Structure of the Document
- Section 2 - Overview
-
Provides an over of the Data Access component, within the context of the wider Common Architecture design.
- Section 3 - [mainDesign]
-
Provides the design of the Data Access component.
1.3. Reference Documents
The following is a list of Reference Documents with a direct bearing on the content of this document.
Reference | Document Details | Version |
---|---|---|
EOEPCA - Use Case Analysis |
Issue 1.0, |
|
Exploitation Platform - Functional Model, |
Issue 1.0, |
|
Thematic Exploitation Platform Open Architecture, |
Issue 1, |
|
OGC Testbed-14: WPS-T Engineering Report, |
18-036r1, |
|
OGC WPS 2.0 REST/JSON Binding Extension, Draft, |
1.0-draft |
|
Common Workflow Language Specifications, |
v1.0.2 |
|
OGC Testbed-13, EP Application Package Engineering Report, |
17-023, |
|
OGC Testbed-13, Application Deployment and Execution Service Engineering Report, |
17-024, |
|
OGC Testbed-14, Application Package Engineering Report, |
18-049r1, |
|
OGC Testbed-14, ADES & EMS Results and Best Practices Engineering Report, |
18-050r1, |
|
OpenSearch GEO: OpenSearch Geo and Time Extensions, |
10-032r8, |
|
OpenSearch EO: OGC OpenSearch Extension for Earth Observation, |
13-026r9, |
|
OGC EO Dataset Metadata GeoJSON(-LD) Encoding Standard, |
17-003r1/17-084 |
|
OGC OpenSearch-EO GeoJSON(-LD) Response Encoding Standard, |
17-047 |
|
The Payment Card Industry Data Security Standard, |
v3.2.1 |
|
CEOS OpenSearch Best Practise, |
v1.2, |
|
OpenID Connect Core 1.0, |
v1.0, |
|
OGC Catalogue Services 3.0 Specification - HTTP Protocol Binding (Catalogue Services for the Web), |
v3.0, |
|
OGC Web Map Server Implementation Specification, |
v1.3.0, |
|
OGC Web Map Tile Service Implementation Standard, |
v1.0.0, |
|
OGC Web Feature Service 2.0 Interface Standard – With Corrigendum, |
v2.0.2, |
|
OGC Web Coverage Service (WCS) 2.1 Interface Standard - Core, |
v2.1, |
|
Web Coverage Processing Service (WCPS) Language Interface Standard, |
v1.0.0, |
|
Amazon Simple Storage Service REST API, |
API Version 2006-03-01 |
1.4. Terminology
The following terms are used in the Master System Design.
Term | Meaning |
---|---|
Admin |
User with administrative capability on the EP |
Algorithm |
A self-contained set of operations to be performed, typically to achieve a desired data manipulation. The algorithm must be implemented (codified) for deployment and execution on the platform. |
Analysis Result |
The Products produced as output of an Interactive Application analysis session. |
Analytics |
A set of activities aimed to discover, interpret and communicate meaningful patters within the data. Analytics considered here are performed manually (or in a semi-automatic way) on-line with the aid of Interactive Applications. |
Application Artefact |
The 'software' component that provides the execution unit of the Application Package. |
Application Deployment and Execution Service (ADES) |
WPS-T (REST/JSON) service that incorporates the Docker execution engine, and is responsible for the execution of the processing service (as a WPS request) within the ‘target’ Exploitation Platform. |
Application Descriptor |
A file that provides the metadata part of the Application Package. Provides all the metadata required to accommodate the processor within the WPS service and make it available for execution. |
Application Package |
A platform independent and self-contained representation of a software item, providing executable, metadata and dependencies such that it can be deployed to and executed within an Exploitation Platform. Comprises the Application Descriptor and the Application Artefact. |
Bulk Processing |
Execution of a Processing Service on large amounts of data specified by AOI and TOI. |
Code |
The codification of an algorithm performed with a given programming language - compiled to Software or directly executed (interpretted) within the platform. |
Compute Platform |
The Platform on which execution occurs (this may differ from the Host or Home platform where federated processing is happening) |
Consumer |
User accessing existing services/products within the EP. Consumers may be scientific/research or commercial, and may or may not be experts of the domain |
Data Access Library |
An abstraction of the interface to the data layer of the resource tier. The library provides bindings for common languages (including python, Javascript) and presents a common object model to the code. |
Development |
The act of building new products/services/applications to be exposed within the platform and made available for users to conduct exploitation activities. Development may be performed inside or outside of the platform. If performed outside, an integration activity will be required to accommodate the developed service so that it is exposed within the platform. |
Discovery |
User finds products/services of interest to them based upon search criteria. |
Execution |
The act to start a Processing Service or an Interactive Application. |
Execution Management Service (EMS) |
The EMS is responsible for the orchestration of workflows, including the possibility of steps running on other (remote) platforms, and the on-demand deployment of processors to local/remote ADES as required. |
Expert |
User developing and integrating added-value to the EP (Scientific Researcher or Service Developer) |
Exploitation Tier |
The Exploitation Tier represents the end-users who exploit the services of the platform to perform analysis, or using high-level applications built-in on top of the platform’s services |
External Application |
An application or script that is developed and executed outside of the Exploitation Platform, but is able to use the data/services of the EP via a programmatic interface (API). |
Guest |
An unregistered User or an unauthenticated Consumer with limited access to the EP’s services |
Home Platform |
The Platform on which a User is based or from which an action was initiated by a User |
Host Platform |
The Platform through which a Resource has been published |
Identity Provider (IdP) |
The source for validating user identity in a federated identity system, (user authentication as a service). |
Interactive Application |
A stand-alone application provided within the exploitation platform for on-line hosted processing. Provides an interactive interface through which the user is able to conduct their analysis of the data, producing Analysis Results as output. Interactive Applications include at least the following types: console application, web application (rich browser interface), remote desktop to a hosted VM. |
Interactive Console Application |
A simple Interactive Application for analysis in which a console interface to a platform-hosted terminal is provided to the user. The console interface can be provided through the user’s browser session or through a remote SSH connection. |
Interactive Remote Desktop |
An Interactive Application for analysis provided as a remote desktop session to an OS-session (or directly to a 'native' application) on the exploitation platform. The user will have access to a number of applications within the hosted OS. The remote desktop session is provided through the user’s web browser. |
Interactive Web Application |
An Interactive Application for analysis provided as a rich user interface through the user’s web browser. |
Key-Value Pair |
A key-value pair (KVP) is an abstract data type that includes a group of key identifiers and a set of associated values. Key-value pairs are frequently used in lookup tables, hash tables and configuration files. |
Kubernetes (K8s) |
Container orchestration system for automating application deployment, scaling and management. |
Login Service |
An encapsulation of Authenticated Login provision within the Exploitation Platform context. The Login Service is an OpenID Connect Provider that is used purely for authentication. It acts as a Relying Party in flows with external IdPs to obtain access to the user’s identity. |
EO Network of Resources |
The coordinated collection of European EO resources (platforms, data sources, etc.). |
Object Store |
A computer data storage architecture that manages data as objects. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier. |
On-demand Processing Service |
A Processing Service whose execution is initiated directly by the user on an ad-hoc basis. |
Platform (EP) |
An on-line collection of products, services and tools for exploitation of EO data |
Platform Tier |
The Platform Tier represents the Exploitation Platform and the services it offers to end-users |
Processing |
A set of pre-defined activities that interact to achieve a result. For the exploitation platform, comprises on-line processing to derive data products from input data, conducted by a hosted processing service execution. |
Processing Result |
The Products produced as output of a Processing Service execution. |
Processing Service |
A non-interactive data processing that has a well-defined set of input data types, input parameterisation, producing Processing Results with a well-defined output data type. |
Products |
EO data (commercial and non-commercial) and Value-added products and made available through the EP. It is assumed that the Hosting Environment for the EP makes available an existing supply of EO Data |
Resource |
A entity, such as a Product, Processing Service or Interactive Application, which is of interest to a user, is indexed in a catalogue and can be returned as a single meaningful search result |
Resource Tier |
The Resource Tier represents the hosting infrastructure and provides the EO data, storage and compute upon which the exploitation platform is deployed |
Reusable Research Object |
An encapsulation of some research/analysis that describes all aspects required to reproduce the analysis, including data used, processing performed etc. |
Scientific Researcher |
Expert user with the objective to perform scientific research. Having minimal IT knowledge with no desire to acquire it, they want the effort for the translation of their algorithm into a service/product to be minimised by the platform. |
Service Developer |
Expert user with the objective to provide a performing, stable and reliable service/product. Having deeper IT knowledge or a willingness to acquire it, they require deeper access to the platform IT functionalities for optimisation of their algorithm. |
Software |
The compilation of code into a binary program to be executed within the platform on-line computing environment. |
Systematic Processing Service |
A Processing Service whose execution is initiated automatically (on behalf of a user), either according to a schedule (routine) or triggered by an event (e.g. arrival of new data). |
Terms & Conditions (T&Cs) |
The obligations that the user agrees to abide by in regard of usage of products/services of the platform. T&Cs are set by the provider of each product/service. |
Transactional Web Processing Service (WPS-T) |
Transactional extension to WPS that allows adhoc deployment / undeployment of user-provided processors. |
User |
An individual using the EP, of any type (Admin/Consumer/Expert/Guest) |
Value-added products |
Products generated from processing services of the EP (or external processing) and made available through the EP. This includes products uploaded to the EP by users and published for collaborative consumption |
Visualisation |
To obtain a visual representation of any data/products held within the platform - presented to the user within their web browser session. |
Web Coverage Service (WCS) |
OGC standard that provides an open specification for sharing raster datasets on the web. |
Web Coverage Processing Service (WCPS) |
OGC standard that defines a protocol-independent language for the extraction, processing, and analysis of multi-dimentional coverages representing sensor, image, or statistics data. |
Web Feature Service (WFS) |
OGC standard that makes geographic feature data (vector geospatial datasets) available on the web. |
Web Map Service (WMS) |
OGC standard that provides a simple HTTP interface for requesting geo-registered map images from one or more distributed geospatial databases. |
Web Map Tile Service (WMTS) |
OGC standard that provides a simple HTTP interface for requesting map tiles of spatially referenced data using the images with predefined content, extent, and resolution. |
Web Processing Services (WPS) |
OGC standard that defines how a client can request the execution of a process, and how the output from the process is handled. |
Workspace |
A user-scoped 'container' in the EP, in which each user maintains their own links to resources (products and services) that have been collected by a user during their usage of the EP. The workspace acts as the hub for a user’s exploitation activities within the EP |
1.5. Glossary
The following acronyms and abbreviations have been used in this report.
Term | Definition |
---|---|
AAI |
Authentication & Authorization Infrastructure |
ABAC |
Attribute Based Access Control |
ADES |
Application Deployment and Execution Service |
ALFA |
Abbreviated Language For Authorization |
AOI |
Area of Interest |
API |
Application Programming Interface |
CMS |
Content Management System |
CWL |
Common Workflow Language |
DAL |
Data Access Library |
EMS |
Execution Management Service |
EO |
Earth Observation |
EP |
Exploitation Platform |
FUSE |
Filesystem in Userspace |
GeoXACML |
Geo-specific extension to the XACML Policy Language |
IAM |
Identity and Access Management |
IdP |
Identity Provider |
JSON |
JavaScript Object Notation |
K8s |
Kubernetes |
KVP |
Key-value Pair |
M2M |
Machine-to-machine |
OGC |
Open Geospatial Consortium |
PDE |
Processor Development Environment |
PDP |
Policy Decision Point |
PEP |
Policy Enforcement Point |
PIP |
Policy Information Point |
RBAC |
Role Based Access Control |
REST |
Representational State Transfer |
SSH |
Secure Shell |
TOI |
Time of Interest |
UMA |
User-Managed Access |
VNC |
Virtual Network Computing |
WCS |
Web Coverage Service |
WCPS |
Web Coverage Processing Service |
WFS |
Web Feature Service |
WMS |
Web Map Service |
WMTS |
Web Map Tile Service |
WPS |
Web Processing Service |
WPS-T |
Transactional Web Processing Service |
XACML |
eXtensible Access Control Markup Language |
2. Overview
TBD
3. Data Access Interfaces
3.1. Enpoints
3.1.1. Cache
3.1.1.1. GET Capabilities
GET /cache/ows/wmts/1.0.0/WMTSCapabilities.xml
3.1.1.1.1. Description
Returns the WMTS capabilities document, detailing all available tilesets, their respective grids and formats.
3.1.1.1.2. Content Type
-
text/xml
3.1.1.1.3. Responses
Code | Message | Datatype |
---|---|---|
200 |
The WMTS Capabilities Document |
text/xml |
3.1.1.2. GET Tile
GET /cache/ows/wmts/1.0.0/{layer}/{style}/{starttime}--{endtime}/{tilematrixset}/{tilematrix}/{tilerow}/{tilecol}.{extension}`
3.1.1.2.1. Description
Returns the specified tile from the given layer in the specified style from the
3.1.1.2.2. Parameters
Name | Description | Required |
---|---|---|
layer |
The data layer to retrieve tiles from |
X |
style |
The layer rendering style |
X |
starttime |
The start time of the time-dimension to include tiles from |
X |
endtime |
The end time of the time-dimension to include tiles from |
X |
tilematrixset |
X |
|
tilematrix |
The used tile matrix set |
X |
tilematrix |
The tile matrix (zoom level) to get the tiles from |
X |
tilerow |
The tiles Y coordinate (row) |
X |
tilecol |
The tiles X coordinate (column) |
X |
extension |
The filename extension. Use |
X |
3.1.1.2.3. Content Type
-
image/jpeg
-
image/png
3.1.1.2.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
The resulting image tile. |
image/jpeg / image/png |
404 |
The tile was not found. Returns an empty image. |
image/jpeg / image/png |
500 |
3.1.2. Renderer
3.1.2.1. OpenSearch Description
GET /opensearch/
3.1.2.1.1. Description
Returns the root OpenSearch Description document of the Renderer.
3.1.2.1.2. Content Type
-
application/opensearchdescription+xml
3.1.2.1.3. Responses
Code | Message | Datatype |
---|---|---|
200 |
application/opensearchdescription+xml |
|
404 |
||
500 |
3.1.2.2. OpenSearch Collection Search
GET /opensearch/{format}/
3.1.2.2.1. Description
Performs a collection search of all collections on the Renderer using the search parameters provided.
3.1.2.2.2. Parameters
Name | Description | Required |
---|---|---|
format |
The format to encode the search result in. One of |
X |
3.1.2.2.3. Content Type
-
application/atom+xml
-
application/rss+xml
-
application/vnd.google-earth.kml+xml
-
application/vnd.geo+json
-
text/html
3.1.2.2.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
||
404 |
||
500 |
3.1.2.3. OpenSearch Collection Description
GET /opensearch/collections/{collection}
3.1.2.3.1. Description
Returns the description OpenSearch Description document of the Renderer.
3.1.2.3.2. Parameters
Name | Description | Required |
---|---|---|
collection |
The collection to generate the description document |
X |
3.1.2.3.3. Content Type
-
application/opensearchdescription+xml
3.1.2.3.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
application/opensearchdescription+xml |
|
404 |
||
500 |
3.1.2.4. OpenSearch Record Search
GET /opensearch/collections/{collection}/{format}
3.1.2.4.1. Description
Returns the root OpenSearch Description document of the Renderer.
3.1.2.4.2. Parameters
Name | Description | Required |
---|---|---|
collection |
The collection to perform the record search on |
X |
format |
The format to encode the search result in. One of |
X |
q |
||
count |
The maximum amount of elements to be returned in one single response. |
|
startIndex |
The used start index when used when paging through bigger result sets |
|
product |
||
parentIdentifier |
||
productionStatus |
One of |
|
acquisitionType |
One of |
|
orbitNumber |
||
orbitDirection |
One of |
|
track |
||
frame |
||
swathIdentifier |
||
productVersion |
||
productQualityStatus |
One of |
|
productQualityDegradationTag |
||
processorName |
||
processingCenter |
||
creationDate |
||
modificationDate |
||
processingDate |
||
sensorMode |
||
archivingCenter |
||
processingMode |
||
availabilityTime |
||
acquisitionStation |
||
acquisitionSubType |
||
startTimeFromAscendingNode |
||
completionTimeFromAscendingNode |
||
illuminationAzimuthAngle |
||
illuminationZenithAngle |
||
illuminationElevationAngle |
||
polarisationMode |
One of |
|
polarizationChannels |
One of |
|
antennaLookDirection |
One of |
|
minimumIncidenceAngle |
||
maximumIncidenceAngle |
||
acrossTrackIncidenceAngle |
||
alongTrackIncidenceAngle |
||
dopplerFrequency |
||
incidenceAngleVariation |
||
cloudCover |
||
snowCover |
||
lowestLocation |
||
highestLocation |
||
bbox |
The 4-tuple of a WGS84 bounding box for geospatial filters. |
|
geom |
A WKT encoded geometry to use in a geospatial filter. |
|
lon |
The longitude of a point/radius search |
|
lat |
The latitude of a point/radius search |
|
r |
The radious for |
|
georel |
One of |
|
uid |
A specific product ID. |
|
start |
The start date-time for a specified time interval encoded in an ISO8601 datetime string |
|
end |
The end date-time for a specified time interval encoded in an ISO8601 datetime string |
|
timerel |
One of |
|
cql |
A CQL encoded filter |
3.1.2.4.3. Content Type
-
application/atom+xml
-
application/rss+xml
-
application/vnd.google-earth.kml+xml
-
application/vnd.geo+json
-
text/html
3.1.2.4.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
||
404 |
||
500 |
3.1.2.5. WMS Map Rendering
GET /ows
3.1.2.5.1. Description
Renders a map from the specified parameters.
3.1.2.5.2. Parameters
Name | Description | Required |
---|---|---|
service |
The OWS service to use. Fixed to 'WMS' in this request. |
X |
version |
The OWS service version to use. One of '1.0.0', '1.1.0', or '1.3.0'. |
X |
request |
The OWS request type to use. Fixed to 'GetMap' in this request. |
X |
layers |
The comma-separated list of layers to be rendered. Layers specified later will be drawn over layers set earlier in the list |
X |
styles |
A comma separated list of rendering styles (one for each layer). Can be left empty to use default styles. |
X |
format |
The format to encode the render result in. One of |
X |
srs or crs |
Version 1.3 uses |
X |
bbox |
The image bounding box to render the layer from. Expressed in the units of the |
X |
width |
The width of the resulting image. |
X |
height |
The height of the resulting image. |
X |
transparent |
Whether or not to use transparency for not filled areas. Only usable with formats supporting transparency (e.g PNG). Use |
|
bgcolor |
The background color to use for non-filled areas encoded as a hexadecimally encoded RGB sextuplet: RRGGBB. Default is |
|
time |
The time-filter to apply when rendering an image. Use two ISO8601 encoded datetime strings, separated with a '/' to denote the start and the end time bounds. |
X |
3.1.2.5.3. Content Type
-
image/png
-
image/jpeg
-
image/tiff
3.1.2.5.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
The rendered map image |
image/png / image/jpeg / image/tiff |
400 |
Wrong parameter values. |
|
500 |
3.1.2.6. WCS Coverage Description
GET /ows
3.1.2.6.1. Description
Returns the coverage description of a specified coverage.
3.1.2.6.2. Parameters
Name | Description | Required |
---|---|---|
service |
The OWS service to use. Fixed to 'WCS' in this request. |
X |
version |
The OWS service version to use. One of '2.0.0' or '2.1.0'. |
X |
request |
The OWS request type to use. Fixed to 'DescribeCoverage' in this request. |
X |
coverageId |
The coverage to get the description for. |
X |
3.1.2.6.3. Content Type
-
application/xml
3.1.2.6.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
The coverage description |
application/xml |
400 |
Wrong parameter values. |
|
500 |
3.1.2.7. WCS EO Coverage Set Description
GET /ows
3.1.2.7.1. Description
Returns the coverage description of a specified coverage.
3.1.2.7.2. Parameters
Name | Description | Required |
---|---|---|
service |
The OWS service to use. Fixed to 'WCS' in this request. |
X |
version |
The OWS service version to use. One of '2.0.0' or '2.1.0'. |
X |
request |
The OWS request type to use. Fixed to 'DescribeEOCoverageSet' in this request. |
X |
eoId |
The dataset series identifier or coverage identifier to explore. |
X |
eoId |
The dataset series identifier or coverage identifier to explore. |
X |
subset |
The spatial and/or temporal subset in the form: |
|
containment |
Whether the subsets shall be applied as intersections or overlaps. Values are one of |
|
count |
Limit the maximum number of returned records to this amount. |
3.1.2.7.3. Content Type
-
application/xml
3.1.2.7.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
The EO coverage set description |
application/xml |
400 |
Wrong parameter values. |
|
500 |
3.1.2.8. WCS Coverage Rendering
GET /ows
3.1.2.8.1. Description
Renders the specified coverage according to the parameters
3.1.2.8.2. Parameters
Name | Description | Required |
---|---|---|
service |
The OWS service to use. Fixed to 'WCS' in this request. |
X |
version |
The OWS service version to use. One of '2.0.0' or '2.1.0'. |
X |
request |
The OWS request type to use. Fixed to 'GetCoverage' in this request. |
X |
coverageId |
The input coverage to use |
X |
format |
The format to encode the rendered coverage in. One of |
X |
mediatype |
||
subset |
Specifying a dimension-trim or -slice along a given axis. Must be in the form of either |
|
subsettingCrs |
The coordinate reference system the subsets are expressed in. Also defines the output CRS when not explicitly overridden. |
|
outputCrs |
The output coordinate reference system to use. If necessary, the output coverage will be transformed to that CRS. |
|
scaleFactor |
Specifies a scale factor to be applied on all axes. The size of each axis will be increased (or decreased) by that factor. |
|
scaleAxes |
A comma-separated list of items in the form: |
|
scaleSize |
A comma-separated list of items in the form: |
|
scaleExtent |
||
interpolation |
Define the interpolation method when reprojecting or scaling data. One of http://www.opengis.net/def/interpolation/OGC/1/nearest-neighbour (default), http://www.opengis.net/def/interpolation/OGC/1/average, or http://www.opengis.net/def/interpolation/OGC/1/bilinear |
|
geotiff:compression |
Define the internal compression method when output format |
|
geotiff:jpeg_quality |
Define the compression quality when JPEG compression is defined. |
|
geotiff:predictor |
Define the used compression predictor method for LZW or Deflate compression methods. One of: None, Horizontal, or FloatingPoint |
|
geotiff:interleave |
Specify whether the internal sample planes of TIFF results shall be pixel or band interleaved. Value must be one of Pixel, or Band. |
|
geotiff:tiling |
Defines whether or not the resulting TIFF file shall be internally tiled or stripped. Values must either be |
|
geotiff:tilewidth |
Specifies the width of each internal tile when |
|
geotiff:tileheight |
Specifies the height of each internal tile when |
3.1.2.8.3. Content Type
-
image/png
-
image/jpeg
-
image/tiff
3.1.2.8.4. Responses
Code | Message | Datatype |
---|---|---|
200 |
The rendered map image |
image/png / image/jpeg / image/tiff |
400 |
Wrong parameter values. |
|
500 |
<< End of Document >>