Metadata-Version: 2.4
Name: cumulus-message-adapter-python
Version: 2.4.0
Summary: A handler library for cumulus tasks written in python
Home-page: https://github.com/cumulus-nasa/cumulus-message-adapter-python
Author: Cumulus Authors
Author-email: info@developmentseed.org
License: Copyright © 2018 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.
        
        Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
        You may obtain a copy of the License at
        
            http://www.apache.org/licenses/LICENSE-2.0
        
        Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS,
        WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
        
        ---
        
        TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
        1. Definitions.
        
        "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
        
        "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
        
        "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
        
        "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
        
        "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
        
        "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
        
        "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
        
        "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
        
        "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
        
        "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
        
        2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
        
        3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
        
        4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
        
            You must give any other recipients of the Work or Derivative Works a copy of this License; and
            You must cause any modified files to carry prominent notices stating that You changed the files; and
            You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
            If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
        
            You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
        
        5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
        
        6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
        
        7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
        
        8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
        
        9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
        
        END OF TERMS AND CONDITIONS
        
Requires-Python: ~=3.12, <4.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cumulus-message-adapter~=2.0.5
Provides-Extra: dev
Requires-Dist: nose2; extra == "dev"
Requires-Dist: mock; extra == "dev"
Dynamic: author-email
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# cumulus-message-adapter-python

[![CircleCI]](https://circleci.com/gh/nasa/cumulus-message-adapter-python)
[![PyPI version]](https://badge.fury.io/py/cumulus-message-adapter-python)

## What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management
prototype for NASA's future Earth science data streams.

Read the [Cumulus Documentation]

## What is the Cumulus Message Adapter?

The Cumulus Message Adapter is a library that adapts incoming messages in the
Cumulus protocol to a format more easily consumable by Cumulus tasks, invokes
the tasks, and then adapts their response back to the Cumulus message protocol
to be sent to the next task.

## Installation

```plain
pip install cumulus-message-adapter-python
```

## Task definition

In order to use the Cumulus Message Adapter, you will need to create two
methods in your task module: a handler function and a business logic function.

The handler function is a standard Lambda handler function which takes two
parameters (as specified by AWS): `event` and `context`.

The business logic function is where the actual work of your task occurs. It
should take two parameters: `event` and `context`.

The `event` object contains two keys:

* `input` - the task's input, typically the `payload` of the message, produced
  at runtime
* `config` - the task's configuration, with any templated variables resolved

The `context` parameter is the standard Lambda context as passed by AWS.

The return value of the business logic function will be placed in the
`payload` of the resulting Cumulus message.

Expectations for input, config, and return values are all defined by the task,
and should be well documented. Tasks should thoughtfully consider their inputs
and return values, as breaking changes may have cascading effects on tasks
throughout a workflow. Configuration changes are slightly less impactful, but
must be communicated to those using the task.

## Cumulus Message Adapter interface

The Cumulus Message adapter for python provides one method:
`run_cumulus_task`. It takes four parameters:

* `task_function` - the function containing your business logic (as described
  above)
* `cumulus_message` - the event passed by Lambda, and should be a Cumulus
  Message, *or* a CMA parameter encapsulated message (see [Cumulus Workflow
  Documentation](https://nasa.github.io/cumulus/docs/workflows/input_output)):

  ```json
  {
     "cma": {
        "event": "<cumulus message object>",
        "SomeCMAConfigKey": "CMA configuration object>"
     }
  }
  ```

* `context` - the Lambda context
* `schemas` - optional: a dict with `input`, `config`, and `output` properties.
  Each should be a string set to the filepath of the corresponding JSON schema
  file. All three properties of this dict are optional. If ommitted, the message
  adapter will look in `/<task_root>/schemas/<schema_type>.json`, and if not
  found there, will be ignored.
* `taskargs` - Optional. Additional keyword arguments for the `task_function`

## Example

Simple example of using this package's `run_cumulus_task` function as a wrapper
around another function:

```python
>>> from run_cumulus_task import run_cumulus_task

# simple task that returns the event
>>> def task(event, context):
...     return event

# handler that is provided to aws lambda
>>> def handler(event, context):
...     return run_cumulus_task(task, event, context)

```

For a full example see the [example folder](./example).

## Creating a deployment package

Tasks that use this library are just standard AWS Lambda tasks. See
[creating release packages].

## Usage in a Cumulus Deployment

For documenation on how to utilize this package in a Cumulus Deployment, view
the [Cumulus Workflow Documenation].

## Development


### Setting up a Development Environment

Install all runtime and development dependencies using pip and pyproject.toml:

```bash
$ pip install -e '.[dev]'
```

This will install the package in editable mode along with all development dependencies (such as nose2 and mock) required for testing and development.

### Logging with `CumulusLogger`

Included in this package is the `cumulus_logger` which contains a logging class
`CumulusLogger` that standardizes the log format for Cumulus. Methods are
provided to log error, fatal, warning, debug, info, and trace.

**Import the `CumulusLogger` class:**

```python
>>> from cumulus_logger import CumulusLogger

```

**Instantiate the logger inside the task definition (name and level are
optional):**

```python
>>> import logging
>>> logger = CumulusLogger("event_name", logging.ERROR)

```

**Use the logging methods for different levels:**

```python
>>> logger.trace('<your message>')

>>> logger.debug('<your message>')

>>> logger.info('<your message>')

>>> logger.warn('<your message>')

>>> logger.error('<your message>')

>>> logger.fatal('<your message>')

```

**It can also take additional non-keyword and keyword arguments as in Python
Logger.**

The `msg` is the message format string, the `args` and `kwargs` are the
arguments for string formatting.

If `exc_info` in `kwargs` is not `False`, the exception information in the
`exc_info` or `sys.exc_info()` is added to the message.

```python
>>> logger.debug(msg, *args, **kwargs)

```

**Example usage:**

```python
>>> import os
>>> import sys

>>> from run_cumulus_task import run_cumulus_task
>>> from cumulus_logger import CumulusLogger

# instantiate CumulusLogger
>>> logger = CumulusLogger()

>>> def task(event, context):
...     logger.info('task executed')
... 
...     # log error when an exception is caught
...     logger.error("task formatted message {} exc_info ", "bar", exc_info=True)
... 
...     # return the output of the task
...     return { "example": "output" }

>>> def handler(event, context):
...     # make sure event & context metadata is set in the logger
...     logger.setMetadata(event, context)
...     return run_cumulus_task(task, event, context)

```

### Running Tests

Running tests requires [localstack](https://github.com/localstack/localstack).

Tests only require localstack running S3, which can be initiated with the
following command:

```plain
$ SERVICES=s3 localstack start
```

And then you can check tests pass with the following nosetests command:

```plain
$ CUMULUS_ENV=testing nose2
```

### Linting

```plain
$ pylint run_cumulus_task.py
```

## Why?

This approach has a few major advantages:

1. It explicitly prevents tasks from making assumptions about data structures
   like `meta` and `cumulus_meta` that are owned internally and may therefore
   be broken in future updates. To gain access to fields in these structures,
   tasks must be passed the data explicitly in the workflow configuration.
1. It provides clearer ownership of the various data structures. Operators own
   `meta`. Cumulus owns `cumulus_meta`. Tasks define their own `config`,
   `input`, and `output` formats.
1. The Cumulus Message Adapter greatly simplifies running Lambda functions not
   explicitly created for Cumulus.
1. The approach greatly simplifies testing for tasks, as tasks don't need to
   set up cumbersome structures to emulate the message protocol and can just
   test their business function.

## License

[Apache 2.0](LICENSE)

[circleci]:
  https://circleci.com/gh/nasa/cumulus-message-adapter-python.svg?style=svg
[pypi version]:
  https://badge.fury.io/py/cumulus-message-adapter-python.svg
[Cumulus Documentation]:
  https://nasa.github.io/cumulus/
[creating release packages]:
  https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html
[cumulus workflow documenation]:
  https://nasa.github.io/cumulus/docs/workflows/input_output
