@cumulus/ingest

@cumulus/ingest is a collection of modules for discovering and ingesting data.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/ingest

Testing

Running tests locally requires localstack.

With localstack running, you can run tests using:

LOCALSTACK_HOST=localhost npm test

Modules

All modules are accessible using require: require('@cumulus/ingest/<MODULE_NAME>') or import: import <MODULE_NAME> from '@cumulus/ingest/<MODULE_NAME>'.

• consumer - consumer for SQS messages
• crypto - provides encryption and decryption methods with a consistent API but differing mechanisms for dealing with encryption keys
• ftp - for accessing FTP servers
• granule - discovers and ingests granules
• http - for accessing data via HTTP and HTTPS
• lock - creates locks for S3 data
• log - stringifies JS object logs for ElasticSearch indexing
• parse-pdr - tools for validating PDRs and generating PDRD and PAN messages
• pdr - discovers and ingests pdrs
• queue - creates queues for ingesting data
• recursion - handles recursion of a FTP/SFTP list operation
• sftp - for accessing SFTP servers

Contributing

To make a contribution, please see our contributing guidelines.