Welcome to nesta! This repository houses our fully-audited tools and packages, as well as our in-house production system. If you’re reading this on our GitHub repo, you will find complete documentation at our Read the Docs site.
Nesta’s collection of tools for meaty tasks. Any processes that go into production come here first, but there are other good reasons for code to end up here.
Nesta’s production system is based on Luigi pipelines, and are designed to be entirely run on AWS via the batch service. The main Luigi server runs on a persistent EC2 instance. Beyond the well documented Luigi code, the main features of the nesta production system are:
luigihacks.autobatch, which facilates a managed
Luigi.Taskwhich is split, batched and combined in a single step. Currently only synchronous jobs are accepted. Asynchonous jobs (where downstream
Luigi.Taskjobs can be triggered) are a part of a longer term plan.
scripts.nesta_prepare_batchwhich zips up the batchable with the specified environmental files and ships it to AWS S3.
scripts.nesta_docker_buildwhich builds a specified docker environment and ships it to AWS ECS.