old_nesta_daps

[Archived]

Contents:

• Packages
  • Code and scripts
    • Meetup
    • Health data
    • NLP Utils
    • Geo Utils
    • Format Utils
    • Decorators
  • Code auditing
• Production
  • How to put code into production at nesta
  • Code and scripts
    • Routines
    • Batchables
    • ORMs
    • Ontologies and schemas
    • Elasticsearch mappings
    • Luigi Hacks
    • Scripts
    • Elasticsearch
    • Containerised Luigi
• FAQ
  • Where is the data?
  • Why don’t you use Aurora rather than MySQL?
  • Where are the production machines?
  • Where is the latest config?
  • Where do I start with Elasticsearch?
• Troubleshooting
  • I’m having problems using the config files!
  • How do I restart the apache server after downtime?
  • How do I restart the luigi server after downtime?
  • How do I perform initial setup to ensure the batchables will run?
  • How can I send/receive emails from Luigi?
  • How do I add a new user to the server?

Packages

Nesta’s collection of tools for meaty tasks. Any processes that go into production come here first, but there are other good reasons for code to end up here.

Production

Nesta’s production system is based on Luigi pipelines, and are designed to be entirely run on AWS via the batch service. The main Luigi server runs on a persistent EC2 instance. Beyond the well documented Luigi code, the main features of the nesta production system are:

• luigihacks.autobatch, which facilates a managed Luigi.Task which is split, batched and combined in a single step. Currently only synchronous jobs are accepted. Asynchonous jobs (where downstream Luigi.Task jobs can be triggered) are a part of a longer term plan.
• scripts.nesta_prepare_batch which zips up the batchable with the specified environmental files and ships it to AWS S3.
• scripts.nesta_docker_build which builds a specified docker environment and ships it to AWS ECS.

License

MIT © 2018 Nesta