Routines

All of our pipelines, implemented as Luigi routines. Some of these pipelines (at least partly) rely on batch computing (via AWS Batch), where the ‘batched’ scripts (run.py modules) are described in core.batchables. Other than luigihacks.autobatch, which is respectively documented, the routine procedure follows the core Luigi documentation.

• Examples
  • S3 Example
  • Database example
• arXiv data (technical research)
  • Collection task
  • Date task
  • arXiv enriched with MAG (API)
  • arXiv enriched with MAG (SPARQL)
  • arXiv enriched with GRID
• CORDIS (EU funded research)
• Crunchbase (private sector companies)
• EURITO (piping data to Elasticsearch)
• Gateway to Research (UK publicly funded research)
• NiH (health research)
• Meetup (social networking data)
  • Topic discovery