Wrap Up

Overview

Teaching: 15 min
Exercises: 15 min

Questions

• What did we just learn?

• Where can I go for more information?

• What other training is available?

Objectives

• Summarize and recap the lessons

• Learn about NextFlow training

• Give feedback on training

Summary

• HPC is more than just lots of desktops joined together
• Multiple users share the system and run jobs via a queue scheduler
• Plan your work and map it onto a workflow for SLURM to execute
• Use array jobs and dependencies for easier workflow management
• Find software with LMOD or Singularity / Docker hub
• Parallelism is not about optimization per se
• Understanding your workflow is key to planning your parallel processing
• Start with what is easy, make things harder only when necessary
• If using python leverage numpy data structures and functions
• HPC time is like telescope time - you need science + resource justification
• The best way of estimating resources is to run your workflow on test data

Further reading

• Learning resources on the ADACS LMS page
• How-to videos on the ADACS YouTube channel
• Links to previous training events on the ADACS workshop page
• NCMAS info page
• OzSTAR user documentation
• Pawsey user documentation
• NCI user training

NextFlow training

Soon (date TBC) ADACS will be running a training session focused on the NextFlow workflow manager. This training will build on much of what we have learned here, with a focus on building workflows that take a lot of the pain out of scheduling and monitoring jobs and managing singularity containers. When we have a date for this training we will ping back to this list to invite people to participate. Until then, practice what you have learned.

Survey time

Please fill in this feedback survey so that we can continue to provide training like this and improve.

Key Points

• Practice makes better

• More training available in the future

• Feedback helps us help you