Abstract
This chapter will explore the issue of executing High-Throughput Computing (HTC) workflows on managed High Performance Computing (HPC) systems that have been tailored for the execution of “traditional” HPC applications. We will first define data-oriented workflows and HTC, and then highlight some of the common hurdles that exist to executing these workflows on shared HPC resources. Then we will look at Launcher, which is a tool for making large HTC workflows appear—from the HPC system’s perspective—to be a “traditional” simulation workflow. Launcher’s various features are described, including scheduling modes and extensions for use with Intel®;Xeon PhiTM coprocessor cards.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
D.P. Anderson, Boinc: a system for public-resource computing and storage, in Proceedings. Fifth IEEE/ACM International Workshop on Grid Computing, 2004 (IEEE, New York, 2004), pp. 4–10
F. Broquedis, T. Gautier, V. Danjean, libkomp, an efficient openmp runtime system for both fork-join and data flow paradigms, in OpenMP in a Heterogeneous World, ed. by B. Chapman, F. Massaioli, M.S. Müller, M. Rorro. Lecture Notes in Computer Science, vol. 7312 (Springer, Berlin/Heidelberg, 2012), pp. 102–115. doi:10.1007/978-3-642-30961-8_8. http://dx.doi.org/10.1007/978-3-642-30961-8_8
J. Cao, S. Jarvis, S. Saini, G.R. Nudd et al., Gridflow: workflow management for grid computing, in Proceedings of the 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGrid 2003 (IEEE, New York, 2003), pp. 198–205
J. Dean, S. Ghemawat, Mapreduce: simplified data processing on large clusters. Commun. ACM 51 (1), 107–113 (2008)
E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, M.H. Su, K. Vahi, M. Livny, Pegasus: mapping scientific workflows onto the grid, in Grid Computing (Springer, Berlin, 2004), pp. 11–20
D.W. Erwin, D.F. Snelling, Unicore: a grid computing environment, in Euro-Par 2001 Parallel Processing (Springer, Heidelberg, 2001), pp. 825–834
W. Gentzsch, Sun grid engine: towards creating a compute power grid, in Proceedings. First IEEE/ACM International Symposium on Cluster Computing and the Grid, 2001 (IEEE, New York, 2001), pp. 35–36
R.W. Green, OpenMP* thread affinity control (2012), https://software.intel.com/en-us/articles/openmp-thread-affinity-control
B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A.D. Joseph, R.H. Katz, S. Shenker, I. Stoica, Mesos: a platform for fine-grained resource sharing in the data center, in NSDI, vol. 11 (2011), pp. 22–22
C. Newburn, C. Dmitriev, R. Narayanaswamy, J. Wiegert, R. Murty, F. Chinchilla, R. Deodhar, R. McGuire, Offload compiler runtime for the Intel Xeon Phi coprocessor, in 2013 IEEE 27th International Parallel and Distributed Processing Symposium Workshops PhD Forum (IPDPSW) (2013), pp. 1213–1225. doi:10.1109/IPDPSW.2013.251
K. Seymour, H. Nakada, S. Matsuoka, J. Dongarra, C. Lee, H. Casanova, Overview of GridRPC: a remote procedure call API for grid computing, in Grid Computing—GRID 2002 (Springer, Berlin, 2002), pp. 274–278
O. Trott, A.J. Olson, Autodock Vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. J. Comput. Chem. 31 (2), 455–461 (2010)
A. Tsaregorodtsev, M. Bargiotti, N. Brook, A.C. Ramo, G. Castellani, P. Charpentier, C. Cioffi, J. Closier, R.G. Diaz, G. Kuznetsov et al., Dirac: a community grid solution. J. Phys. Conf. Ser. 119, 062048 (2008)
V.K. Vavilapalli, A.C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth et al., Apache Hadoop yarn: yet another resource negotiator, in Proceedings of the 4th Annual Symposium on Cloud Computing (ACM, New York, 2013), p. 5
U. Viswanathan, S.M. Tomlinson, J.M. Fonner, S.A. Mock, S.J. Watowich, Identification of a novel inhibitor of dengue virus protease through use of a virtual screening drug discovery web portal. J. Chem. Inform. Model. 54 (10), 2816–2825 (2014)
L.A. Wilson, J.M. Fonner, Launcher: a shell-based framework for rapid development of parallel parametric studies, in Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment, XSEDE ’14 (ACM, New York, 2014), pp. 40:1–40:8. doi:10.1145/2616498.2616534. http://doi.acm.org/10.1145/2616498.2616534
A.B. Yoo, M.A. Jette, M. Grondona, Slurm: simple linux utility for resource management, in Job Scheduling Strategies for Parallel Processing (Springer, Berlin, 2003), pp. 44–60
J. Yu, R. Buyya, A taxonomy of workflow management systems for grid computing. J. Grid Comput. 3 (3–4), 171–200 (2005)
S. Zhou, LSF: load sharing in large heterogeneous distributed systems, in Proceedings of the Workshop on Cluster Computing (1992)
Acknowledgements
The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources that have contributed to the research reported within this paper.
The Launcher is available for download on GitHub: https://github.com/TACC/launcher.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Wilson, L.A. (2016). Using Managed High Performance Computing Systems for High-Throughput Computing. In: Arora, R. (eds) Conquering Big Data with High Performance Computing. Springer, Cham. https://doi.org/10.1007/978-3-319-33742-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-33742-5_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-33740-1
Online ISBN: 978-3-319-33742-5
eBook Packages: Computer ScienceComputer Science (R0)