Ningfang Mi
ABSTRACT
The design of autonomic systems often relies on representative benchmarks for evaluating system performance and scalability. Despite the fact that experimental observations have established that burstiness is a common workload characteristic that has deleterious effects on user-perceived performance, existing client-server benchmarks do not provide mechanisms for injecting burstiness into the workload. In this paper, we introduce a new methodology for generating workloads that emulate the temporal surge phenomenon in a controllable way, thus provide a mechanism that enables testing and evaluation of client-server system performance under reproducible bursty workloads. This new methodology allows to inject different amounts of burstiness into the arrival stream using the index of dispersion, a single parameter that is as simple to use as a turnable knob.
We exemplify the effectiveness of this new methodology by introducing a new module into the TPC-W, a benchmark that is routinely used for capacity planning of e-commerce systems. This new module injects burstiness into the arrival process of clients in a controllable manner, and hence, enables understanding system performance degradation due to burstiness. Detailed experimentation on a real system shows that this benchmark modification can stress the system under different degrees of burstiness, making a strong case for the usefulness of this modification for capacity planning of autonomic systems.
- V. Almeida, M. Arlitt, J. Rolia. Analyzing a web-based system's performance measures at multiple time scales. ACM Perf. Eval. Rev. 30(2), pp. 3--9, Sep. 2002. Google Scholar
Digital Library
- V. Almeida, A. Bestavros, M. Crovella, and A. de Oliveira. Characterizing Reference Locality in the WWW. In IEEE Conference on Parallel and Distributed Information Systems, Miami Beach, Florida, pp. 92--103, Dec. 1996. Google ScholarDigital Library
- M. Arlitt, R. Friedrich, and T. Jin. Workload Characterization of a Web Proxy in a Cable Environment. ACM Perf. Eval. Rev. 27 (2), pp. 25--36, Aug. 1999. Google ScholarDigital Library
- M. Arlitt and T. Jin. Workload characterization of the 1998 world cup web site. Technical Report HPL-1999-35R1, HP Labs Technical Report, 1999.Google Scholar
- M. Arlitt and C. Williamson. Web Server Workload Characterization:the Search for Invariants. In Proc. 1996 ACM Sigmetrics Conf. Measurement & Modeling of Computer Systems Philadelphia, PA, pp. 126--137, May 1996. Google ScholarDigital Library
- P. Barford and M. Crovella. Generating Representative Web Workloads for Network and Server Performance Evaluation. ACM Perf. Eval. Rev. 26 (1), pp. 151--160, 1998. Google ScholarDigital Library
- G. Casale, P. Cremonesi, and R. Turrin. Robust workload estimation in queueing network performance models. In Proc. of Euromicro PDP pp. 183--187, 2008. Google ScholarDigital Library
- G. Casale, E. Zhang, and E. Smirni. KPC-toolbox: Simple yet effective trace fitting using markovian arrival processes. In Proc. of QEST pp. 83--92, 2008. Google ScholarDigital Library
- L. Cherkasova, P. Phaal. Session Based Admission Control: a Mechanism for Peak Load Management of Commercial Web Sites. IEEE J. Transactions on Computers, (TOC), 51 (6), pp. 669--685, June 2002. Google ScholarDigital Library
- M. Crovella and A. Bestravos. Self-Similarity in Word Wide Web Traffic: evidence and possible causes. IEEE/ACM Transactions on Networking, 5 (6), pp. 835--846, 1997. Google ScholarDigital Library
- D. Garcia, J. Garcia. TPC-W E-commerce benchmark evaluation. IEEE Computer pp. 42-48, Feb. 2003. Google ScholarDigital Library
- R. Gusella. Characterizing the variability of arrival processes with indexes of dispersion. IEEE JSAC 19(2), pp. 203--211, 1991.Google Scholar
- K. Kant, V. Tewary, and R. Iyer. An Internet Traffic Generator for Server Architecture Evaluation. In Proc. Workshop Computer Architecture Evaluation Using Commercial Workloads Jan. 2001.Google Scholar
- D. Krishnamurthy and J. Rolia. Predicting the QoS of an Electronic Commerce Server:Those Mean Percentiles. ACM Sigmetrics Performance Evaluation Review, 26 (3), pp. 16--22, December 1998. Google ScholarDigital Library
- D. Krishnamurthy, J. Rolia, and S. Majumdar. A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems. IEEE Transactions on Software Engineering, 32 (11), pp. 868--882, Nov. 2006. Google ScholarDigital Library
- D. Menasce, V. Almeida, R. Reidi, F. Pelegrinelli, R. Fonesca, and W. Meira Jr. In Search of Invariants in E-Business Workloads. In Proc. ACM Conf. Electronic Commerce pp. 56--65, Oct. 2000. Google ScholarDigital Library
- N. Mi, G. Casale, L. Cherkasova, and E. Smirni. Burstiness in multi-tier applications: Symptoms, causes, and new models. In ACM/IFIP/USENIX 9th Int'l Middleware Conf. Leuven, Belgium, pp. 265--286, 2008. Google ScholarDigital Library
- N. Mi, Q. Zhang, A. Riska, E. Smirni, andE. Riedel. Performance impacts of autocorrelated flows in multi-tiered systems. Perform. Eval. 64(9-12), pp. 1082--1101, 2007. Google ScholarDigital Library
- M. F. Neuts. Structured Stochastic Matrices of M/G/1 Type and Their Applications Marcel Dekker, 1989.Google Scholar
- D. Mosberger and T. Jin. httperf: A Tool for Measuring Web Server Performance. In Proc. Workshop Internet Server Performance pp. 59--67, June 1998.Google ScholarDigital Library
- V. Paxon and S. Floyd. Wide Area Traffic:The Failure of Poisson Modeling. IEEE/ACM Trans. Networking, 3 (3), pp. 226--244, June 1995. Google ScholarDigital Library
- S. Ranjan, J. Rolia, H. Fu, E. Knightly. QoS-Driven Server Migration for Internet Data Center. In Proc. Int'l Workshop Quality of Service pp. 3--12, May 2002.Google ScholarCross Ref
- Slashdot effect, Wikipedia.Google Scholar
- A. Williams and M. Arlitt and C. Williamson and K. Barker. Web Workload Characterization: Ten Years Later. 2, pp. 3--21, Springer US, 2005.Google Scholar
- Q. Zhang, L. Cherkasova, and E. Smirni. A regression-based analytic model for dynamic resource provisioning of multi-tier applications. In Proc. of 4th ICAC pp. 27, June 2007. Google ScholarDigital Library
Index Terms
- Injecting realistic burstiness to a traditional client-server benchmark
Recommendations
Subsetting the SPEC CPU2006 benchmark suite
On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for ...
A Benchmark Characterization of the EEMBC Benchmark Suite
Benchmark consumers expect benchmark suites to be complete, accurate, and consistent, and benchmark scores serve as relative measures of performance. However, it is important to understand how benchmarks stress the processors that they aim to test. This ...
Requirements-driven database systems benchmark method
Benchmarks are the vital tools in the performance measurement, evaluation, and comparison of relational database management systems (RDBMS). Standard benchmarks such as the TP1, TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, Wisconsin, and AS3 Ap ...
Comments