The ever-growing choice in diverse services is making service orchestration variability an essential aspect of a composite web service. Influence of this variation on the Quality of Service (QoS) of a composite service is critical and the focus of our work. In this paper, we present a methodology to first model orchestration variability using a feature diagram (FD). The FD specifies a product line of orchestrations represented as configurations of invoked/rejected atomic services. Second, due to the potentially large set of configurations we employ combinatorial testing techniques to automatically generate configurations covering all valid pairwise interactions between services. Third, we analyze QoS variation for each configuration using probabilistic models of QoS. Using a crisis management system case study we experimentally show that pairwise generation covers all QoS outliers and eliminates analysis of > 75% of all possible configurations. The QoS analysis of the pairwise configurations reveals unsafe/ineffective configurations, helps determine realistic Service Level Agreements (SLAs), and provides valuable feedback to help remodel an orchestration.
Keywords: Intelligent Web service languages; Quality concepts; Model Validation and Analysis.