Categorical missing data imputation for software cost estimation by multinomial logistic regression

Abstract

A common problem in software cost estimation is the manipulation of incomplete or missing data in databases used for the development of prediction models. In such cases, the most popular and simple method of handling missing data is to ignore either the projects or the attributes with missing observations. This technique causes the loss of valuable information and therefore may lead to inaccurate cost estimation models. On the other hand, there are various imputation methods used to estimate the missing values in a data set. These methods are applied mainly on numerical data and produce continuous estimates. However, it is well known that the majority of the cost data sets contain software projects with mostly categorical attributes with many missing values. It is therefore reasonable to use some estimating method producing categorical rather than continuous values. The purpose of this paper is to investigate the possibility of using such a method for estimating categorical missing values in software cost databases. Specifically, the method known as multinomial logistic regression (MLR) is suggested for imputation and is applied on projects of the ISBSG multi-organizational software database. Comparisons of MLR with other techniques for handling missing data, such as listwise deletion (LD), mean imputation (MI), expectation maximization (EM) and regression imputation (RI) under different patterns and percentages of missing data, show the high efficiency of the proposed method.

Introduction

The importance of software cost estimation (SCE) as one of the most crucial phases of software development has been recognized since long. Attempts to estimate the effort and time involved in the development of a software product usually involve the construction of one or more cost estimation models (Shepperd and Schofield, 1997, Jeffery et al., 2000) by applying statistical methods to historical data sets with completed software projects. Most commonly, cost models are obtained by applying regression methods (Angelis et al., 2001, Strike et al., 2001).

A major problem in building such a model arises from the fact that missing values are often encountered in these historical data sets (Strike et al., 2001, Briand et al., 1992). The lack of data values in several important project attributes is a common phenomenon, which may cause misleading results regarding the models’ accuracy and prediction ability. The fact is that most of the software databases suffer from this problem, which is the result of several reasons related to the demanding process of collecting adequate data. Indeed, the data collection requirements include consistence, experience, time, cost and methodology for a company. Furthermore, when the model is based on multi-organizational data, the problem of missing values is caused by the different methods the various companies use to measure and record their data.

There are various techniques dealing with missing data. The most common one, known as listwise deletion (LD), simply ignores the cases with missing observations. The major advantage of the method is its simplicity and the ability to do statistical calculations on a common sample base of cases. The disadvantages of the method are the dramatic loss of information in data sets with high percentages of missing values and the possible bias in the data. These drawbacks are more or less always apparent especially when there is some type of pattern in the missing data, i.e., when the distribution of missing values is depended on certain valid observations in the data.

According to some other techniques, the missing values are replaced by estimates obtained from statistical procedures. The complete data set resulting from such a process is then analyzed by standard statistical methods (for example regression analysis). These techniques are commonly known as imputation methods. The problem is that most of the imputation methods produce in general continuous estimates, which are not realistic replacements of the missing values when the variables are categorical. Since the majority of the variables in the software data sets are categorical with many missing values, it is reasonable to use an imputation method producing categorical values in order to fill the incomplete data set and then to use it for constructing a prediction model.

In this paper we investigate the possibility of using the statistical procedure known as multinomial logistic regression (MLR) as imputation method for categorical variables. The data we used for experimentation and comparisons come from the International Software Benchmarking Standards Group (ISBSG) multi-organizational software database (ISBSG, 2001). MLR was applied on a set of carefully selected software projects that have been pre-processed by statistical analysis and was compared with four other missing data techniques: listwise deletion (LD), mean imputation (MI), expectation maximization (EM) and regression imputation (RI).

In order to compare the efficiency of the above methods in various incompleteness situations, the data set we used was originally complete (no missing values) and the missing values were created by simulating three different mechanisms: missing completely at random (MCAR), missing at random (MAR) and non-ignorable missingness (NIM).

The comparisons were conducted on the basis of the predictive accuracy of a regression model fitted to the data after applying each missing data technique. The results of the study indicate that for small percentages of missing values, MLR gives as satisfactory results as all the other methods do. However, when the percentages of missing values increase, MLR seems to usually outperform with respect to other methods.

The structure of the paper is the following: In Section 2, we outline related work in the area. In Section 3, we describe the different mechanisms used to create missing data and the most common techniques for handling them. In Section 4, we present the data set used in the analysis and the statistical methods in details. In Section 5, we give the results of the statistical analysis conducted using as data the prediction accuracy outputs and finally, in Section 6, we conclude by discussing the findings as well as some directions for future work.