A New Framework for Interactive Images Segmentation

Image segmentation has become a widely studied research problem in image processing. There exist different graph based solutions for interactive image segmentation but the domain of image segmentation still needs persistent improvements. The segmentation quality of existing techniques generally depends on the manual input provided in beginning, therefore, these algorithms may not produce quality segmentation with initial seed labels provided by a novice user. In this work we investigated the use of cellular automata in image segmentation and proposed a new algorithm that follows a cellular automaton in label propagation. It incorporates both the pixels’ local and global information in the segmentation process. We introduced the novel global constraints in automata evolution rules; hence proposed scheme of automata evolution is more effective than the automata based earlier evolution schemes. Global constraints are also effective in deceasing the sensitivity towards small changes made in manual input; therefore proposed approach is less dependent on label seed marks. It can produce the quality segmentation with modest user efforts. Segmentation results indicate that the proposed algorithm performs better than the earlier segmentation techniques.


INTRODUCTION
here, also falls in semi-automatic category where the process extracts multiple objects from the image after a user deûnes certain pixels as their initial seeds. The input provided by a user at the start can help to segment particularly the complex images [1]. Cellular Automata were introduced by John Von Neumann and Burks [2].
They have been used for modeling the variety of dynamical systems in different application domains.
Their use in image segmentation has provided the number of advantages over graph-based approaches [3], which are: I mage segmentation is a process which divides an image into multiple homogeneous regions and now it has become a widely studied research problem in image processing. The Segmentation process can be categorized as: fully automatic, semiautomatic(interactive) and completely manual [1]. Fully autonomous segmentation schemes often fail due to the diverse background in images, it has inûuenced a number of researchers towards the wide investigation on semi-automatic (interactive)multi label image segmentation as shown in Fig. 1 There exist several automata-based segmentation approaches [3][4][5][6][7], but their results are highly dependent on label seed-marks. This is because they only use the local smoothness information in automata evolution rules.
This work is mainly focused to minimize the sensitivity towards changes made in label seed marks. To resolve the problem of high dependence on label seed marks, we additionally incorporated the pixels' label-likelihood information and introduced the novel global constraints in automata evolution rules. These constraints were used just to create the barriers against neighbor attacks. In case of multi-label image segmentation we need to use M number of global constraints (i.e. a separate global constraint for each label). We formulated these global constraints in such a way that lower barrier potential for a label is generated only in that region where pixels have relatively same color features as the particular seed pixels have. Lower values of a global constraint are supposed to be defeated by the neighbors so that a particular label may propagate in surroundings while the higher values of a global constraint can not be defeated easily, hence they are supposed to stop the false propagation of that label. This made proposed algorithm more robust and more user friendly than automata-based existing segmentation techniques. The combined use of global constraints and local smoothness term has made proposed method more effective than other segmentation algorithms. It is capable of producing the satisfactory segmentation results even with those user scribbles which have the crude deûnition of labels' initial regions. The Proposed algorithm performs the multi-label image segmentation. Segmentation results show that proposed approach performs better than the other segmentation methods. The computation of global constraints depend on pixels' label-likelihood information. The proposed algorithm uses GMM (Gaussian Mixture Models) [8] separately to estimate the label-likelihood of a label. The description of steps followed in constraint computation is given in Section 3.1. al. [9], but this method become popular after the publication of Boykov et. al. [10]. The techniques [10][11][12][13] consider an image as a graph that consists of nodes and edge-s. The edges contain t-links and n-links for each node. The t-links are used to implement the region-based topological constraints, they are obtained by assigning the negative log likelihood of GMMs (separately for foreground and background) to each pixel. The n-links are used to implement boundary cues and they are obtained by setting boundary penalties. It uses max flow/ min-cut algorithms for the efficient computation of pixel labeling which his globally optimal with respect to the energy cost function deûned in Equation (1). Graphcut algorithm certainly optimizes the total energy cost for binary cut (segmentation with two labels) but it often loses the guaranteed optimality for multi-way cut (segmentation with multi-labels). Pixel labeling is done automatically once a user provides the initial deûnition of object and background regions. Grabcut [14] is the extension of graphcut. The approach did two enhancements: (1) it iteratively re-implement the graphcut algorithm where it re-estimates the GMM at each iteration;

FIG. 1. MULTI-LABEL IMAGE SEGMENTATION
(2) it applies alpha-expansion to compute the pixels' opacity in a narrow region speciûed around the segmentation boundary. To give an initial approximation of the object region a user covers the object of interest by drawing a rectangle around it. Hard segmentation is performed using iterative graphcut as an initial segmentation, then alpha-expansion is applied to implement border matting. Additional seed pixels can be provided to reûne obtained segmentation results similar to the graphcut. Leo Grady [15] formulated the Random Walk algorithm on a graph that took advantage of using combinatorial operators which produce no discretization error or ambiguity. The algorithm is based on a simple mechanism which allows an efûcient computation of a Ktuple vector for each unlabeled pixel by using a small number of pixels with user-deûned labels. The K-tuple vector for a pixel contains its probability of belonging to each of the K-labels. Final segmentation is computed by assigning a label with maximum probability to each pixel.
Random Walk showed some robustness against weak boundaries which often cause segmentation errors. RWR (Random Walk With Restarts) was proposed by Kim et.
al. [16]. It is a generative segmentation approach, which takes the probability of steady-state as part of likelihood term. The steady-state probability was considered because it reûects the image texture effects. The method is based on a graph, and it constructs a generative model for each label independently. The author claimed for its good performance with weak boundaries and texture problems. The Interactive region growing is an effective segmentation method. Heimann et. al. [17] proposed this algorithm to minimize the leaking through weak boundaries and contour correction. Initially, a user deûnes some seed pixels inside the object of interest, then their region grow iteratively by adding the neighboring pixels.
The region grows only when some homogeneity criterion is met. Its major drawback is the leakage, which allows the region growing through weak boundaries. The method is applicable for two labels, object and background only.

Vezhnevets and Konouchine [3] introduced a technique
called "Growcut" based on cellular automaton. Growcut is an interactive segmentation technique, where a user marks certain pixels to specify foreground/background seeds. Initially, the strength of seed pixels is set to 1, and all other pixels to 0. The label of seed pixels iteratively propagates to other unlabeled pixels. The process continues, until an equilibrium is achieved by the automaton. In that labeling process marked pixels try to occupy the neighboring pixels. At each iteration a pixel that is already labeled, attack its neighboring pixels in order to give them its label. The label transfer depends on the strength of the attacking pixel weighted by a function . This function reduces strength of the attacking pixel in case of a higher color contrast between attacking and the target pixels. According to [3], the competition rules are deûned in such away that an attacking pixel can occupy the target pixel only if the attack force of the attacker is greater than or equal to the strength of that pixel. Growcut has multiple beneûts over the graph based globally optimizing segmentation techniqueslike graphcut and grabcut, but it is highly sensitive to the location of user-deûned seed pixels [5].
This dependency is due to the sole reliance on local smoothness information which is used in labels' propagation.

PROPOSED METHOD
The proposed method based on cellular automaton in label propagation by incorporating pixels information bothat global and local levels. The introduction of novel global constraint in label propagation has increased the segmentation accuracy compared toearlier approach [3] based on cellular automata. The algorithm starts like other interactive approaches [7,[15][16][17][18] it consists of a very simple segmentation scheme, where certain image pixels are marked with different color markers. Pixels lying under scribbled marks specify the labels' initial region, which is necessary to start the segmentation process. The process itself consists of two steps; (1) the system computes a constraint vector for each pixel (either labeled or unlabeled); (2) it iteratively evolves the labels' region until an equilibrium is achieved. The iterative labeling process gives feedback to a user, and allows him/her to make instant corrections when the labeling process is being computed. Although we didn't use this feedback but it could be utilized to apply additional seed marks in those cases where reliable image segmentation is difûcult to compute.

Basic Algorithm
A cellular automaton is an algorithm that operates on a lattice of sites p∈P⊆N n . The lattice P which is defned in an n-dimensional cellular space Z, must have the same spatial dimension as the input image. The automaton we followed in this method, is discrete in space and time. It can be taken as triplet A=(S,N,δ) which consist of a nonempty state set S, a neighborhood system N and a local transition function δ:SN→S. In this work the state set of a site contains the site's current label L p , constraint vector φ p , and color feature vector C p . The neighborhood systems generally used are defined by Von Neumann and Moore [2,19] which are mathematically written as follows Equations (2-3): • Von Neumann 4-connected neighborhood system; • Moore 8-connected neighborhood system; Where the 4-connected neighborhood of a targeted site consists of those adjacent sites which have a Manhattan distance of 1 from that site. The 8-connected neighborhood system is based on the Chebyshev distance. According to Equation (2) the sites having a distance of 1 from target are included in the Moore neighborhood of that targeted site. In this work Moore neighborhood system is used for deciding the targets' neighboring sites. The sites included in target neighborhood, are used for determining the current state of that targeted site. The function δ:SN→S defines transition criteria between neighboring sites. It consists of local transition rules which collectively determine the current state of a site by using the previous state of neighboring sites. Once label seed-pixels are specified, the system computes pixels' label-likelihood for that label.
A GMM [8,11] is used to estimate the pixels' labellikelihood information. If seed pixels of a label have multiple colors then they are divided into several homogeneous components (clusters). Each component separately yields the pixels' normal density component.
We formulated a multivariate normal density component as follows: where d,µ and Σ −1 in Equation (4), represents the dimensions, the mean color and covariance of a homogeneous cluster. A GMM that determines the pixels' label-likelihood information is constructed by accumulating the pixels' all density components. The formulation of a GMM with K density components is given as follows: where P rp in Equation (5) Where C p is the pixels' color feature vector deûned in RGB color space. When a user apply label seed marks, the process first finds the labels' seed pixels then it computes the pixels' label-likelihood for each label. This information is used to compute the pixels' constraint vectors. Before starting the automata evolution process we need to assign the initial strength to each labeled site.
The initial strength is assigned as follows: The final goal of image segmentation is to assign each pixel one of the M possible labels. The evolution scheme we defined for label propagation, is based on a cellular automaton. To control the labeling process more precisely, global constraints are used in local transition rules. The Proposed evolution scheme that works with M number of labels is given in Algorithm-1.
If (Force>α a ) then 13 To determine the local smoothness term between attacker and the target, we have defined a linear transfer function as: The function is monotonically decreasing; it is used for determining how much strength of the attacker is being • To occupy an unlabeled site, we need to conquer only the corresponding barrier potential of that site; • To occupy a labeled site, we need to conquer both: (1) The corresponding barrier potential of that site.

Controllable Boundary Smoothing
It is observed in some cases that obtained segmentation results contain the ragged segments' boundary. Boundary rags keep the complete boundary information, they are significant in a few medical applications, where the smallest boundary information is necessary for diagnosis.
Certain boundary rags are ignored in photo editing because the information obtained in boundary rags is not necessary in editing. To obtain the segmentation results having smooth segments' boundary, following two modifications are proposed in local transition rules: (1) A site having too many enemies around, Enemies t (p) > T1, is prohibited to attack its neighbors.
(2) A site surrounded by too many enemies, Enemies t (p) > T2 is compelled to be occupied by the enemy which is lowest in strength.
The number of enemies in the Moore neighborhood of a site is defined as: The smoothness of a segment boundary can be controlled by adjusting the thresholds T1 and T2. When a cellular automaton use the 8-connected Moore neighborhood system, it allows a labeled site to occupy maximum 8 adjacent sites. For T1 = T2 = 8 the system performs no boundary smoothing, these boundary rags can be reduced by using the lower thresholds' value. Boundary smoothing with different thresholds' value is shown in Fig. 3.

User Interaction
A user applies multiple paint strokes on a given image with different color markers. The paint strokes of a color marker specify the seed pixels for a label related to that color marker. Once the labels' seed marks are specified by a user, the system follows a sequence of steps as follows: • Estimate the pixels' label-likelihood for each label as defined in Equation (4).
• Compute a constraint vector for each pixel.
• Set the initial strength of seed marks as defined in Equation (7).
• Propagate the labels' region to assign each pixel one of the M possible labels.

•
The necessary information obtained from seed marks, is often sufficient to produce satisfactory segmentation results, but this does not happens always. In such cases where the necessary information provided in the beginning is not Segmentation results with different sets of labels' seed marks also reflect the stability and robustness of the proposed method. It can be seen in Fig. 4 and Table 1.
The dataset consists of 151 images, 99 from PASCAL VOC (Visual Object Classes) [13], 49 from Grab-cut data [14] and 3 from Alpha-matting database. The PASSCAL VOC has become a benchmark for interactive image segmentation [20]. A semiautomatic segmentation approach is evaluated by considering the following three criteria [3]: • User convenience.

User Convenience
The overall manual input required by a semi-automatic approach determines, either image segmentation with that approach, is convenient to a user or not. A segmentation scheme which requires simple manual input, can make image segmentation more convenient but the existing techniques are not capable of producing quality segmentation with simply defined manual input.
In this work we investigated, how to decrease the sensitivity towards small variations made in user manual input. The proposed method has minimized this sensitivity; hence the improvement in segmentation results shown in Fig. 4 indicates that proposed method is more robust against the variations made in userdefined label marks.

Speed
Algorithm speed determines the time an algorithm takes to perform the task. It is an important factor that determines how much time-efficient is the algorithm?
System with the following configuration is used to perform the segmentation process.

TABLE 1. ALGORITHMS' RUN TIME (SECONDS) FOR COMPARATIVE SEGMENTATION RESULTS
The proposed method takes more time than the original growcutalgorithm, its constrained label propagation needs more iterations to bypass the unlikely regions.
The run time shown for the proposed method is its total evolution time, while the desired output is achieved much earlier (when 80% of total iterations are performed) and later evolution does not change the segmentation results. Like [7], the speed of proposed algorithm can be significantly enhanced by its parallel implementation.

Segmentation Quality
Experiments show that the quality of segmentation results somehow depends on the manual input provided by a user. The label marks which cover more critical regions surely produce the good segmentation results. According to [3] the segmentation results of each semi-automatic approach can be improved by providing the proper definition of labels' initial region.
Image segmentation with two labels can be evaluated because the data-sets contain 2-label segmentation masks (ground-truth). Most of the data-sets lack segmentation masks with more than two labels, hence the evaluation of multi-label image segmentation is not possible yet. Segmentation results with 2-labels are evaluated by taking several quantitative measures [24][25][26][27]. Region-based evaluation determines proportional region-overlap between mask and output. It is performed by taking several quantitative measures (accuracy, precision and recall) [26][27][28][29]. These measures determine the fraction of truly classified region in image, output and mask. Boundary-based evaluation determines the match between mask and output contours. It is performed by taking precision recall and boundary proximity [24,29]. Precision and recall determine the fraction of matched contour. Boundary proximity determines the nearness between two contours and it is found by determining the accumulated distance of all boundary pixels. Here two arrays B O and B G contain the location of output and ground-truth boundary pixels. We formulated the accumulated distance as: • The distance of output boundary pixels from the nearest ground-truth boundary pixels ( ) ( ) The distance of mask boundary pixels from the nearest output boundary pixels: where K1 and K2 represent the length of B o and B G respectively. According to [21] the proximity between two boundaries (i.e. output and mask boundary) is formulated as: The quantitative evaluation for segmentation results shown in Fig. 5, is illustrated in Fig. 6. The evaluation indicates that proposed method produces quality segmentation at modest user efforts.

CONCLUSION
The segmentation results of a semi-automatic approach somehow depend on the manual input provided in the beginning. Each algorithm is capable of producing good segmentation results but this is true in those cases where a well-defined manual input is provided. A user needs extra drawing skills to provide well-defined label seed marks, therefore existing segmentation schemes are not convenient to a user. The proposed method follows a cellular automaton, it may have several advantages (mentioned in Section 1) over the graph based approaches.