A Survey of Multi Sensor Satellite Image Fusion Techniques

Multi sensor image fusion is the technique used to combine heterogeneous images of the same scene obtained using different sensors. The objective of image fusion is to produce a single image containing the best aspects of the fused images. Some desirable aspects of Image Fusion include high spatial resolution and high spectral resolution (multispectral and panchromatic satellite images), areas in focus (microscopy images), functional and anatomic information (medical images), different spectral information (optical and infrared images), or color information and texture information (multispectral and synthetic aperture radar images). Image fusion can also be used for providing some protection against illegal copying by embedding water-marks. For all of the schemes, it is assumed that the images have been co-registered and resampled. The aim of this survey is to present a review of publications related to Multi Sensor Image Fusion. This paper paints a comprehensive picture of Multi Sensor Image Fusion methods and their applications. This paper is an introduction for those new to the field, an overview for those working in the field and a reference for those searching for literature on a specific application. Methods are classified according to the different aspects of Multi Sensor Image Fusion.


Introduction
With the recent rapid developments in the field of sensing technologies, a wide variety of imaging sensors are available, but it is impossible to capture an image that includes all salient features using only one sensor. To produce a more comprehensive synthetic image for a scene, fusing multiple images is important. Image Fusion is a process of combining the relevant information from a set of images to form a fused image which will be more informative and complete than any of the input images. The main purpose of image fusion is the integration of disparate and complementary data to enhance the information apparent in the images as well as to increase the reliability of the interpretation. This leads to more accurate data and increased utility. It is also stated that a fused data provides robust operational performance, increased confidence, reduced ambiguity, improved reliability and improved classification. Image fusion techniques can improve the quality and increase the accuracy of further processing. With rapid advancements in technology, it is now possible to obtain information from multiple source images to produce a high quality fused image with improved spatial and spectral information. Image fusion is an effective way for optimum utilization of largevolumes of image from multiple sources. Image fusion is a process of combining images, obtained by sensors of different wavelengths viewing the same scene, to form a composite image. The composite image is formed to improve the image content and to make it easier for the user to detect, recognize, and identify targets and increase his situational awareness. The objective of image fusion is to produce a single image containing the best aspects of the fused images. The process of information fusion can be seen as an information transfer problem in which two or more information sets are combined into a new one that should contain all the information from the original sets. During the process of fusion, input images A and B are combined into a new fused image F by transferring, ideally all of their information into F. This is illustrated graphically using a simple Venn diagram in Figure 1. The paper is organized as follows. It begins with an introduction in section 1, followed by multi sensor image fusion in section 2, Classification of different fusion algorithms in spatial domain and in transform domain in section 3, Fusion in Frequency domain in section 4, Hybrid Fusion techniques in section 5 and finallythe survey ends with Conclusion and Discussion in section 6.

Multi Sensor Image Fusion
Multi sensor image fusion is the technique used to combine heterogeneous images of the same scene obtained using different sensors. Combining two or more images from heterogeneous sources usually produce a better application wise visible image [1]. The fusion of different images can reduce the uncertainty related to a single image. The resultant fused images are usually efficiently used in many remote sensing and medical applications, such as target detection, object tracking, weapon detection, night vision, etc. With the availability of multisensor data in many fields, such as remote sensing, medical imaging or machine vision, the sensor fusion has emerged as a new and promising research area. The problem that image fusion tries to solve is to combine information from several sensors taken from thesame scene in order to obtain a newfused image, which contains the best information coming from the original images. Hence, the resultant fused image has better quality than any of the original images. The benefits of multi-sensor image fusion include: 1. Extended range of operation -multiple sensors that operate under different operating conditions can extend the effective range of operation. For example different sensors can be used for day/night operation. 2. Extended spatial and temporal coverage -joint information from sensors that differ in spatial resolution can increase the spatial coverage. The same is true for the temporal dimension. 3. Reduced uncertainty-joint information from multiple sensors can reduce the uncertainty associated with the sensing or decision process. 4. Increased reliability-the fusion of multiple measurements can reduce noise and therefore improve the reliability of the measured quantity. 5. Robust system performance-redundancy in multiple measurements can help in systems robustness. In case one or more sensors fail or the performance of a particular sensor deteriorates, the system can depend on the other sensors Compact representation of information-fusion leads to compact representations. For example, in remote sensing, instead of storing imagery from several spectral bands, it is comparatively more efficient to store the fused information. Image Fusion can be performed at the following three different processing levels. An illustration of the concept of fusion at different processing levels is shown in Figure 2. 1. Pixel level Fusion Image fusion at pixel level means fusion at the lowest processing level referring to the merging of measured physical parameters. An illustration of the concept of image fusionis shown in Figure 2. Pixel based fusion is performed on a pixel-by-pixel basis. It generates a fused image in which the information associated with each pixel is determined from a set of pixels in source images to improve the performance of further processing tasks like segmentation. It is the simplest technique. 2. Feature level Fusion Image fusion at feature level requires the extraction of objects recognized in the various data sources, e.g., using segmentation procedures. Features correspond to characteristics extracted from the initial images which are depending on their environment such as extent, shape and neighborhood. These similar objects (e.g., regions) from multiple sources are assigned to each other and then fused for further assessment using statistical approaches or Artificial Neural Networks (ANN). 3. Decision level Fusion Decision level fusion consists of merging information at a higher level of abstraction. It combines the results from multiple algorithms to yield a final fused decision. Input images are processed individually for information extraction. The obtained information is then combined by applying decision rules to reinforce common interpretation.

Classification of Fusion Algorithms
Image fusion methods can be broadly classified into the following two groups, namely (i) Spatial domain fusion method and (ii

Arithmetic Methods
All operations which deal with mathematical combinations of image channels are grouped together. It comprises addition, subtraction, multiplication and division (ratio) as well as combinations of these operations. Certain combinations of bands of images can lead to image sharpening, i.e., higher spatial resolution. Common combinations are SPOT PAN and SPOT XS or SPOT PAN and Landsat TM. But the integration of SAR can also improve the spatial structure of an image because it introduces the surface roughness to the image.

Addition and Multiplication
To enhance the contrast, addition and multiplication of images are useful. Anexample is the multiplication process expressed by Equation (1) and (2) DN f = A (w 1 D N a + w 2 D N b ) +B (1) Where A and B are scaling factors and w 1 and w 2 weighting parameters. DN f , DNa and DN b refer to digital numbers of the final fused image and the input images a and b, respectively. This method was successfully applied to Landsat-TM and SPOT PAN data.
The multiplicative model or the product fusion method combines two data sets by multiplying each pixel in each band k of MS data by the corresponding pixel of the PAN data. To compensate for the increased brightness, the square root of the mixed data set is taken. The square root of the Multiplicative data set, reduce the data to combination reflecting the mixed spectral properties of both sets. The fusion formula is expressed by Equation (3).

Difference and Ratio Images
Difference and ratio images are very suitable for change detection. The ratio method is useful because of its capability to emphasize more on the slight signature variations. In somecases the resulting difference image contains negative values. Therefore a constant has to be added to produce positive digital numbers. However, differences do not always refer to changes since other factors, like differences in illumination, atmospheric conditions, sensor calibration, ground moisture conditions and registration of the two images, can lead to differences in radiance values. In ratioing two images from different dates are divided, band by band if the image data have more than one channel. If the intensity of the reflected energy is nearly the same in each image then the ratio image pixel is one, it indicates no change. The critical part of this method is the selection of appropriate threshold values in the lower and upper tails of the distribution representing change pixel values. In this respect the normalization of the data is of advantage as indicated in Equation (4).

−
+ C(C = 128 for positivevalues) (4) A ratio for spatial enhancement is summarized by thefollowing Equation (5). The aim of this method is to maintain the radiometric integrity of the data while increasing the spatial resolution.
DN Hybrid XS (j) = (5) where DN HybridXS (i) -i th band of the fused high resolution image, DN PAN -corresponding pixel in high resolution input PAN image, DN XS(i) -super-pixel in i th band of input low resolution XS image, DN SynPAN-corresponding pixel in low resolution synthetic PAN image, created from low resolution multispectral bands that overlap the spectral response of the input high resolution PAN. The algorithm is shown in Equation (6), where DN fused means the DN of the resulting fused image produced from the input data in n multispectral bands multiplied by the high resolution image DN highres.

Principal Component Analysis (PCA)
The fusion method based on PCA is very simple. PCA is a general statistical technique that transforms multivariate datawith correlated variables into one with uncorrelated variables. These new variables are obtained as linear combinationsof theoriginal variables. PCA has been widely used in image encoding, image data compression, image enhancement and image fusion. In the fusion process, PCA method generates uncorrelated images (PC1, PC2, …, PCn, where n is the number of input multispectral bands). As with the IHS scheme, preprocessing steps can be applied before the fusion and histogram matching can be performed between the PAN image and PC1. Land cover types tend to behave similarly in adjacent bands of the spectrum, causing a significant amount of redundancy in information collected by the sensor. By applying a PCA, this redundant information can be organized in such a way that, each output band is uncorrelated with others. PCA has been used for spectral transformation because, the first principal component (PC1) consists of the most variance, making it a suitable choice to replace the PAN component. The first principal component (PC1) is replaced with the panchromatic band, which has higher spatial resolution than the multispectral images. Afterwards, the inverse PCA transformation is applied to obtain the image in the RGB color model. In PCA image fusion, dominant spatial information and weak color information is often a problem. The first principal component, which contains maximum variance, is replaced by PAN image. Such replacement maximizes the effect of panchromatic image in the fused product. One solution could be stretching the principal component to give a spherical distribution. Besides, the PCA approach is sensitive to the choice of area to be fused. If the grey values of the PAN image are adjusted to the grey values similar to PC1 component before the replacement, the color distortion is significantly reduced. The PCA fusion scheme has an advantage over the IHS scheme in that it can be applied to all bands in the MS image simultaneously. However, the image must be primarily vegetation to achieve good results because if large areas of the image have very different spectral characteristics, such as water and vegetation, these spectral differences will be contained in PC1. The PCA approach is sensitive to the choice of the area to be analyzed. Chiang [2] proposed a knowledge based PCA approach for the fusion of SPOT 4 MS and PAN images. When the histograms of PC1 and PAN images are more similar, less spectral information is lost in the fusion process.

Bayesian Fusion
In the Bayesian model, two statistical estimators generally considered are, the Minimum Mean Square Estimator (MMSE) and the Maximum A posteriori (MAP) estimator. Computation of MMSE requires multidimensional integrations, which is complex for high dimensional data sets. Hence most of theestimators used to fuse HS and MS images use a MAP frame work [3]. Wei et al. [4] proposed a new hierarchical Bayesian model for the fusion of multispectral and hyperspectral images when the spectral response of the multispectral sensor is unknown. The image to be recovered was assumed to be degraded by physical transformations included within a forward model. An appropriate prior distributionwas introducedfor the high spatial and high spectral resolution image to be recovered, defined in a lower dimensional subspace is introduced. The resulting posterior distribution was sampled using a hybrid Gibbs sampler. Numerical experiments showed that the proposed method compares favorably with other state of the art methods, with the advantage of jointly estimating the spectral response of the multispectral sensor. The advantages of the proposed method are (i) it allows the noise variances to be estimated jointly with the image to be recovered and (ii) it can be generalized to more complicated fusion models such as those based on non-Gaussian image priors.

Dempster-Shafer Evidence Theory
Dempster Shafer decision theory is considered a generalized Bayesian theory, used when the datacontributing to the determinationof the analysis of the images is subject to uncertainty. It allows distributing support for proposition not only to a proposition itself but alsoto the union of propositions that include it. Borotschnig et al. [5] compared the three frame works namely, probability theory, possibility theory and Dempster-Shafer theory of evidence for information fusion. The results indicated that Dempster-Shafer decision theory based sensor fusion method will achieve much higher performance improvement, and it provides estimates of imprecision and uncertainty of the information derived from different sources. Wuet al. [6] presented a system frame work that manages information overlap and resolves conflicts, and the system provides realizable architectural support that facilitates sensor fusion. Compared with Bayesian theory, the Dempster-Shafer theory of evidence feels closer to our human perception and reasoning processes. Its capability to assign uncertainty or ignoranceto propositions is a powerful tool for dealing with a large range of problems that otherwise would seem intractable. Hegarat et al. [7] applied the Dempster-Shafer theory of evidence for the fusion of SPOT/HRV image and NOAA/AVHRR series. The results show major improvement in fusion performance.

Multiple Regression Variable Substitution
Multiple regressions derived a variable, as a linear function of multivariable data that will have maximum correlation with univariate data. In image fusion the regression procedure is used to determine a linear combination (replacement vector) of an image channel that can be replaced by another image channel. This method is called regression variable substitution (RVS). To achieve the effect of fusion, the replacement vector should account for a significant amount of variance or information in the original multi variate data set. The method can be applied to spatially enhanced data or for change detection withthe assumption that pixels acquired at time one are a linear function of another set of pixels received at time two. Using the predicted value obtained from the least-square regression, the difference image is the regression value pixel of time one.

IHS Fusion Method
In the IHS color space, intensity (I) is a measure of brightness, with zero representing black, or no brightness, and one representing white, or full brightness. It is also sometimes called luminance (L). Hue (H) is the color, measured as the angle around a color wheel or color hexagon, while saturation (S) is the amount of color, with zero representing grey, or no color, and one representing full color. The main steps, of the standard IHS fusion scheme shown in Figure 4, include conversion from RGB to IHS, which is followed by replacing I component with PAN image and conversion of this image back to RGB.
There are several different formulae for the IHS transformation, demonstrate that the best results can generally be obtained when the intensity is calculated using Equation (7).
One of the drawbacks of the IHS transformation is that it can only be applied to three bands at a time. If there are more than three bands in the MS image, the transformation can be applied to the optimal triplet for each band. IHS methods are capable of quickly merging massive volumes of data. This is one of the most used methodsby many researchers for fusing panchromatic and multispectral images. The IHS pan-sharpening method gives good spatial quality and is a commonly used algorithm for its speed and simplicity. IHS-based methods are used due to their simple computation, high spatial resolution and efficiency. The fused image results in high spatial resolution and low spectral resolution. Many modifications have been proposed to enhance its spectral quality. The IHS fusion technique converts a color image from RGB space to the IHS color space. Here, the intensity band is replaced by the panchromatic image. The final fused IHS image usually has high spatial but low spectral resolution.

Brovey Transform
The Brovey Transform uses ratios to sharpen the MS image [8]. It was created to produce RGB images, and therefore only three bands at a time can be merged. Many researchers used the BT to fuse a RGB image with a high resolutionPan chromatic image [9]. The Brovey methodfirst multiplies each MS band by the high resolution PAN band, and then divides each product by the sum of the MS bands. Equation (8)  Alparone et al. [10], proposed a fast pan sharpeningmethod which is a modified version of Brovey method. Here optimization of the intensity component was achieved through multivariate regression of Pan to MS, and the adjustment of the modulus of the spatial detail vector to be injected was based on a minimization of spatial distortion. The results are competitive with those of the most advanced methods, with a computational complexity comparable with that of Brovey transform fusion, which is the baseline version of the above method.

Gram-Schmidt (GS) Method
The GS orthogonalization procedure is the basis for defining a powerful pan sharpening method. Since GS is a generalization of PCA, in which PC1 may be arbitrarily chosen and the remaining components are calculated to be orthogonal/uncorrelated to one another and to PC1. Like IHS and PCA methods, this method also requires forward and backward transformation of MS image. IHS transformation or the Brovey method work up to three MS bands only. This drawback outwitted in this method, is also offered by standard software packages, e.g., ENVI, ESRI, etc.
The GS spectral sharpening method enhances the spatial resolution of the MS image by merging the high resolution PAN image with the low spatial resolution MS image. The GS fusion simulates the PAN band from the lower spatial resolution spectral bands. In general, this is achieved by averaging the MS bands. As the next step, a GS transformation is performed for the simulated PAN band and the MS bands, where the simulated PAN band is employed as the first band. Then the high spatial resolution PAN band is replaced with the first GS band. Finally, an inverse GS transform is applied to create the pan-sharpened MS bands.

Un Mixing Based Fusion Methods
More recently, Hyperspectral (HS) imaging, which acquires a scene in several hundreds of contiguous spectral bands, has opened a new range of relevant applications such as target detection and spectral un mixing. Hyper spectral (HS) satellite images cover a wider spectral range with moderate to high resolution. However, as compared to MS images, HS images have a better spectral resolution, that may result in a very high number of bands having low spatial resolution. To obtain images with good spectral and spatial resolutions, the remote sensing community has been devoting increasing research efforts to the problem of fusing HS with MS or PAN. The fusion of HS and MS differs from pansharpening, since both spatial and spectral information is contained in multiband images. Due to the low spatial resolution of hyperspectral image, mixed pixels originate in hyper spectral pixe land classification of end members is difficult, which is a major issue and it has been studied extensively. Pixels which contain spectral signature of only one endmember is known as pure pixels and pixels with more than one endmember is known as mixed pixels [11]. Spectral unmixing algorithms are applied to the data for better endmember detection. Hyper spectral unmixing is the decomposition of the pixel spectra into a collection of constituent spectra, or spectral signatures or endmembers and their corresponding fractional abundances that indicate the proportion of each endmember present in the pixel. Depending on the mixing scales at each pixel, the observed mixture is either linear or nonlinear. The linear mixing model holds when the mixing scale is macroscopic. The nonlinear model holds when the mixing scale is microscopic. The linear model assumes negligible interaction among distinct endmembers. The nonlinear model assumes that incident solar radiation is scattered by the scene through multiple bounces involving several endmember.
Linear spectral unmixing has been intensively researched in the recent years [11,12]. Under this model, and assuming that the number of substances and their reflectance spectra are known, hyperspectral unmixing is a linear problem to which many solutions have been proposed (e.g., maximum likelihood estimation, spectral signature matching [13], spectral angle mapper, subspace projection methods [14] and constrained least squares.

Fusion in Frequency Domain
The Ehlers fusion [15] is based on an IHS transform coupled with a Fourier domain filtering. This technique is extended to include more than 3 bands by using multiple IHS transforms until the number of bands is exhausted. A subsequent Fourier transform of the intensity component and the panchromatic image allows an adaptive filter design in the frequency domain. Using Fast Fourier transform (FFT) techniques, the spatial components to be enhanced or suppressed can be directly accessed. The intensity spectrum is filtered with a Low Pass Filter (LPF) whereas the panchromatic spectrum is filtered with an inverse High Pass Filter (HPF). After filtering, the images are transformed back into the spatial domain with an inverse FFT and added together to form a fused intensity component with the low frequency information from the low resolution multispectral image and the high frequency information from the TerraSAR-X image. This new intensity component and the original hue and saturation components of the multispectral image form a new IHS image. As the last step, an inverse IHS transformation produces a fused RGB image. These steps can be repeated with successive 3 band selections until all bands are fused with the panchromatic image.

Multiresolution Based Approach
Multiresolution based image fusion methods decompose the input image into multiple resolution levels using different decomposition methods. Then they combine the decomposition images of specific levels using fusion rules to obtain the fused output. Recently, the wavelet transform approach has been used for fusing data. Wavelet based approaches show some favorable properties compared to the Fourier transform. Wavelet representation provides good localization in both frequency and space domains. Wavelet based fusion approaches are discussed in the following sub sections

Wavelet Based Fusion Technique
In the wavelet based fusion schemes, detail information is extracted from the PAN image using wavelet transforms and injected into the MS image. Distortion of the spectral information is minimized. The wavelet based approach is appropriate for performing fusion tasks for the following reasons: 1. It is a multiscale (multiresolution) approach well suited to manage the different image resolutions. 2. The Discrete Wavelet Transform (DWT) allows the preservation of the image information under different image decomposition levels. 3. Wavelet coefficients coming from different images can be appropriately combined to obtain new coefficients, so that the information in the original images is collectedappropriately. 4. Once the coefficients are merged, the final fused image is achieved through the inverse discrete wavelets transform (IDWT), where the information in the mergedcoefficients is also preserved. In earlier studies wavelet based schemes were generally assessed in comparison to standard schemes; more recent studies propose hybrid schemes, which used wavelets to extract the detail information from one image and standard image transformations to inject it into another image, or propose improvements in the method of injecting information. These approaches seem to achieve better results than either the standard image fusion schemes (e.g. IHS, PCA) or standard wavelet-based image fusion schemes (substitution, addition). However, they involve greater computational complexity.
The key step in image fusion based on wavelets is that of coefficient combination, namely, the process of merging the coefficients in an appropriate way in order to obtain the best quality in the fused image. This can be achieved by a set ofstrategies. The most simple is to take the average of the coefficients to be merged, but there are other merging strategies with better performances, which are discussed in the following sections.
An improved wavelet based image fusion method called window spectral response method, takes into account the physical electromagnetic spectrum responses of sensors during the fusion process, which produces images closer to the image obtained by the ideal sensor than those obtained by usual waveletbased image fusion methods. Usual image fusion methods inject features from a high spatial resolution panchromatic sensor into every low spatial resolution multispectral band trying to preserve spectral signatures and improve spatial resolution to that of the panchromatic sensor. This method produces some undesirable effects, such as resolution overinjection images and slightly modified spectral signatures in some features. The above said undesirable effects are eliminated in the improved wavelet based method.
Yong Yang [16]used a variance based scheme for the selection of the high frequency coefficients and an edge based scheme for the selection of the low frequency coefficients.

Dual Tree Complex Wavelet Transform Based Fusion
A major problem with the DWT is its shift variant nature caused by sub sampling which occurs at each level. The dual-tree complex wavelet transform (DT-CWT) is an over complete wavelet transform that providesboth good shift invariance and directional selectivityover the DWT, although there is an increased memory andcomputational cost. Two fully decimated trees are produced, one for the odd samples and one for the even samples generated at the first level. DT CWT able to distinguishbetween positive and negative orientations giving sixdistinct sub-bands at each level, the orientations of whichare ±15 degrees, ±45 degrees, ±75 degrees. The DT-CWT gives perfect reconstructionas the filters are chosen from a perfect reconstruction bi-orthogonal set. It is applied to images by separablecomplex filtering in two dimensions. Ejaily et al. [17] proposed a DT CWT based fusion scheme to fuse Quickbird and Worldviewsatellite data, and obtained better results than DWT based fusion scheme.

Contourlet Transform Based Fusion
Do & Vetterli [18] proposed a true two-dimensional transform called the Contourlettransform. By virtue of the Laplacian pyramids (LP) and Directional Filter Banks (DFB), the CT provides the multi resolution decomposition and directional decomposition, respectively. The contourlet name is so called because of its ability to capture and link the point of discontinuities to form a linear structure (contours). The CT can capture the intrinsic geometric structure information of images and achieves better expression than Discrete Wavelet Transform (DWT), especially for edges and contours. The two stage process used to derive the Contourlet coefficients involves a multi scale transform and a local directional transform. The point of discontinuities and multiscale transformation is obtained via the Laplacian pyramid. The local directional filter bank is used to group these waveletlike coefficients to obtain a smooth contour. Contourlets provide 2L directions at each scale, where L is the number of required orientation. This flexibility of having different numbers of direction at each scale makes Contourletsdifferent from other available multi scale and directional image representation. Atunable Q Contourlet transform for multi sensor texture-image fusion was proposed by Wang et al.,. The tunable-Q CT is anti-aliasing, and its basis is sharply localized in the desired area of the frequency domain. The tunable-Q CT preserves spectral information at the same time extracts texture information.
Choi et al. [19] proposed a fusion scheme based on a new pyramidal structure, for the multiscale decomposition instead of the commonly used Laplacian pyramid, in the contour let basedfusion of panchromatic and multispectral image. In this method the low frequency coefficients are weightaveraged. The correlation between a pixel and its neighboring pixels is often larger thanothers. Thefusion scheme computesthe region energy of the center pixel and its neighboring pixels. The region energy is larger when the image features are salient. Hence, the region energy for each pixel is compared and high frequency coefficients of the pixel with larger region energy areselected to be used as high frequency coefficients of the fused image. This way, the salient imagefeatures, such as edge and texture information are preserved. They also observed that decomposition level three produced better results than the other levels.

Non Sub Sampled Contourlet Transform Based Fusion
The NSCT with fast implementation provides a complete shift invariant and multiscale representation, similar to the redundant wavelet transform. Hui & Cheng (2008) proposed a novel adaptive strategy for the fusion of panchromatic high resolution image and multispectral image in non sub sampled contourlet (NSCT) domain.. An adaptive intensity component addition method based on the IHS transform is introduced into NSCT domain to preserve spatial resolutions and color content simultaneously. Experiments show that the proposed method can improve spatial resolution and keep the spectral information, simultaneously. There are improvements both in visual effects and quantitative analysis, which is compared with that of the traditional PCA method, IHS transform technique, wavelettransform weighted fusion method, the corresponding wavelet transform based fusion method and the contourlet transform based fusion methods, respectively.
Wang et al. fused the Panchromatic (PAN) and low spatial resolution Multispectral (MS) images to get high spatial resolution of the latter. In this technique, PCA is applied to the MS image to obtain the Principal Component (PC) images. Then the PAN and each PC images are decomposed using NSCT. Fourth Order Correlation Coefficient is used as criterion to select PC. The, relative entropy is then used as a criterion to reconstruct high frequency detailed images. Finally, inverse NSCT is applied to selected PC's low frequency approximation image and reconstructed high frequency\detailed images to obtain high spatial resolution MS image. The Non subsampled contourlet helps to retain the intrinsic structural information while decomposing and reconstructing the image components. Egfin et al. [20] used Fuzzy logic to compute weights based on activity level measurement for fusing the low pass NSCT Coefficients. Mangalraj et al. [21] proposed a NSCT based method for fusing multi sensor satellite images.

Hybrid Fusion Methods
High correlation among the neighboring pixels both spatially and spectrally in a multispectral image makes it necessary to use an efficient data transformation approach beforeperforming pansharpening. Wavelets and Principal ComponentAnalysis methods have been a popular choice for spatial and spectral transformations, respectively. A PCA\Wavelet model based fusionmethod for fusing multispectral and hyperspectralImages was proposed by Palsson et al. [22]. Zhang et al. [23] proposed a method based on the MAP estimation of the UDWT coefficients for the PCs.
The proposed approach extends the Wavelet based Bayesian fusion method by employing a PCA transforms of the observation model. In the proposed method, the first r PCs are estimated in the presence of noiseand it has the option of simply discarding the remaining PCs. This is a much more effective strategy of dealing with the up scaled noise. The proposed method is, therefore, able to handle much larger data sets. The benefits of the approach are substantially lower computational requirements and very high tolerance to noise.
An efficient pansharpening method via a CombinedAdaptive PCA Approach and Contourlets was proposed by Shah et al. [24]. Here the adaptive PCA is used to reducethe spectral distortion and the use of nonsubsampledcontourletsfor spatial transformation in pansharpening is incorporated toovercome the limitation of the wavelets in representing the directional information efficiently and capturing intrinsic geometricalstructures of the objects. The efficiency of the presented method is tested by performing pan sharpening of the high resolution (IKONOS and QuickBird) and the medium resolution (Landsat-7 Enhanced Thematic Mapper Plus) datasets. The evaluation of the pansharpened images using global validation indexes reveal thatthe adaptive PCA approach helps reducing the spectral distortionand its merger with contourlets provides better fusion results.
An efficient pan sharpening technique based on merging two fusion approaches Local variation + Contourlet Transform and APCA + Contourlet Transformwas proposed by Akula et al. [25], and is found to give better results than PCA, APCA, PCA+CT, PCA+WT and APCA+WT. The final pan sharpened image is found to retain the natural color of the multispectral input image producing minimum spectral distortion and the spatial details of the panchromatic image as proved with the performance measure ERGAS. The hybrid schemes show improvement over the standard schemes, and the basic wavelet schemes.

Discussion and Conclusion
From the above survey it becomes clear that all the works that were carried out in Multi Sensor Image fusion are aimed at achieving, a better spatial resolution at the same time preserving spectral information with minimal distortion. Many satellites carry both high resolution multispectral or Panchromatic imaging sensor and low resolution hyper spectral sensor. With the high spatial resolution panchromatic image and multispectral image, detailed geometricfeatures can easily be recognized, while the hyperspectral images contain rich spectral information. Thecapabilities of the images can be enhanced iftheadvantages of both high spatial and spectral resolutioncan be integrated into one single image. The detailedfeatures of such an integrated image can thus be easily recognized and will benefit many applications, such asurban and environmental studies.
Many solutions have been proposed in the literature to solve the Pan sharpening problem. Among these methods, Multi Resolution Analysis (MRA) methods split the spatial information of an image into band pass channels using various frequency extraction techniques, and the high-frequency channels of a panchromatic image are injected to the multispectral bands of the corresponding channels [26]. MRA is known to cause minimal distortion in the spectral information of fusion results, although it is sensitive to spatial misalignments between multispectral and panchromatic images. After the emergence of hyper spectral images, existing multispectral image fusion methods using wavelets, Gram-Schmidt (GS) and Principal Component Analysis (PCA) were applied to hyper spectral and panchromatic images. The fusion of HS and MS differs from pan sharpening since both spatial and spectral information is contained in multiband images [27]. Therefore a lot of pan sharpening methods, such as component substitution and relative spectral contribution [22], are inapplicable or in efficient for the fusion of HS and MS images. Since the fusion problem is generally ill posed. Stochastic or statistical algorithms were also proposed for the fusion of hyper spectral and multispectral images. A Maximum A Posteriori (MAP) estimation method using a stochastic mixing model was developed to estimate the spectral characteristics and to optimize the fused results of the hyper spectral and multispectral images. Standard data fusion methods may not be satisfactory to merge a high resolution panchromatic image and a low resolution Hyper spectral image because they distort the spectral characteristics of the original hyper spectral data. The Fusion of Hyper spectral and Multi spectral images differ from the traditional pan sharpening methods since both of them are multiband images containing both spatial and spectral information. Hence the existing pan sharpening methods cannot be used for the fusion of Hyper Spectral and Multi Spectral images. Jia and Qian [28] used, the two inherent characteristics of hyperspectral data, piecewise smoothness (both temporal and spatial) of spectral data and sparseness of abundance fraction of every material, as constraints in the NNMF algorithm. The adaptive potential function from discontinuity adaptive Markov random field model is used to describe the smoothness constraint while preserving discontinuities in spectral data. A gradient-based optimization algorithm is used, and the monotonic convergence of the algorithm is proved. An unmixing based fusion frame work called coupled non-negative matrix factorization technique was proposed by Yokoya et al. [29] to fuse hyper spectral and multi-spectral images. In this method the hyper spectral and multispectral images are iteratively unmixed to improve the quality of the end member and abundance map. But, Coupled Non Negative Matrix Factorization unmixing based fusion offers very low performance in terms of spectral preservation. This algorithm is straight forward and easy to implement owing to its simple update rules. Cheng bo et al., proposed a numerical procedure to compute directly the unmixed abundance fractions of given endmembers, completely bypassing high-complexity tasks involving the hyper spectral data cube itself. They proposed a reconstruction model which minimizes the total variation of the abundance fractions subject to a preprocessed fidelity equation with a significantly reduced size and other side constraints. Dong et al., [30] proposed a Hyperspectral super-resolution technique for resolution improvement of HS images by jointly unmixing the two input images into the pure reflectance spectra of the observed materials and the associated mixing coefficients. The formulation leads to a coupled matrix factorisation problem, with a number of useful constraints imposed by elementary physical properties of spectral mixing. Hyperspectral unmixing based on a Robust Collaborative Sparse Regression method was proposed. Nonlinear unmixing models were used by Chen et al. [31]. A generalized Bi linear model based Unmixing, using semi non negative matrix factorization was proposed by Yokoya et al. [32]. These techniques can able to handle spectrally complex Hyperspectral images. The price to pay for handling nonlinear interactions induced by multiple scattering effects or intimate mixtures is the computational complexity and a possible degradation of unmixing performance when processing large hyperspectral images.
Recently many Deep Learning models were proposed for Hyper Spectral Image Fusion. Renwei Dian et al., [33] proposed a Deep learning based model for Hyper spectral image sharpening. Shutao Li et al., [34] proposed a coupled sparse tensor factorization based fusion scheme. Burak et al., [35], used an adaptive likelihood map based fusion technique.