Joint Cross-Modality Super Resolution

Guy Shacht Sharon Fogel Dov Danon Daniel Cohen-Or Ilya Leizerson

Tel-Aviv University Elbit Systems

Bi-cubic CMSR RGB Input Zoom-in

Figure 1: Super-resolution results of our approach on the NIR (Near-infrared) modality from the EPFL [1] dataset. Ourmethod, CMSR, exploits fine details from an associated RGB input image for enhancing the resolution of the NIR inputimage (naively upscaled on the left side). CMSR uses the input image pair only, to (internally) train its network and generateits final result without any pretraining. In the right, zoomed-in patches are given, for the comparison of the naively upscaledinput and CMSR’s output.

Abstract

Non-visual imaging sensors are widely used in the indus-try for different purposes. Those sensors are more expen-sive than visual (RGB) sensors, and usually produce imageswith lower resolution. To this end, Cross-Modality Super-Resolution methods were introduced, where an RGB imageof a high-resolution assists in increasing the resolution of alow-resolution modality. However, fusing images from dif-ferent modalities is not a trivial task; the output must beartifact-free and remain loyal to the characteristics of thetarget modality. Moreover, the input images are never per-fectly aligned, which results in further artifacts during thefusion process.

We present CMSR, a deep network for Cross-ModalitySuper-Resolution. The network is trained on the two in-put images only, learns their internal statistics and corre-lations, and applies them to up-sample the target modality.CMSR contains an internal transformer that is trained on-the-fly together with the up-sampling process itself, withoutexplicit supervision, to allow dealing with weakly alignedimages. We show that CMSR succeeds to increase the reso-

lution of the input image, gaining valuable information fromits RGB counterpart, yet in a conservative way, without in-troducing artifacts or irrelevant details that originate fromthe RBG image only.

1. Introduction

Super-Resolution (SR) methods are used to increase thespatial resolution and improve the level-of-detail of digitalimages, while preserving the image content. Such methodshave important applications for multiple industries, such ashealth-care, agriculture, defense and film [28]. In recentyears, more advanced methods of SR have been heavilybased on Deep Learning [11, 23, 5] where one learns themapping between Low-Resolution (LR) images and theirHigh-Resolution (HR) counterparts, and applies the learnedmapping to an unseen low-resolution input.

The need for super-resolution becomes even moreprominent when dealing with sensors that capture othermodalities, different than the visible light spectrum, sincethose sensors typically produce images with lower resolu-

1

arX

iv:2

004.

0996

5v2

[cs

.CV

] 1

6 Ju

l 202

0

tion [20, 26]. For example, Infra-Red (IR) camera sensorsare more expensive than classical camera sensors, and theiroutput images commonly have much lower spatial resolu-tion. While the aforementioned SR methods can still workon such images, there is still a big gap between their re-sulting outputs level-of-detail, and the one found in com-mon RGB images. To bridge that gap, Joint Cross-Modalitymethods were developed. The idea is to use the higher-resolution RGB modality to guide the process of super-resolution on images taken by the lower resolution sensor,taking advantage of the finer details found in the RGB im-ages. The challenge is to remain loyal to the target modal-ity characteristics and to avoid adding redundant artifacts ortextures that may be present only in the RGB modality [2].

In this work, learning is performed internally, relyingsolely on the input pair of images. This approach doesnot require any training data, and therefore avoids the needfor a modal-specific dataset, relying solely on the inter-nal image-specific statistics instead [31]. Using an internalsuper-resolution method is particularly strong in the con-text of cross-modality, since it allows the network to fit tothe unique properties and the modality characteristics of thespecific input pair. This feature stands in contrast to thesomewhat impractical task of generalizing to a large cross-modal image dataset. Moreover, it allows working on un-seen modalities using a single architecture. Therefore, itis more suitable for cases where external modality imagedatasets are hard to obtain, or overly varied, making super-vision practically impossible.

State-of-the-art Joint Cross-Modality SR methods relyon the assumption that their multiple inputs are well aligned[2, 3, 38, 7, 29]. Thus, they perform well only when theinput images were captured by different sensors placed inthe exact same position, and taken at the exact same time.In real-life scenarios, perfect alignment of multiple sensorsis often hard to achieve. Aligning the images in a pre-process typically yields only a weak alignment, dimmingthe effectiveness of joint cross-modality methods. In ourwork, we present new means to allow the two modalitiesto be moderately misaligned, namely Weakly Aligned. Ournetwork contains a learnable deformation component thatimplicitly aligns details in the two images together. Morespecifically, our architecture includes a deformation modelthat aligns details from the RGB image to the target modal-ity in a coarse-to-fine manner, before they are fused to-gether. The network does not use any explicit supervisionfor the deformation sub-task, but rather optimizes the de-formation parameters to adhere to the super-resolution goal.Furthermore, since most multi-modal pairs are not perfectlyaligned, we are able to improve results even on supposedlywell-aligned datasets, compared to previous methods (seeSection 4.1).

A notable benefit of our approach is its ability to avoidover-transferal of information. Previous approaches per-forming multi-modal fusion often suffer from this problem,and tend to add redundant details from the higher resolutionmodality to the lower resolution modality such as textures[2, 3]. Our method is designed to transfer details from thehigher resolution image carefully and conservatively, avoid-ing the transfer of redundant details, learning only the de-tails which aid improving the super-resolution task. Fig-ure 2 presents an example with cross modality ambiguity.Namely, the RGB modality contains an object which doesnot exist in the target modality; this object should ideally beignored in the super-resolution process. Our network suc-cessfully avoids transferring it, whereas a competing cross-modality method results in unwanted artifacts and texturesin its place.

We show that our network achieves state-of-the-art re-sults, while being generic in supporting any modality as in-put, requiring no training data and adjusting to any imagesize.

2. Related WorksSuper-Resolution has been extensively studied through-

out the last two decades. See [28] for a survey covering var-ious SR techniques. Recent surveys [37, 5] cover more ad-vanced methods, including Deep-Learning based methods.The first notable deep network-based method of SR methodis SRCNN [11], a simple fully convolutional method thatshowed superior results to traditional methods. Like mostmethods, SRCNN uses external image datasets, like T91,Set5 and Set14 [22, 23] for training and evaluation.

However, it was claimed [16, 39, 31] that methods whichrely on large external datasets do not learn the internalimage-specific properties of the given input. In [16, 39], thesubject of internal patch recurrence is investigated, leadingto quantifiable results suggesting that patches of differentscales tend to recur in the same image more than in ex-ternal image datasets. This strong observation gave rise topowerful Zero-Shot methods [15, 31, 8], and most notablyZSSR [31], which applies random cropping to its input im-age, effectively creating an internal image-specific datasetof patches taken solely from a single input. The method wepresent builds upon these ideas to deal with cross-modality,enjoying the same advantages of being dataset independentand relying solely on the internal statistics of both inputimages.

2.1. Joint Cross-Modality

In the Joint Cross-Modality setting the two differentmodalities are jointly analyzed to enhance one of them.As mentioned earlier, camera sensors capturing the RGBmodality produce images with richer HR details than othermodalities. Thus, a common setting is the usage of a visual

2

Figure 2: To demonstrate the ability of CMSR to ignore details that appear only in the RGB image and are irrelevant tothe target NIR (Near-infrared) modality, we added a standing man in the RGB modality. As can be clearly observed in theclose-up views, CMSR does not add ghosts, while VTSRGAN does. The quantitative measures (PSNR, SSIM) show thatCMSR also surpasses the baseline single-modality method, ZSSR, which only operates on the NIR input, without utilisingthe RGB input.

HR version of the image, alongside with a LR version takenby the other modality sensor. This setting was adopted byall relevant joint cross-modality methods.

Visual-Depth In [29], a learning-based visual-depthmethod is presented. It is based on a CNN architectureoperating on a LR depth-map and a sharp edge-map ex-tracted from the HR visual modality. The network is trainedon visual-depth aligned pairs from the Middlebury dataset[30]. In [38], a GAN-based method (CDcGAN) is pre-sented. The method adds auxiliary losses that encouragekeeping the resulting depth-maps smooth and texture-free,and is also trained on the Middlebury dataset.

Visual-Thermal (Infrared) In [7], a non learning-basedjoint visual-thermal method is presented. It is based onguided filtering of an up-sampled LR thermal input in ar-eas that correlate well with the HR visual input. It is testedon visual-thermal pairs whose capturing sensors were man-ually calibrated to be aligned. Almastri et al. [2] intro-duced the learning-based visual-thermal SR methods VT-SRCNN and VTSRGAN, built on top of the existing SR-CNN and SRGAN. They perform joint visual-thermal SRby concatenating feature maps extracted from each inputmodality, and are trained and evaluated on the ULB17-VT[4] visual-thermal dataset consisting of well aligned pairs.

Cross-Modal Misalignment As noted by Almasri et al.[2], in the context of cross-modal super-resolution, mis-alignment is a major limitation in producing artifact-freeSR results. In their paper, it is claimed that the artifactsadded to the SR result appear where there is cross-modaldisplacements, and a better synchronized capturing devicewould likely solve that problem. Our method’s approach inhandling cross-modal misalignment is to deform the RGBmodality and align details that improve the SR objective tothe target modality.

Our Method Our method differs from the aforemen-tioned joint cross-modality techniques in two central as-pects. First, it does not require any training data, and there-fore avoids the need for a modal-specific dataset, relying onthe internal image-specific statistics instead. This featureis especially attractive for cross-modality super-resolution;learning from the single input pair encourages the networkto adapt to the specific cross-modal properties existing inthat particular pair, which may be unique. Second, it re-quires only weak alignment, as opposed to the aforemen-tioned techniques which rely on well aligned pairs.

2.2. Image Registration

The subject of multi-modal image registration has beenstudied mainly in the context of medical imaging. Deepmethods [32, 10, 9] have mostly based their architectureson a regressor, a spatial transformer and a re-sampler. Theyuse supervision to optimize their regression and deforma-tion models. It is also possible to use similarity metrics(like cross-correlation) [9] instead of training a regressorwith supervision, and obtain an unsupervised registrationframework.

In our work, multi-modal image registration is not per-formed per se. Our goal is not to register the two inputmodalities together, but to give the network enough free-dom to align only the details that assist and adhere to thesuper-resolution task. The alignment phase is integratedinto the main SR task. We use the same SR reconstruc-tion loss to optimize our deformation parameters. Thus, wedo not require aligned pairs for training. The deformationframework used in our method consists of three steps per-formed in a coarse-to-fine manner [9, 12]. We first trans-form our image using global affine transformation for aninitial rough approximation. Then, we further align our twomodalities using CPAB [13, 33] transformation, which actsin a piecewise yet continuous manner. Finally, we use thin-plate spline (TPS) transformation [33] for the final refine-ment of our alignment task.

3

3. Cross-Modality Super Resolution

One of the fundamental problems of cross-modality su-per resolution is that it is hard to transfer only the relevantdetails from the higher resolution image to the lower res-olution one, while ignoring details which are not supposeto be visible in the lower-resolution modality. Those irrele-vant details often cause ghosts and unwanted artifacts (suchas those in Figure 2 and Figure 9). When training on a largedataset of cross-modality image pairs, it is hard (and of-ten impossible) for a network to learn which details exactlyshould be transferred and which should not be, for eachgiven cross-modality pair. This is mostly because similarobjects might be present (and thus, should be transferred)in some pairs in the dataset, and not in others. For thisreason, previous super-resolution cross-modality methods,which are trained on large datasets, often add redundant de-tails to the output image (see the aforementioned Figure 2and Figure 9). To avoid this problem, we opted to use a su-per resolution method trained on a single input pair. Fittingto a specific pair enables the network to learn the internalstatistics specific to this pair and allows it to avoid transfer-ring objects which exist only in the high resolution image.In Figure 2, we evaluate our network (trained on the singleinput pair) compared to a dataset based method, and showthat our network, which is only trained internally, avoidstransferring redundant, unwanted object existing only in thehigh-resolution image, whereas the dataset based methodproduces ghosts and artifacts caused by that redundant ob-ject.

3.1. Network Architecture

Our network builds upon the ZSSR network introducedby Shocher et al. [31]. it includes a patch selection compo-nent which generates a training set out of a single pair of im-ages, and a super-resolution network. Our method enablesdealing with misaligned pairs by including a deformationphase, done internally, which aligns objects in both imagesright before they enter the SR network (see Figures 3 and4).

We hereby introduce and describe the components of ournetwork, which are incorporated into our training and infer-ence schemes as covered in Sections 3.2 and 3.3.

Alignment using Learnable Deformation Our networkcorrects displacements between the two modalities on-the-fly, through a local deformation process applied to the RGBmodality as a first gate to the network, optimized implicitlyduring training. To that end, instead of using explicit su-pervision to optimize the deformation parameters, they aretrained with the super-resolution loss and therefore deformonly parts which are relevant to this task. Hence, the goalof the deformation step is not to form a perfect alignment

between the images, but rather to allow partial alignmentto boost the super-resolution task, where needed. Our de-formation process consists of three different transformationlayers, performing the learned alignment in a coarse-to-finemanner.

The first layer of our deformation framework is the orig-inal Affine STN layer by Jaderberg et al. [18]. It captures aglobal affine transformation that is used to position the twomodalities together as a rough initial approximation.

The second layer is a DDTN transformation layer (DeepDiffeomorphic Transformation Network, [33]), a variant ofthe original STN layer supporting more flexible and expres-sive transformations. Our chosen transformation model isCPAB (Continuous Piecewise-Affine Based, [13, 33]). Itis based on the integration of Continuous Piecewise-Affine(CPA) velocity fields, and yields a transformation that isboth differentiable and has a differentiable inverse. It isContinuous Piecewise-Affine w.r.t a tessellation of the im-age into cells. For this reason, it is well suited to our align-ment task; each cell can be deformed differently, yet con-tinuity is preserved between neighboring cells, yielding adeformation that can express local (per-cell) misalignmentswhile preserving the image semantics.

The third and last layer of our deformation frameworkperforms a TPS (Thin-plate spline) transformation, a tech-nique that is widely used in computer vision and particularlyin image registration tasks [6]. Our implementation (alsotaken from [33]) learns the displacements of uniformly-distributed keypoints in an arbitrary way, while each key-point’s surrounding pixels are displaced in accordance toit, using interpolation [6]. Since TPS displaces its key-points freely, the displacement is unconstrained to any im-age transformation model, and has the power to align thefine-grained objects of the scene, providing the final refine-ment of our alignment task.

Patch Selection Similarly to ZSSR [31] we produce ourtraining set from a single pair of images by samplingpatches using random augmentations. In our implemen-tation we use scale, rotation, shear and translations. Thisrandom patch selection yields two patches that correspondto roughly the same area in the scene: one taken from thetarget modality and the second is taken from the deformedRGB modality which was previously aligned to the targetmodality.

CMSR network The CMSR network is the main compo-nent of our architecture as it is the component responsiblefor performing super-resolution. Namely, it produces a HRversion of its target modality LR input image, guided by itsHR RGB input. As Figure 3 and Figure 4 suggest, this com-ponent can be applied to varying image sizes, thanks to itsfully convolutional nature.

4

Figure 3: Training process. The RGB image first goes through a deformation step which aligns it to the target modality (inblue). Then, random patches are selected by an augmentation step (in Red) and down-sampled (in green). The patches areused to train the CMSR network (in orange) and the deformation parameters. The loss function is measured between thesuper-resolved output and the input target modality images.

Figure 4: Inference. During inference, the learned defor-mation parameters and the CMSR component are used toup-sample the original LR modality input image, guided bythe HR RGB input image.

The fully convolutional architecture of CMSR is basedon the one from Shocher et al. [31]. However, a fewchanges have been made to better apply it to cross-modalitySR (see Figure 5). The first gate to the network is up-sampling of the LR modality input to the size of the RGBinput. This is done naively, using the Bi-cubic method, incase no specific kernels are given. 1 From the up-sampledmodality input we generate a feature map using a numberof convolutional layers, denoted as Feature-Extractor 1 inFigure 5. From the RGB modality input that was previ-

1Optimal blur kernels can be directly estimated as shown in [16], andare fully supported by our method as an additional input to the network.

ously aligned to target modality input, we generate a featuremap using Feature-Extractor 2. We perform summationof the two resulting feature maps, one from each Feature-Extractor block, alongside with an up-sampled version ofthe LR target modality image, in a residual manner. Thisyields our HR super-resolved output.

3.2. Training

During each training iteration, we perform local defor-mation on the RGB modality input and produce a displacedversion of it, aligned to the target modality image, as de-scribed in Section 3.1. Then, a random patch is selectedfrom the input pair (illustrated in Figure 3), yielding twocorresponding patches; one taken from the target modality,and the second from the displaced (aligned) RGB modality,as described in 3.1. The patch selection phase is an integralpart of the network, and is done in a differentiable man-ner, so as to allow the gradients to backpropagate throughit to the deformation model. This enables us to optimizethe transformation on the entire RGB image despite usingpatches of the image during training.

In order to generate supervision for the training process,we down-sample the two patches and use the original targetmodality patch as ground-truth. We use L1 reconstructionloss between the reconstructed patch and original input tar-get modality patch. Note that there is no ground truth for aperfectly aligned RGB modality. Instead, the deformationparameters are optimized using the same L1 reconstructionloss as an integral part of the SR task.

Alternating Scales As mentioned above, after the PatchSelection (3.1) step of our training scheme, we down-sample both patches (Figure 3, in Green) by our desired

5

Figure 5: CMSR operates in a straightforward manner;it performs three-way summation. Two of the resultingfeature maps, one from each modality, are summed to-gether, element-wise, with the original modality input thatis naively up-sampled, in a residual manner.

SR ratio (e.g., 2x, 4x), denoted as r . The modality patchis down-sampled to allow training the network to recon-struct it with self-supervision, whereas the RGB patch isdown-sampled accordingly, to keep the ratio between thetwo patches equal to r .

Instead of down-sampling the RGB patch, it is also pos-sible to naively up-sample the modality patch, and still pre-serve the same ratio, r , between patches. We found thatby alternating between up-sampling and down-sampling ofthe aforementioned patches, we are able to significantly im-prove the results. More details regarding this technique canbe found in the supplementary material.

3.3. Inference

At inference time, we use the trained CMSR network anddeformation parameters, to perform SR on the entire targetmodality image guided by the RGB modality image (seeFigure 4).

Since CMSR is fully convolutional, it can operate on anyimage size (e.g., both image patches of different scales, andfull images) using the same network. We first apply thealignment dictated by the optimized deformation parame-ters, and then feed the LR target modality image and thealigned HR RGB image to the SR network which outputs aHR version of the target modality image.

After the HR target modality image is obtained, we per-form two additional refinement operators aimed to furtherimprove our SR results. The first operator, Geometric Self-Ensemble, is an averaging technique shown to improve SR

Figure 6: The visual-depth pairs from the Middleburydataset (top row) and the visual-thermal pairs from theULB17-VT dataset (bottom row) show strong multi-modalregistration. Under less than optimal imaging conditions,such alignment is hard to achieve.

results [24, 34, 31]. The second operator, Iterative Back-Projection, is an error-correcting technique that was usedsuccessfully in the context of SR [14, 17, 31].

4. Results and Evaluation

Our model is implemented in Tensorflow 1.11.0 andtrained on a single GeForce GTX 1080 Ti GPU. The fullcode and datasets will be published upon acceptance in theproject’s GitHub page. We typically start with a learningrate of 0.0001 and gradually decrease it to 10−6, depend-ing on the slope of our reconstruction error line, whereasthe learning rates of our transformation layers follow thesame pattern, multiplied by constant factors. Those fac-tors are treated as hyper-parameters, and should typically belarger when dealing with highly displaced input pairs, likein the case of weakly aligned modalities (Figure 7). Per-forming a 4x SR on an input of size 60x80 typically takes30 to 60 seconds, depending on the desired number of iter-ations. To achieve SR of higher scales, we perform gradualSR with intermediate scales, as this further improves the re-sults [21, 35, 31].

For Feature-Extractor 1 we use eight hidden layers,each containing 64 channels and a filter size of 3x3. Weplace a ReLU activation function after each layer except forthe last one. The size of feature maps remains the samethroughout all layers in the block. For Feature-Extractor 2we typically use four to eight hidden layers with number ofchannels ranging from 4 to 128, a filter size of 3x3 and aReLU activation function. The last layer has no activationand a filter size of 1x1. We find that highly detailed RGBinputs require Feature-Extractor 2 to have more channels.The hyper-parameters rarely require adjustments; they onlyrequire manual tuning when dealing with inputs that areunique, unusual, or ones that reflect very unusual displace-ments.

6

Figure 7: Two examples of Weakly Aligned modality pairs.To visualize the misalignment, we overlaid them with semi-transparency. Note, the ghosting effect where cross-modalmisalignment occurs.

4.1. Evaluation with State-of-the-arts

Thermal (Infrared). We compared our method tocross-modal state-of-the-art SR methods on visual-thermalpairs. We used the ULB17-VT dataset [4], consisting ofpairs (two examples are shown in Figure 6, bottom row)that are mostly well aligned. We have included the resultsof our evaluation in Table 1, showing that our method, de-spite not being previously trained, beats competing meth-ods, averaged across the ULB17-VT dataset. In Figure 8 avisual result from that evaluation is included.

NIR (Near-infrared). In Figures 10, 9 and 2 we in-clude visual results from our evaluation on the EPFL NIRdataset [1]. The conservative approach of our method en-ables it to surpass state-of-the-art cross-modal methods, de-spite the fact that those competing methods were pre-trainedextensively on the full dataset, whereas our method operateson its single input pair, without pre-training, in a Zero-Shot[31] manner.

Depth. The Middlebury dataset [30] contains stronglyaligned depth-visual pairs as shown in Figure 6 (top row).In that dataset, multiple angles and different sensor place-ments are included, for each pair. To obtain weakly-alignedpairs, we shuffled the pairs together such that the result-ing pairs would correspond to a small sensor misplacement,shown in Figure 7 (left pair). We further increased the sizeof the dataset through random augmentations. We denotethe new resulting dataset as Shuffled-Middlebury. CMSRsurpasses competing cross-modal methods on those weaklyaligned pairs by using a coarse-to-fine alignment approach,as summarized in Table 1.

Single Modality. We evaluated CMSR against thebaseline state-of-the art single modality method, ZSSR[31]. Our experiment shows that our method leverages thefine details in its RGB input and produces a SR output thatis both appealing to the eye, and numerically closer to aGround-Truth version, as shown in Table 1 and in Figures10, 12, 8 and 11.

4.2. Analysis

RGB Artifacts. A fusion of multiple image sources,often causes the transfer of unnecessary artifacts from onemodality to the other (e.g., [2]). Those artifacts not onlysabotage the quality of the image, but harm the modalitycharacteristics and could potentially make it unusable. Ourmethod learns only the relevant RGB information that im-proves SR results; Figures 2, 11 and 9 show cases wherethe RGB modality input contains a great amount of textu-ral information, yet our SR output remains texture-free. InFigure 8, the learned RGB residual is given; it contains noirrelevant textures and it resembles an edge-map, used tosharpen our output image.

Local Deformation. As shown in Figure 15, ourmethod aligns the RGB modality input to the target modal-ity input on-the-fly, to aid the joint cross-modal SR task.Although we have no aligned RGB ground-truth image,nor any target modality ground-truth image, we still correctthose cross-modal misalignment successfully, thanks to anexpressive deformation framework integrated into our ar-chitecture. Our deformation component provides the net-work the ability to align only details that assist and adhereto the super-resolution task, rather than committing to animage-to-image alignment per se. The deformation param-eters are optimized using the SR reconstruction loss, thus,we learn only the deformations that are needed to minimizethat loss and assist in the SR task.

Ablation Study. To show the necessity of each layerof our coarse-to-fine deformation framework, we evaluatedCMSR on a weakly aligned pair, adding one layer at a timeand averaging across multiple runs. The results indicatethat each layer is indeed necessary and plays a different rolein the alignment process as can be seen visually in Figure14, and numerically in Figure 13. Our goal is not to per-form perfect registration between the images, but rather toalign only the necessary details to improve the quality of thehigher resolution output. Hence, we measure the quality ofthe alignment through the generated SR result, and not byconventional image registration metrics.

5. ConclusionsWe have introduced CMSR, a method for cross-modality

super-resolution. Our method utilises an associated high-resolution RGB image of the scene to boost its accuracy.The method presented is generic and yet outperforms state-of-the-art methods, even when its two modalities are mis-aligned, as elaborated below.

Generic. To the best of our knowledge, CMSR is thefirst self-supervised cross-modal SR method. It requiresno training data, a prominent advantage when dealing with

7

Figure 8: We compare our method to its baseline method, ZSSR [31], as well as to another cross-modality method, VTSR-GAN [2]. On the right, the output of Feature-Extractor 2 (Figure 5) is given as the learned RGB residual which is added toour output. This RGB residual is artifact-free, contains no unwanted textures, and in fact, resembles an edge-map. For thisreason, CMSR produces images that are visually pleasing, free of artifacts, and numerically better than competing methods.

Figure 9: CMSR uses its RGB input, conservatively. Wedemonstrate the results of three methods on a NIR patch,with the corresponding RGB image which is rich in tex-tures (the colorful image on the right). We compared CMSR(bottom left) to VTSRCNN (top left) and VTSRGAN (topright). CMSR avoids introducing noticeable redundant arti-facts and textures induced by RGB modality. Ground-Truth(bottom right) is given as reference.

Metric Dataset VTSRGAN VTSRCNN CMSRPSNR U-VT 27.988 27.968 29.928SSIM U-VT 0.8202 0.8196 0.882PSNR SMB 27.925 28.189 28.652SSIM SMB 0.9547 0.9386 0.9341

Table 1: We compared CMSR to competing cross-modalSR methods, VTSRCNN and VTSRGAN [2], on theStrongly Aligned ULB17-VT dataset [4], as well as on theWeakly Aligned Shuffled-Middlebury dataset created by us.We have taken the mean PSNR / SSIM scores, measuredagainst the modality 4x GT versions.

scarce and unique modalities. It is trained on the targetimage only, and can thus, take any modality as input, and

learns its internal, possibly unique, statistics, adapting tothe unknown imaging conditions and down-scaling kernels.

Furthermore, the method can be applied to any imagesizes, and to any ratio between the two inputs. This is unlikeother architectures that use strides for up-sampling [2], thusthey are fixed to a specific image size and constant scalefactor.

Performance. Our method is conservative, in the sensethat it learns from its RGB features only when it contributesto the up-sampling process, without introducing outliers,ghosts, halos, or other artifacts.

We achieve state-of-the-art results, qualitatively (visu-ally) and quantitatively, compared to competing cross-modal methods, as well as to our state-of-the-art single-modality baseline. Specifically, we show that the RGBmodality indeed greatly contributes as a guide to the up-sampling process.

Misalignment A unique property of our method is thatit is robust to cross-modal misalignment. This property isimperative, since in real life conditions, sight misalignmentis, more often than not, unavoidable. It should be empha-sized that the alignment is done without pre-training or anysupervision.

In the future, we would like to further enhance our tech-nique by applying the deformation in the feature space in-stead of the RGB pixel-space. The hope is that in thisway, it would be possible to adopt a deformation-per-featurescheme that would reflect different displacements for differ-ent scene objects, possibly using segmentation.

8

Figure 10: We compared CMSR both to its single-modality baseline, ZSSR [31], and to a competing cross-modality method,VTSRCNN [2], on the NIR modality. Our method, CMSR, is able to produce super-resolved images that are both visuallypleasing, and numerically closer to a Ground-Truth version.

9

Figure 11: CMSR does not introduce irrelevant details.Note that despite the large amount of textural informationin its RGB input (d), CMSR (b) ignores it and learns onlythe relevant information, resulting in a better super-resolveddepth-map than the baseline model, ZSSR (a), in compari-son to a GT version (c).

Figure 12: From left to right, respectively: Bi-cubic, CMSR(our method), Ground-Truth IR, and the RGB input. Here,CMSR succeeded to produce the building’s windows dur-ing the SR process, despite never seeing their Thermal (IR)representation.

Figure 13: We let CMSR perform 4x SR on a WeaklyAligned visual-thermal pair, with different transformationlayers, averaged across 5 runs. The results indicate that eachlayer contributes to the final PSNR, which can also be seenvisually in Figure 14

.

Figure 14: We evaluated CMSR using different transforma-tion layers. In the leftmost column, the resulting deformedRGB image is given. In the other columns we show theresulting alignment, visualized through blending of the R-G (Red-Green) channels of the aforementioned deformedRGB image, together with the Ground-Truth thermal image(which is unavailable to CMSR).

Figure 15: To demonstrate the alignment capabilities ofCMSR, we evaluated it on a severely misaligned visual-thermal pair, (a) and (b), containing both global and lo-cal displacements. We overlaid the images with semi-transparency, once before training (c), and once after train-ing the network (d). CMSR deformed its RGB input andaligned it to the thermal modality in a coarse-to-fine man-ner, on-the-fly, and without supervision.

10

Weakly Aligned Joint Cross-Modality Super Resolution -Supplementary Material

6. Alternating Scales

In Section 3.2 of the submitted paper, the AlternatingScales technique is briefly discussed. It corresponds totraining CMSR using two different scales, alternating be-tween them across iterations. Here, we wish to further elab-orate on this technique.

6.1. Alternating Scales - Elaboration

Denoting our desired SR ratio (e.g. 2x, 4x) by r , our net-work, CMSR, takes a target modality input of size H x Walongside with an RGB input of size rH x rW , and pro-duces a target modality output of size rH x rW . Hence,by design, a ratio of r must be preserved between CMSR’stwo inputs (The architecture of CMSR is given in the origi-nal paper, Figure 6). Since CMSR is trained to reconstructa random patch taken from its modality input (Figure 4 ofthe original paper, Training process), this random patch isdown-sampled, by ratio r , before it is reconstructed by theCMSR network. However, since the ratio between CMSR’stwo inputs must remain r , the corresponding RGB patchis also down-sampled accordingly, by ratio r . This way, wepreserve the same ratio between CMSR’s two input patches,as needed.

Nonetheless, instead of down-sampling the RGB patchto match this required ratio, it is also possible to naivelyup-sample the modality patch by ratio r . Clearly, this hasthe same effect on the ratio between the two patches, whichyet again remains r . However, this way, we obtain a differ-ent training scheme. Figure 16 compares the two differentschemes, corresponding to the two different scales CMSRoperates on.

We found that by alternating between the two schemesduring training, we are able to significantly improve our re-sults. We name this combination of training schemes as theAlternating Scales technique. It allows our network to beoptimized using patches of their original scale, as explainedin Table 2. We observe that training our network on patchesof their original scale improves its generalization capabili-ties, since during the inference stage, the network operateson the full input pair, at its original scale.

6.2. Alternating Scales - Ablation Study

We have conducted an experiment to show the improve-ment obtained by the Alternating Scale technique. Wetrained CMSR using the two schemes (see Figure 16 and Ta-ble 2 for information on the schemes), alternating betweenthem randomly. We used the Upsampling-Based scheme

Training Scheme Modality Scale RGB ScaleDown-sampling Original Down-scaled

Up-sampling Up-scaled Original

Table 2: In the Downsampling-Based training scheme,CMSR takes a down-sampled RGB input patch, but itsmodality input patch is reconstructed at its true, originalscale. However, in the Upsampling-Based scheme, CMSRtakes an original RGB input patch, at its true scale, butreconstructs a modality patch that was up-sampled before-hand.

with probability p and the Downsampling-Based with prob-ability 1 − p.

According to the results, summarized in Figures 17 and18, the best PSNR was obtained when p = 0.3, which startsdecaying when p > 0.3. We notice that p > 0 always yieldsbetter results than p = 0. This observation is important,since the risk of using sub-optimal p values on new, unseeninput pairs is minimal; using this technique is always betterthan not using it, regardless of p.

11

Figure 16: The difference between the two training schemes lies in the scale CMSR (in Orange) operates on. The twoschemes start with the exact same input pair, but in the Upsampling-Based training scheme (right), CMSR is fed inputs oflarger scale. This scale difference is also explained in Table 2.

Figure 17: We evaluated CMSR using different alternationprobabilities. Namely, we trained it using the Upsampling-Based training scheme (Figure 16) in fraction p iterations,and using the Downsampling-Based scheme in the remain-ing fraction 1− p. We averaged this experiment across mul-tiple runs. According to the results, p = 0.3 yields the bestPSNR (32.476 dB). This can be also seen visually, in Figure18

Figure 18: We compare two patches taken from the Alter-nating Scales ablation study results, summarized in Figure17. According to our experiment, the best SR result is ob-tained when using p = 0.3 as the alternation probability.

12

7. Acknowledgements

This research was supported by a generic RD program ofthe Israeli innovation authority, and the Phenomics consor-tium.

References

[1] Rgb-nir scene dataset.

[2] Feras Almasri and Olivier Debeir. Multimodal sensor fu-sion in single thermal image super-resolution. arXiv preprintarXiv:1812.09276, 2018.

[3] Feras Almasri and Olivier Debeir. Rgb guided thermal super-resolution enhancement. In 2018 4th International Confer-ence on Cloud Computing Technologies and Applications(Cloudtech), pages 1–5. IEEE, 2018.

[4] Feras Almasri and Olivier Debeir. ULB17-VT. 2 2019.

[5] Saeed Anwar, Salman Khan, and Nick Barnes. A deep jour-ney into super-resolution: A survey. CoRR, abs/1904.07523,2019.

[6] Fred L. Bookstein. Principal warps: Thin-plate splines andthe decomposition of deformations. IEEE Trans. PatternAnal. Mach. Intell., 11(6):567–585, 1989.

[7] Xiaohui Chen, Guangtao Zhai, Jia Wang, Chunjia Hu, andYuanchun Chen. Color guided thermal image super resolu-tion. In 2016 Visual Communications and Image Processing(VCIP), pages 1–4. IEEE, 2016.

[8] Zhen Cui, Hong Chang, Shiguang Shan, Bineng Zhong,and Xilin Chen. Deep network cascade for image super-resolution. In European Conference on Computer Vision,pages 49–64. Springer, 2014.

[9] Bob D. de Vos, Floris F. Berendsen, Max A. Viergever, Hes-sam Sokooti, Marius Staring, and Ivana Isgum. A deep learn-ing framework for unsupervised affine and deformable imageregistration. CoRR, abs/1809.06130, 2018.

[10] Bob D. de Vos, Floris F. Berendsen, Max A. Viergever, Mar-ius Staring, and Ivana Isgum. End-to-end unsupervised de-

formable image registration with a convolutional neural net-work. CoRR, abs/1704.06065, 2017.

[11] Chao Dong, Chen Change Loy, Kaiming He, and XiaoouTang. Image super-resolution using deep convolutional net-works. CoRR, abs/1501.00092, 2015.

[12] Tobias Fechter and Dimos Baltas. One shot learning fordeformable medical image registration and periodic motiontracking. CoRR, abs/1907.04641, 2019.

[13] O. Freifeld, S. Hauberg, K. Batmanghelich, and J. Fisher III.Transformations based on continuous piecewise-affine ve-locity fields. In IEEE Transactions on Pattern Analysis andMachine Intelligence (TPAMI), 2017.

[14] Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from a single image. In ICCV, 2009.

[15] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Sin-gle image super-resolution from transformed self-exemplars.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), June 2015.

[16] Daniel Glasner Shai Bagon Michal Irani. Super-resolutionfrom a single image. In Proceedings of the IEEE Interna-tional Conference on Computer Vision, Kyoto, Japan, pages349–356, 2009.

[17] Michal Irani and Shmuel Peleg. Improving resolution byimage registration. CVGIP: Graph. Models Image Process.,53(3):231–239, Apr. 1991.

[18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, andKoray Kavukcuoglu. Spatial transformer networks. CoRR,abs/1506.02025, 2015.

[19] Daeyoung Kim and Kuk-jin Yoon. High-quality depth mapup-sampling robust to edge noise of range sensors. In 201219th IEEE International Conference on Image Processing,pages 553–556. IEEE, 2012.

[20] Y Kiran, V Shrinidhi, W Jino Hans, and N Venkateswaran.A single-image super-resolution algorithm for infrared ther-mal images. INTERNATIONAL JOURNAL OF COMPUTERSCIENCE AND NETWORK SECURITY, 17(10):256–261,2017.

[21] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast andaccurate super-resolution. CoRR, abs/1704.03915, 2017.

[22] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Fast and accurate image super-resolution withdeep laplacian pyramid networks. IEEE Transactions on Pat-tern Analysis and Machine Intelligence, 2018.

[23] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero,Andrew P. Aitken, Alykhan Tejani, Johannes Totz, ZehanWang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. CoRR,abs/1609.04802, 2016.

[24] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, andKyoung Mu Lee. Enhanced deep residual networks for singleimage super-resolution. CoRR, abs/1707.02921, 2017.

[25] Hai-cang Liu, Shu-tao Li, and Hai-tao Yin. Infrared surveil-lance image super resolution via group sparse representation.Optics Communications, 289:45–52, 2013.

[26] Emanuele Mandanici, Luca Tavasci, Francesco Corsini, andStefano Gandolfi. A multi-image super-resolution algorithmapplied to thermal imagery. Applied Geomatics, Feb 2019.

13

[27] Yuxing Mao, Yan Wang, Jintao Zhou, and Haiwei Jia. An in-frared image super-resolution reconstruction method basedon compressive sensing. Infrared Physics & Technology,76:735–739, 2016.

[28] Kamal Nasrollahi and Thomas B. Moeslund. Super-resolution: A comprehensive survey. Mach. Vision Appl.,25(6):1423–1468, Aug. 2014.

[29] Min Ni, Jianjun Lei, Runmin Cong, Kaifu Zheng, Bo Peng,and Xiaoting Fan. Color-guided depth map super resolutionusing convolutional neural network. IEEE Access, 5:26666–26672, 2017.

[30] Daniel Scharstein and Richard Szeliski. A taxonomy andevaluation of dense two-frame stereo correspondence algo-rithms. Int. J. Comput. Vision, 47(1-3):7–42, Apr. 2002.

[31] Assaf Shocher, Nadav Cohen, and Michal Irani. ”zero-shot” super-resolution using deep internal learning. CoRR,abs/1712.06087, 2017.

[32] Martin Simonovsky, Benjamın Gutierrez-Becker, Diana Ma-teus, Nassir Navab, and Nikos Komodakis. A deep metricfor multimodal registration. CoRR, abs/1609.05396, 2016.

[33] Nicki Skafte Detlefsen, Oren Freifeld, and Soren Hauberg.Deep diffeomorphic transformer networks. In Proceedingsof the IEEE Conference on Computer Vision and PatternRecognition, pages 4403–4412, 2018.

[34] Radu Timofte, Rasmus Rothe, and Luc Van Gool. Sevenways to improve example-based single image super resolu-tion. CoRR, abs/1511.02228, 2015.

[35] Yifan Wang, Federico Perazzi, Brian McWilliams, Alexan-der Sorkine-Hornung, Olga Sorkine-Hornung, and Christo-pher Schroers. A fully progressive approach to single-imagesuper-resolution. CoRR, abs/1804.02900, 2018.

[36] Jun Xie, Cheng-Chuan Chou, Rogerio Feris, and Ming-TingSun. Single depth image super resolution and denoising viacoupled dictionary learning with local constraints and shockfiltering. In 2014 IEEE International Conference on Multi-media and Expo (ICME), pages 1–6. IEEE, 2014.

[37] Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang,and Jing-Hao Xue. Deep learning for single image super-resolution: A brief review. CoRR, abs/1808.03344, 2018.

[38] Lijun Zhao, Jie Liang, Huihui Bai, Anhong Wang, andYao Zhao. Simultaneously color-depth super-resolutionwith conditional generative adversarial network. CoRR,abs/1708.09105, 2017.

[39] Maria Zontak and Michal Irani. Internal statistics of a singlenatural image. In CVPR 2011, pages 977–984. IEEE, 2011.

14