Neurocomputing 397 (2020) 94–107

Contents lists available at ScienceDirect

Neurocomputing

journal homepage: www.elsevier.com/locate/neucom

A dual-domain deep lattice network for rapid MRI reconstruction

Liyan Sun

a , Yawen Wu

a , Binglin Shu

a , Xinghao Ding

a , ∗, Congbo Cai b , Yue Huang

a , John Paisley

c

a Fujian Key Laboratory of Sensing and Computing for Smart City, Xiamen University, Fujian, China b School of Electronic Science and Engineering, Xiamen University, China c Department of Electrical Engineering, Columbia University, New York, NY, USA

a r t i c l e i n f o

Article history:

Received 17 March 2019

Revised 18 August 2019

Accepted 17 January 2020

Available online 22 January 2020

Communicated by Dr. Shenghua Gao

Keywords:

Compressed sensing

Magnetic resonance imaging

Dual domain

Deep neural network

a b s t r a c t

Compressed sensing is utilized with the aims of reconstructing an MRI using a fraction of measurements

to accelerate magnetic resonance imaging called compressed sensing magnetic resonance imaging (CS-

MRI). Conventional optimization-based CS-MRI methods use random under-sampling patterns and model

the MRI data in the image domain as the classic CS-MRI paradigm. Instead, we design a uniform under-

sampling strategy and explore the potential of modeling the MRI data directly in the measured Fourier

domain. We propose a dual-domain deep lattice network (DD-DLN) for CS-MRI with variable density uni-

form under-sampling. We train the networks to learn the mapping between both image and frequency

domains. We observe the dual networks have complementary advantages, which motivates their combi-

nation via a lattice structure. Experiments show that the proposed DD-DLN model provides promising

performance in CS-MRI under the designed variable density uniform under-sampling.

© 2020 Elsevier B.V. All rights reserved.

p

f

x

w

s

a

m

t

fi

s

[

t

a

i

i

s

i

1. Introduction

Magnetic resonance imaging (MRI) is an important technique

in the field of medical imaging. Despite its high resolution in

soft tissues and low radiation, slow data acquisition is a major

limitation of MRI [1] . The raw measurements in MRI are Fourier

“k-space” coefficients, and the diagnostic MR image is then ob-

tained by an inverse 2D Fast Fourier Transform (FFT), which is

also the case where parallel imaging is utilized that multi-channel

k-space data are obtained to produce image domain result. One

technique for accelerating MRI is compressed sensing [2,3] , which

has attracted much attention since it can be combined with other

accelerating methods, e.g., parallel imaging techniques such as

SENSE [4,5] . According to compressed sensing theory, the MRI

scan can obtain much fewer k-space measurements than the

classic Nyquist sampling theorem requires [1] while still allow-

ing for very accurate reconstructions. With compressed sensing

techniques being approved by US Food and Drug Administration

(FDA) to two main MRI vendors: GE and Siemens in year 2017,

more MRI data are expected to be generated using compressed

sensing [6] .

∗ Corresponding author.

E-mail address: dxh@xmu.edu.cn (X. Ding).

k

i

m

https://doi.org/10.1016/j.neucom.2020.01.063

0925-2312/© 2020 Elsevier B.V. All rights reserved.

Under the subsampling environment, the problem of com-

ressed sensing for magnetic resonance imaging (CS-MRI) can be

ormulated as

= arg min

x

λ

2

‖

F u x − y ‖

2 2 + ρ( x ) , (1)

here x ∈ C N × 1 denotes the vectorized MR image to be recon-

tructed, F u ∈ C M × N denot es the under-sampled Fourier matrix

nd y ∈ C M × 1 ( M < N ) denotes the vectorized k-space measure-

ents with unsampled positions removed. The term ρ( x ) is used

o regularize the ill-posed problem and the first term is the data

delity ensuring consistency on the Fourier coefficients of recon-

truction at the measured locations.

The classic CS-MRI problem contains three key ingredients

1] : (1) The MRI should have a sparse representation in some

ransform domain, such as wavelets or a learned dictionary basis,

s imposed by the regularization ρ . (2) The aliasing artifacts

ntroduced by under-sampling in k-space should be incoherent,

ndicating that the sampling mask should be as random as pos-

ible, so that the under-sampled Fourier operator F u is highly

ncoherent. (3) The reconstruction is performed using the partial

-space measurements based on certain sparse regularization.

Following these three prerequisites, research in CS-MRI falls

nto proposing effective sparse regularization functions ρ( · ) for

odeling the MRI and algorithms for their efficient optimization.

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 95

Fig. 1. Different under-sampling patterns and the corresponding under-sampled k-space data and zero-filling MRI.

L

c

t

i

a

a

d

[

D

s

t

l

s

F

t

M

M

i

o

T

s

f

S

r

m

d

V

C

i

w

w

t

t

a

1

a

M

A

s

a

T

s

t

z

fi

i

M

n

s

p

M

c

ustig et al. [1] proposed the pioneering SparseMRI, where the

lassic objective function is formed by a data fidelity with to-

al variation minimization [7,8] and � 1 wavelet regularization us-

ng conjugated gradient method for optimization. Some work has

imed at optimizing this objective function more efficiently, such

s TVCMRI [9] , RecPF [10] and FCSA [11] . Others have focused on

esigning adaptive basis transforms using wavelets such as PBDW

12] , PANO [13] and GBRWT [14] , or dictionary learning such as

LMRI [15] , BPTV [16] and TLMRI [17] . These methods can offer

parser representations and yield better reconstruction accuracy at

he expense of higher computational burden. Other nonlocal regu-

arizations include NLR [18] and BM3D-MRI [19] . All these can be

ummarized as optimization-based CS-MRI methods.

Recently, deep learning has been utilized into CS-MRI field.

or example, Wang et al. [20] use the vanilla CNN to learning

he mapping from the zero-filling MR images to the full-sampled

R images. Lee et al. [21] proposed the modified U-Net for CS-

RI using the idea of residual learning. The Generative Adversar-

al Networks (GAN) models [22,23] are also utilized in the field

f compressed sensing MRI to generate high-quality MR images.

he above deep models directly learn the mapping from under-

ampled MRI data to full-sampled ones without considering un-

olding the classic inverse optimization in MRI recovery. Notably

chlemper et al. [24] proposed the deep cascade convolutional neu-

al network (DC-CNN) which achieves the state-of-the-art perfor-

ance by virtue of such unfolding structure. Similar unfolding

eep neural networks were also proposed in ADMM-Net [25] and

ariational Network (VN) [26] . Compared with optimization-based

S-MRI methods, deep CS-MRI is more computationally efficient

n application because only a forward pass is required with

ell-learned network parameters of the unfolded structure [27] ,

hile optimization-based methods require additional iterations

hat are often very time-consuming [24] .

From the previous work on CS-MRI, two key observations arise:

he desire for random under-sampling patterns and the need for

ppropriate regularization when reconstructing the MRI.

.1. Variable density random sampling

Variable density random under-sampling patterns have been

dopted due to the constraint of the second prerequisite of CS-

RI, a result of the Restricted Isometry Property (RIP) theorem [3] .

s a result, artifacts should appear to be random noise because of

uch random sampling. We show some illustrations in Fig. 1 with

fully-sampled k-space data and fully-sampled MR image given.

he under-sampling is simulated by applying the mask to the fully-

ampled k-space measurements. In the mask, the sampled posi-

ions are denoted by ones (white) and zeros (black) otherwise. The

ero-filling MRI is obtained via the inverse 2D FFT of the zero-

lling k-space data. We show a 2D 25% random sampling mask

n Fig. 1 , the corresponding under-sampled k-space and zero-filling

RI are also shown. We observe artifacts appear like random

oise [1] .

To see the effect of replacing the random sampling by uniform

ampling, we apply the 2D 25% consistent density uniform sam-

ling and obtain the corresponding k-space data and zero-filling

RI. The MRI suffers noticeably from inadequate sampling in the

entral low-frequency regions. We also fully-sample a small rect-

96 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

Fig. 2. The PSF functions of proposed uniform sampling masks and compared non-uniform ones.

s

d

t

f

I

i

p

I

m

G

t

u

s

s

w

k

o

I

m

i

c

m

m

d

i

1

t

s

p

m

s

1

angular region in the low-frequency center with the same uniform

sampling mask to obtain a variable density uniform sampling mask

with 26% under-sampling ratio 1 and show the k-space measure-

ments and zero-filling MRI. The artifacts are now more regular

and structured. Deep learning can excel in this structured setting

to relax the requirement for randomness in sampling. The auto-

mated transform by manifold approximation (AUTOMAP) [28] and

transfer learning [29] were primarily tested on the misaligned

and has shown potential of deep learning models in handling this

structural artifacts.

Throughout the paper, our proposed deep models are primarily

validated on such uniform under-sampling trajectories based on

two major motivations. In the standard CS-MRI formulation, the

incoherence of the under-sampling pattern is highly desired. In

the pioneering work SparseMRI [1] , the degree of coherence of

an under-sampling mask is measured by its point spread function

(PSF). The PSF is simply defined as P SF ( i, j ) = F H u F u ( i, j ) . The PSF

evaluates how a true underlying pixel leaks its energy to other pix-

els in the zero-filled MRI reconstruction. The PSF of a full-sampled

MRI is the identity and off-diagonal entries vanish. The values on

diagonal entries should be non-zero, however, non-zero or even

large off-diagonal entries indicate severe coherence. Here we plot

the PSF functions of our proposed 1D and 2D variable density

uniform under-sampling patterns versus the 1D and 2D vari-

able density non-uniform under-sampling in Fig. 2 . A number of

non-zeros entries in PSF maps indicate the coherence brought by

such regular sampling schemes. According to the Candes [2], the

number of measurements needed for accurate reconstruction is in

direct proportion to the degree of coherence of sensing matrix and

the inverse proportion to the underlying sparsity of a signal on a

transform domain. Previously the deep learning model has proved

their superior performance in restoring the random-noise-like

artifacts in incoherent sampling since the deep neural networks

could provide sparser representation. However, the strong seman-

tic representation ability of deep neural network is insufficiently

discussed in regarding to relax the requirement for incoherent

samplings according to Lustig et al. [1] . One purpose of designing

1 The variable density here denote the dense uniform sampling in low frequency

regions and sparse uniform sampling in high frequency regions.

p

b

i

s

uch coherence masks is to investigate how the properties of

eep neural network could help when exploring novel sampling

rajectories.

Another major motivation for such coherent sampling comes

rom the concerns of the potential medical image application.

n the field of parallel MRI imaging, the General Autocalibrat-

ng Partially Parallel Acquisition (GRAPPA) is a popular partial

arallel acquisition technique for MR imaging acceleration [30] .

n the GRAPPA model, multiple coils collect a set of k-space

easurements. The 1D Cartesian phase encoding is adopted in

RAPPA since it is suitable in practical MRI. The sampling pat-

ern of each GRAPPA coil is identical to the 1D variable density

niform sampling mask we developed in Fig. 11 . The full-sample

ub-regions in our proposed mask are referred as autocalibration

ignals (ACS). The ACS regions are leveraged to infer the GRAPPA

eights. Then the weights are utilized to interpolate the missing

-space data. The each individual coil image is recovered based

n the reconstructed k-space. The coil images are then combined.

n comparison, the dense sampling in low-frequency regions is

ainly motivated by capturing the majority of signal energy. Also,

n contrast to the linear GRAPPA weights obtained in situ from the

urrent data, the nonlinear network weights (parameters) in our

odel are inferred in massive-data-driven manner. The GRAPPA

odel is well studied in clinical MRI practice. However, the ran-

omness required in conventional sparsity-based compressed sens-

ng MRI is highly undesired in GRAPPA. Right now, the proposed

D variable density uniform sampling can be easily extended

o parallel GRAPPA MRI and meet the requirement for regular

ampling.

Despite the benefits of exploring the uniform sampling

atterns, we still evaluate our models on irregular sampling

asks to make our model more convincing in later experiment

ection.

.2. Frequency acquisition and image modeling

In most CS-MRI algorithms [12,15,17,24] , the reconstruction ap-

roach can be formulated as an iterative denoising that is solved

y alternating between denoising in the image domain and fill-

ng/correction in the k-space domain, as illustrated in Fig. 3 . More

pecifically, the zero-filling MRI is denoised using a particular regu-

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 97

Fig. 3. The standard diagram in designing CS-MRI algorithm.

l

t

c

i

v

s

a

i

b

[

d

i

n

t

fi

m

a

n

1

1

d

i

d

[

a

t

p

p

c

a

1

u

f

M

n

u

p

i

A

w

d

n

o

2

2

A

m

t

d

b

o

i

(

l

c

i

c

r

o

t

[

s

C

t

o

t

b

a

b

�

s

t

c

F

w

m

p

a

u

f

i

L

arization method and then is projected into k-space by the Fourier

ransform where the missing data is filled in. The filled-in Fourier

oefficients are transformed back into the image domain via the

nverse FFT, where the image is again denoised.

From this paradigm, CS-MRI is essentially k-space completion

ia finding a “noise-free” image that agrees with the measured k-

pace values [15] . The missing values of the unsampled positions

re estimated via a model in image domain, where visually mean-

ngful patterns exist. Rather, there may be some further advantages

y modeling k-space directly, since under the uncertainty principle

31] a trade-off in resolution between the spatial and frequency

omains exists. We therefore investigate the advantage of model-

ng in both the image and frequency domains with deep neural

etworks.

Based on the observations, we propose a dual-domain deep lat-

ice network (DD-DLN) for CS-MRI. To our knowledge, this is the

rst work to conduct the research on uniform under-sampling to

easure the MRI data and fuse the information from both im-

ge and frequency domain under the framework of deep neural

etworks.

.3. Related work

.3.1. Dual-domain models

Dual domain models—e.g., working in image and frequency

omains simultaneously—have attracted attention in the field of

mage processing. For example, Knaus and Zwicker developed

ual-domain filtering for image denoising [32,33] . Pierazzo et al.

34,35] further extended this work. Hirani et al. [36,37] proposed

modified Projection Onto Convex Sets (POCS) model to combine

he spatial and frequency information, and Wang et al. [38] pro-

osed the Deep Dual Domain D

3 model for fast restoration of com-

ressed JPEG images, where the deep sparse coding is used to en-

ode the sparsity nature of natural images in both image domain

nd DCT domain.

.3.2. Frequency interpolation

In the signal processing community the k-space interpolation is

sed in the non-uniform FFT (NUFFT) [39,40] to obtain Fourier in-

ormation of a finite length signal at any frequency. In the field of

RI, the structured low-rank matrix completion methods like an-

ihilating filter-based low-rank Hankel matrix approach (ALOHA) is

tilized to convert the CS-MRI models into frequency interpolation

roblem [41–43] . Some previous work attempt to interpolate miss-

ng k-space data in data-driven manner with residual learning [44] .

k-space convolutional neural network and image restoration net-

ork were developed in the KIKI-Net [45] . However, the proposed

ual-domain deep neural networks alternatives in a cascaded man-

er, meaning the sharing between the complementary information

f the dual domain is not fully utilized.

. Method

.1. Structural artifact removal

In Fig. 1 , we design a variable density under-sampling mask.

s discussed in the previous section, we expect the deep network

odel can distinguish structural artifacts from normal image struc-

ures brought on by uniform under-sampling, and so we design a

eep model for this end and compare it with other optimization-

ased CS-MRI models.

An intuitive and popular network design is shown in top row

f Fig. 4 , called deep residual network (ResNet) [46] . The network

n this case is formed by stacking complex-valued residual units

due to the Fourier transform). Each unit contains 4 convolutional

ayers with filters of size 3 × 3. A leaky ReLU with 0.2 slope rate is

hosen as the activation function, except for the last layer, which

s the identity. Details about the complex convolutional operations

an be found in [47] . The residual learning strategy in the complex

esidual unit has been shown to facilitate network training because

f better gradients during back propagation [46,48] .

Inspired by the classic CS-MRI paradigm in Fig. 3 , which is also

he motivation for the state-of-the-art deep CS-MRI model DC-CNN

24] , we propose an image domain reconstruction network (IDRN)

hown in Fig. 4 . The IDRN model shares a identical structure to DC-

NN except the complex convolution is utilized in IDRN, while a

wo-channel scheme is adopted in DC-CNN. In our experiments we

bserve the complex convolution leads to fewer network parame-

ers with comparable model performance. The network is formed

y cascading blocks. Each block contains a complex residual unit

nd a k-space correction. If the model consists of N such inference

locks, we call the model IDRN- N B.

We represent the subset of k-space that has been sampled as

and define y zf ∈ C N × 1 the k-space measurements where un-

ampled positions are filled with zeros. The input and output of

he corrected k-space (KC) is denoted a x in and x out . The k-space

orrection for the k th entries of the x in in IDRN can be written as

x out ( k ) k / ∈ � = F x in ( k ) , F x out ( k ) k ∈ � = y z f ( k ) , (2)

here F is the Fourier transform. For IDRN, similar to other CS-MRI

ethods that model the image domain, the output from the com-

lex residual unit is transformed back into the frequency domain

nd the missing Fourier coefficients are filled in by these new val-

es. The k-space is then transformed back into image space and

ed into the next residual unit. The loss function for training IDRN

s

IDRN ( θ ) =

1

∥∥x f s − f θ(x z f

)∥∥2

2 , (3)

2

98 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

Fig. 4. The network architecture of the ResNet, IDRN and FDRN. KC stands for k-space correction.

Fig. 5. Deep neural networks (IDRN with 5 blocks) can denoise the and remove artifacts better than optimization-based approaches (e.g., PANO).

i

s

t

w

t

t

s

d

r

t

n

F

I

r

v

f

q

q

i

f

c

d

c

r

r

where x fs is the fully-sampled MRI and x zf is the corresponding

zero-filling MRI. f θ ( · ) denotes the function mapping of IDRN with

network parameters θ .

We compare the state-of-the-art optimization-based CS-MRI

method PANO [13] with the proposed IDRN-5B model using the

variable density uniform sampling mask in Fig. 1 . We show the re-

construction on a test MRI data in Fig. 5 . We observe the PANO re-

construction can preserve image structures but struggles to remove

the structural artifacts. IDRN-5B model on the other hand can re-

move these artifacts without interfering with the image structures.

2.1.1. Frequency domain reconstruction network

From Fourier analysis, the 2D discrete Fourier transform on an

image f of the size M × N is denoted as

F [ k, l ] =

1 √

MN

N−1 ∑

n =0

M−1 ∑

m =0

f [ m, n ] e − j2 π( mk M + nl

N )

f [ m, n ] =

1 √

MN

N−1 ∑

l=0

M−1 ∑

k =0

F [ k, l ] e j2 π( mk M + nl

N )

( 0 ≤ m, k ≤ M − 1 , 0 ≤ n, l ≤ N − 1 )

. (4)

Where the first equation is Fourier analysis and the second Fourier

synthesis. According to the Fourier analysis equation, each k-space

coefficient is obtained by a dot product with all pixels across the

entire spatial domain, meaning it contains information from the

entire image. Conversely, the entire frequency domain contributes

nformation to every pixel of the image according to the Fourier

ynthesis equation. This has inspired popular signal processing

ools such as the short-time Fourier transform (STFT) [49] and

avelets [50] to analyze a signal in multi-scale fashion.

In the conventional optimization-based CS-MRI, the approach is

o model statistical characteristics of the MRI, such as sparsity, in

he image domain. Although CS-MRI is essentially the process of k-

pace completion, few models that work directly in the frequency

omain exists. In this work we consider learning a mapping di-

ectly in the frequency domain with a deep network. We name

he architecture for this approach a frequency domain reconstruction

etwork (FDRN), as shown in Fig. 4 . Similarly, we call the model

DRN- N B if it is formed by cascading N inference blocks.

The FDRN is constituted by cascaded inference blocks similar to

DRN, but with two simple differences. First, the complex-valued

esidual unit works on the frequency domain. In the traditional

ariable density sampling scheme, the dense sampling in the low

requency regions and sparse or even no sampling in the high fre-

uency regions, is not suitable for such direct convolution in fre-

uency domain. This phenomenon was also not studied thoroughly

n [44] . While for the variable density uniform sampling, the uni-

orm sampling guarantees that data appears in each step of the

onvolution, which is especially unlikely with variable density ran-

om sampling when the filters are 3 × 3. Second, using k-space

orrection with FDRN, the measured samples in k-space directly

eplace the corresponding positions in the output of the complex

esidual unit in the same block. Unlike IDRN, no image/frequency

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 99

Fig. 6. The network architecture of the dual-domain deep lattice network.

d

d

a

o

y

T

w

F

L

w

s

f

2

a

w

f

s

i

b

n

t

t

c

n

f

I

i

L

T

F

t

t

D

s

e

i

I

F

r

t ∣∣r

a

l

u

i

F

c

f

a

a

m

b

a

T

v

t

i

H

fi

E

t

b

S

r

r

fi

3

F

s

omain transform is necessary for k-space correction (hence the

ifferent color of KC in Fig. 4 ).

In FDRN, the input and output of the k-space correction step

re denoted y in and y out . The k-space correction for the k th entries

f the y in in FDRN can be written as

out ( k ) k / ∈ � = y in ( k ) , y out ( k ) k ∈ � = y z f ( k ) . (5)

he output of the last block in FDRN is the completed k-space,

hich we then transform into the image domain with an inverse

FT. The loss function for training FDRN model is

F DRN ( θ ) =

1

2

∥∥y f s − g θ(y z f

)∥∥2

2 , (6)

here y fs is the fully-sampled k-space data and y zf is the corre-

ponding zero-filling k-space measurements. g θ ( · ) denotes the

unction mapping of FDRN with network parameters θ .

.2. A dual-domain deep lattice network

Based on the proposed IDRN and FDRN models, we propose

dual-domain deep lattice network (DD-DLN), shown in Fig. 6 ,

hich fuses information from IDRN and FDRN in a single objective

ramework. Here the two subnetworks interact in the form of the

hown lattice structure. (In digital filter design, the term “lattice”

s often referred to as a cross-connection structure [51] , which we

orrow here.) Now, for a certain block in the IDRN or FDRN sub-

etwork, the output is not only fed into the next block, but also

ransformed into frequency or image domain and concatenated to

he other subnetwork as indicated by arrows in Fig. 6 . The con-

atenation operation helps to fuse information from both domains.

As before, we call the DD-DLN model DD-DLN- N B if each sub-

etwork contains N blocks. DD-DLN produces a reconstructed MRI

rom each branch (or subnetwork), which we call DD-DLN- N B-

DRN and DD-DLN- N B-FDRN. The loss function for training DD-DLN

s

DD −DLN =

1

2

( L IDRN + L F DRN ) . (7)

his is our final framework, but we also experiment with IDRN and

DRN separately in the next section to evaluate the advantage of

he network shown in Fig. 6 .

To further evaluate our conjecture that information sharing be-

ween the image and frequency domains is feasible, we train the

D-DLN-5B model on the variable density uniform sampling mask

hown in Fig. 1 and give intermediate reconstruction results of

ach block for both IDRN-5B branch and FDRN-5B branch, shown

n the first and second row of Fig. 7 respectively. We observe the

DRN-5B branch has better imaging quality, but to check whether

DRN-5B branch provides a more accurate reconstruction in some

egions we look at the differences between their absolute errors

o the ground truth. We denote the absolute error of IDRN e IDRN =x IDRN − x f s

∣∣ and FDRN e F DRN =

∣∣x F DRN − x f s

∣∣, where x IDRN and x FDRN

epresents the output MR images from the corresponding IDRN-5B

nd FDRN-5B models. We take the difference between the abso-

ute error of IDRN-5B and FDRN-5B and keep the non-negative val-

es, yielding the difference map m di f f = ( e IDRN − e F DRN ) + . The pos-

tive parts indicate the regions of more accurate reconstruction of

DRN-5B than IDRN-5B.

In the last row of Fig. 7 we show the difference maps with the

olormaps in display range [0 0.1], i.e., positive parts of this dif-

erence, corresponding to locations where FDRN branch was more

ccurate than IDRN branch. These difference maps are generated

cross different blocks. We observe in the shallow blocks, the FDRN

odel provides better reconstruction qualities on the edges of the

rain (edges and outlines), while the IDRN model achieves higher

ccuracies on the small structures (low-contrast) in the MR image.

his phenomenon can be attributed to the facts that the deep con-

olutional neural network in image domain has difficulty in cap-

uring large contextual information in shallow layers due to lim-

ted receptive fields especially with small kernel size, e.g., 3 × 3.

owever, for deep models in frequency domain, the large receptive

elds can be achieved in shallow layers according to our analysis of

q. 4 . We also observe in the last block, the IDRN branch achieves

he completely superior performance over FDRN branch, which can

e proved by the fact the difference map is nearly zeros in Fig. 7 .

ince the IDRN produces better reconstruction than the FDRN and

epresents the state-of-the-art performance in under-sampled MRI

econstruction, we reasonably regard the DD-DLN- N B-IDRN as the

nal output.

. Results

In this section we experiment with the proposed deep IDRN,

DRN and fused DD-DLN models on different uniform under-

ampling patterns with different under-sampling ratios to evaluate

100 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

Fig. 7. The performance of DD-DLN-5B-IDRN and DD-DLN-5B-FDRN. The intermediate reconstructions from each block in both IDRN-5B branch and FDRN-5B branch are

shown in the first row and second row, respectively. We show post-processed absolute differences of reconstruction error in the third row. Brightness indicates FDRN branch

is more accurate.

b

d

3

r

1

W

1

T

t

3

d

T

t

m

P

a

h

[

S

their effectiveness. We also compare with several state-of-the-art

non-deep CS-MRI techniques: transform learning MRI (TLMRI) [17] ,

graph-based redundant wavelet transform (GBRWT) [14] , patch-

based nonlocal operator (PANO) [13] and BM3D-MRI [19] . Since

IDRN shares a similar network structure with DC-CNN [24] , it can

be taken as representing the state-of-the-art in CS-MRI, which we

will see is improved upon by DD-DLN. We also compare with the

unfolding deep architecture ADMM-Net [26] .

3.1. Dataset

In the experiment section, we present experimental results

using complex-valued MRI datasets. The datasets are collected in

the second affiliated hospital of Xiamen Medical College. The T1

weighted MRI dataset (size 256 × 256) is acquired on 40 volun-

teers with total 3800 MR images at Siemens 3.0T scanner with

12 coils using the fast low angle shot (FLASH) sequence (TR/TE

= 55/3.6 ms, 220 mm

2 field of view, 1.5 mm slice thickness).

Following the simulation setting in [13] , the SENSE reconstruction

is introduced to compose the gold standard full k-space, which

is used to emulate the single-channel MRI. We randomly select

80% MR images as training set and 20% MR images as testing set.

Informed consent was obtained from the imaging subject in com-

pliance with the Institutional Review Board policy. We apply nor-

malization on the full-sampled MR images. It is worth noting the

experiment data used in the work is complex-valued MR images,

which conforms to the fact the actual measurements of MRI are

complex-valued.

Under-sampled k-space measurements are manually obtained

via 1D Cartesian and 2D random sampling mask. Different under-

sampling ratios are adopted in the experiments. The under-

sampling ratio here is defined as the ratio between the num-

er of partially sampled data and the number of full-sampled

ata.

.2. Implementation details

We use ADAM for network optimization with an initial learning

ate of 0.0 0 01. The training runs for 20,0 0 0 iterations and at every

00 iterations we decrease the learning rate by a multiple of 0.99.

e implement the deep models on Tensorflow using NVIDIA GTX

080 with 8G memory and Intel Xeon CPU E5-2683 at 2.00 GHz.

he deep learning models and conventional non-deep models are

ested on the same device for fair comparison in testing phase.

.3. Evaluation metric

We use peak-signal-to-ratio (PSNR) and structural similarity in-

ex (SSIM) as the objective metric to compare model performance.

he PSNR is a widely-used metric to measure the distance between

he reconstructed MRI x recon and full-sampled MRI x fs , which is for-

ulated as

SNR = 20 log 10

(

1 ∥∥x recon − x f s

∥∥2

2

)

. (8)

Although the PSNR index can assess the absolute reconstruction

ccuracy, the reconstruction quality of structural information can

ardly to be well evaluated. Therefore we use another index SSIM

52] to compare the models. The SSIM is formulated as

SIM =

(2 μ f s μ recon + C 1

)(2 σ f s,recon + C 2

)(μ 2

f s + μ 2

recon + C 1 )(

σ 2 f s

+ σ 2 recon + C 1

) , (9)

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 101

Table 1

Reconstruction time of different methods. Note the PANO, TLMRI, GBRWT and BM3D-MRI methods are non-deep models without the need for GPU computation.

PANO TLMRI GBRWT BM3D-MRI FDRN-5B IDRN-5B DD-DLN-5B

Runtime (seconds) 14.82 s 100.8 s 138.1 s 14.94 s 0.26 s 0.3 s 0.42 s

Fig. 8. The training and validation losses of IDRN-5B, FDRN-5B and DDDLN-5B are

shown.

w

s

σ

i

n

3

t

o

r

t

t

m

M

m

t

p

c

d

t

3

t

F

I

t

d

[

i

a

o

r

t

o

r

a

a

p

h

F

w

t

s

m

e

q

b

i

a

5

o

b

F

p

5

L

f

u

v

c

p

p

4

4

m

t

A

p

4

s

r

m

t

F

here μfs and μrecon represent the mean value of the recon-

tructed full-sampled and reconstructed MR images, and σ fs ,

recons and σ fs,recons denote the standard deviation of the two

mages and their covariance. C 1 and C 2 are two constants for

umeric stability.

.4. Running time

The compressed sensing MRI models regularized with conven-

ional sparse or non-local prior requires time-consuming iterative

ptimization. However, for deep-based CS-MRI models, the forward

econstruction is rapid without needs of iterative optimization if

he network is well-trained. We show the averaged reconstruc-

ion time on the test dataset with uniform 2D 25% under-sampling

ask in Table 1 . We observe the PANO and BM3D-MRI reconstruct

RI data faster than TLMRI and GBRWT, but still the non-deep

odels takes more than 10 s for reconstruction. While the state-of-

he-art deep model IDRN-5B model (similar to DC-CNN [24] ), the

roposed frequency learning model FDRN-5B have the fastest re-

onstruction speed around 0.3 seconds. Our proposed dual domain

eep lattice network takes 0.1 s longer than IDRN-5B and FDRN-5B

o reconstruct a MRI data in average.

.5. Evaluation

We compare the training and validation errors in Fig. 8 using

he 26% 2D variable density uniform sampling mask as shown in

ig. 1 , showing a stable more lower error of DD-DLN model over

DRN and FDRN model.

In Fig. 9 , we compare the DD-DLN-5B model with state-of-

he-art optimization-based CS-MRI methods including several non-

eep models and the ADMM-Net with the unfolding deep structure

o

26] , as well as with IDRN-5B and FDRN-5B on a test MRI data us-

ng 2D variable density uniform sampling. The residual error maps

re also provided in the range 0–0.3. These results correspond to

ne testing image, but represents the common observation. Pa-

ameters of the compared optimization-based CS-MRI methods are

uned to give their best performance using source codes provided

n the respective authors’ websites.

We observe that optimization-based methods have difficulty in

emoving structural artifacts. GBRWT achieves good performance

t this task, but still with obvious artifacts. For TLMRI, the artifacts

re better removed at the expense of image blurring. FDRN-5B

roduces worse reconstruction compared with IDRN-5B, but both

andle artifacts well. DD-DLN-5B improves further on independent

DRN and IDRN results, as seen by the errors of their subnetworks

ithin this joint framework. As a quantitative evaluation, we show

he PSNR and SSIM averaged over the test MRI in Fig. 10 . The re-

ults are consistent with the subjective reconstruction quality.

Because of the lattice structure, the DD-DLN-5B model has

ore network parameters than the IDRN-5B and FDRN-5B mod-

ls. Thus a natural question arises: Is the improved reconstruction

uality of DDDLN model attributed to the enlarged model capacity

rought by more network parameters rather than the dual domain

nformation? We expand the model capacity of IDRN-5B model to

dapt it to the near identical number of parameters of DD-DLN-

B. We name the enlarged IDRN-5B as IDRN-5B-L with the number

f parameters 390k, while the DD-DLN-5B models have the num-

er of parameters of 386k. The comparison results are shown in

ig. 10 . We observe the larger model capacity of IDRN-5B-L out-

erforms the IDRN-5B with some margins, however, the DD-DLN-

B-IDRN still achieves better reconstruction quality over IDRN-5B-

, demonstrating the effectiveness of the dual domain information

usion idea.

We also test DD-DLN using the 1D Cartesian variable density

niform mask shown in Fig. 11 . The residual error maps are pro-

ided in the same range as the 2D case. Also we fully-sample the

entral low-frequency regions to retain most of the energy. The ex-

erimental results in Fig. 11 again show the effectiveness of the

roposed model, and robustness to various sampling patterns.

. Discussions

.1. Discussion on number of blocks

We show how the performances of IDRN, FDRN and DD-DLN

odels varies with number of blocks in Fig. 12 . We observe

he DD-DLN-IDRN model steadily outperforms other three models.

s the network architectures go deep, the DD-DLN-FDRN model

erforms well and approaches the DD-DLN-IDRN model.

.2. Discussion on different under-sampling ratios

We also compare our proposed models with other compressed

ensing MRI methods on uniform 2D masks with under-sampling

atios of 26%, 28%, 30%, 32% and 40%. The higher sampling ratios

eans larger full-sampling central regions in our design. We show

he 26% mask in Fig. 9 and the 28%, 30%, 32% and 40% masks in

ig. 13 .

As shown in Table 2 , the DD-DLN-5B-IDRN model outper-forms

ther models on all sampling patterns. We observe the DD-DLN-

102 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

Fig. 9. The reconstruction results with 26% 2D variable density uniform sampling masks and their corresponding error images.

Table 2

Model comparison conditioned on 2D variable density uniform masks with different under-sampling ratios.

Under-sampling Ratios % 26 28 30 32 40

Evaluation Index PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM

PANO [13] 26.08 0.774 27.41 0.859 28.62 0.882 30.49 0.909 33.32 0.947

TLMRI [17] 25.81 0.785 26.74 0.761 27.33 0.773 28.24 0.796 30.17 0.854

GBRWT [14] 25.71 0.834 29.07 0.894 30.68 0.910 32.57 0.929 35.39 0.957

BM3D-MRI [19] 27.93 0.748 29.87 0.816 30.88 0.846 32.27 0.880 35.38 0.936

ADMM-Net [26] 27.72 0.738 29.86 0.850 31.59 0.905 32.67 0.915 37.96 0.971

FDRN-5B 28.20 0.846 30.77 0.901 31.67 0.917 33.18 0.940 36.06 0.968

IDRN-5B [24] 32.14 0.949 35.50 0.973 36.65 0.977 38.41 0.983 41.18 0.989

DD-DLN-5B-FDRN 32.56 0.949 35.45 0.973 36.71 0.978 38.23 0.983 41.05 0.989

DD-DLN-5B-IDRN 32.80 0.955 35.73 0.974 36.99 0.979 38.50 0.983 41.22 0.989

4

W

m

p

F

s

a

5B-FDRN achieves comparable performance with IDRN-5B model.

Interestingly, with the higher under-sampling ratio, the gap be-

tween the model performance of IDRN-5B and DD-DLN-5B-IDRN

narrows.

The performance of GBRWT boosts rapidly with high under-

sampling ratio because more accurate graph representation can be

achieved with less structural artifacts. However, for BM3D-MRI, the

structural artifacts impose an adverse influence on the searching

of similar patches in non-local prior, resulting in unsatisfactory

reconstructions. a

d

.3. Discussion on conventional non-uniform masks

We validate our model on a 30% 2D random sampling mask.

e employ the same experimental setting as reported in the

anuscript with the only difference being the different sampling

atterns. We show an example MRI construction comparison in

ig. 14 . Along with the reconstructions we also show the corre-

ponding residual error maps and objective metrics such as PSNR

nd SSIM. We observe the DD-DLN-5B-IDRN preserves the fine im-

ge details better compared with the baseline IDRN-5B on this ran-

om sampling mask. Both deep models outperform the non-deep

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 103

Fig. 10. The PSNR and SSIM averaged on test MRI data.

Table 3

Averaged PSNR and SSIM values on the test dataset on the 30% 2D Random under-

sampling pattern.

Metrics TLMRI PANO IDRN-5B DD-DLN-5B-IDRN

PSNR dB 35.87 36.27 38.78 39.12

SSIM 0.897 0.923 0.946 0.951

m

w

i

p

d

n

d

4

o

i

n

r

a

s

b

n

the artifacts.

odels, which is already well established in previous deep CS-MRI

orks.

We also show the PSNR and SSIM values averaged on the test-

ng datasets in Table 3 . We observe the results random sampling

rovides higher reconstruction accuracy reasonably due to the high

egree of incoherence. Even though, our dual domain deep lattice

etwork also offers gains over the state-of-the-art image domain

eep model.

.4. Further analysis

For the IDRN subnetwork of DD-DLN-5B, we show the residuals

f the complex-valued residual unit within in each cascaded

nference block in Fig. 15 . The residuals become smaller as the

etwork goes deeper, because the deeper blocks produce better

econstructions.

In the lower blocks, the residuals should ideally consist of neg-

tive artifacts. We observe that the magnitudes of the residuals

how a similar structure in the artifacts as the block goes deeper,

ut it is less pronounced, meaning the deeper components in the

etwork are better able to distinguish the image structure within

104 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

Fig. 11. The reconstruction results with 26% 1D variable density uniform sampling masks and their corresponding error images.

Fig. 12. The comparison of model performance of FDRN, IDRN and DD-DLN conditioned on the number of cascaded blocks.

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 105

Fig. 13. The uniform 2D masks with under-sampling ratios of 26%, 28%, 30%, 32% and 40%.

Fig. 14. The comparison of models on the 30% 2D random under-sampling pattern.

Fig. 15. The magnitudes of the residuals for each block of the DD-DLN-5B-IDRN subnetwork.

5

b

c

a

b

c

s

w

m

D

c

i

A

S

6

F

G

f

C

a

2

R

. Conclusion

The variable density random under-sampling pattern required

y RIP and appropriate image regularizations are two common fo-

uses in performing CS-MRI inversion. In this paper, we proposed

dual-domain deep lattice network to fuse the information from

oth image and frequency domains using a sampling strategy that

aptures most of the low resolution information. The experiments

how that the proposed deep model has promising performance

ith the uniform sampling strategy compared with non-deep

ethods.

eclaration of Competing Interests

The authors declare that they have no known competing finan-

ial interests or personal relationships that could have appeared to

nfluence the work reported in this paper.

cknowledgment

This work was supported in part by the National Natural

cience Foundation of China under Grants 61571382 , 81671766 ,

1571005 , 81671674 , 61671309 and U1605252 , in part by the

undamental Research Funds for the Central Universities under

rant 20720160075 , 20720180059 , in part by the CCF-Tencent open

und, and the Natural Science Foundation of Fujian Province of

hina (No. 2017J01126 ). L. Sun conducted portions of this work

t Columbia University under China Scholarship Council grant (No.

01806310090 ).

eferences

[1] M. Lustig , D. Donoho , J.M. Pauly , Sparse MRI: the application of compressedsensing for rapid MR imaging, Magn. Reson. Med. 58 (6) (2007) 1182–1195 .

[2] D.L. Donoho , Compressed sensing, IEEE Trans. Inf. Theory 52 (4) (2006)1289–1306 .

[3] E.J. Candes , The restricted isometry property and its implications for com-

pressed sensing, C.R. Math. 346 (9–10) (2008) 589–592 . [4] K.P. Pruessmann , M. Weiger , M.B. Scheidegger , P. Boesiger , et al. , SENSE: sensi-

tivity encoding for fast MRI, Magn. Reson. Med. 42 (5) (1999) 952–962 . [5] D. Liang , B. Liu , J. Wang , L. Ying , Accelerating SENSE using compressed sensing,

Magn. Reson. Med. 62 (6) (2009) 1574–1584 . [6] J.A. Fessler, Medical image reconstruction: a brief overview of past milestones

and future directions. arXiv preprint arXiv: 1707.05927 . [7] V.P. Gopi , P. Palanisamy , K.A. Wahid , P. Babyn , D. Cooper , Multiple regulariza-

tion based MRI reconstruction, Signal Process. 103 (2014) 103–113 .

[8] P. Zhuang , X. Zhu , X. Ding , MRI reconstruction with an edge-preserving filter-ing prior, Signal Process. 155 (2019) 346–357 .

[9] S. Ma , W. Yin , Y. Zhang , A. Chakraborty , An efficient algorithm for compressedMR imaging using total variation and wavelets, in: Computer Vision and Pat-

tern Recognition, IEEE, 2008, pp. 1–8 .

106 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107

[10] J. Yang , Y. Zhang , W. Yin , A fast alternating direction method for TVL1-L2 signalreconstruction from partial fourier data, IEEE J. Sel. Top. Signal Process. 4 (2)

(2010) 288–297 . [11] J. Huang , S. Zhang , D. Metaxas , Efficient MR image reconstruction for com-

pressed MR imaging, Med. Image Anal. 15 (5) (2011) 670–679 . [12] X. Qu , D. Guo , B. Ning , Y. Hou , Y. Lin , S. Cai , Z. Chen , Undersampled MRI re-

construction with patch-based directional wavelets, Magn. Reson. Imaging 30(7) (2012) 964–977 .

[13] X. Qu , Y. Hou , L. Fan , D. Guo , J. Zhong , Z. Chen , Magnetic resonance image re-

construction from undersampled measurements using a patch-based nonlocaloperator, Med. Image Anal. 18 (6) (2014) 843 .

[14] Z. Lai , X. Qu , Y. Liu , D. Guo , J. Ye , Z. Zhan , Z. Chen , Image reconstructionof compressed sensing MRI using graph-based redundant wavelet transform.,

Med. Image Anal. 27 (2016) 93 . [15] S. Ravishankar , Y. Bresler , MR Image reconstruction from highly undersampled

k-space data by dictionary learning, IEEE Trans. Med. Imaging 30 (5) (2011)

1028–1041 . [16] Y. Huang , J. Paisley , Q. Lin , X. Ding , X. Fu , X.-P. Zhang , Bayesian nonparametric

dictionary learning for compressed sensing MRI, IEEE Trans. Image Process. 23(12) (2014) 5007–5019 .

[17] S. Ravishankar , Y. Bresler , Efficient blind compressed sensing using sparsify-ing transforms with convergence guarantees and application to magnetic res-

onance imaging, SIAM J. Imaging Sci. 8 (4) (2015) 2519–2557 .

[18] W. Dong , G. Shi , X. Li , Y. Ma , F. Huang , Compressive sensing via nonlocallow-rank regularization, IEEE Trans. Image Process. 23 (8) (2014) 3618–3632 .

[19] E.M. Eksioglu , Decoupled algorithm for MRI reconstruction using nonlo-cal block matching model: BM3D-MRI, J. Math. Imaging Vis. 56 (3) (2016)

430–440 . [20] S. Wang , Z. Su , L. Ying , X. Peng , S. Zhu , F. Liang , D. Feng , D. Liang , Accelerating

magnetic resonance imaging via deep learning, in: International Symposium

on Biomedical Imaging, 2016, pp. 514–517 . [21] D. Lee , J. Yoo , J.C. Ye , Deep residual learning for compressed sensing MRI, in:

International Symposium on Biomedical Imaging, 2017, pp. 15–18 . [22] G. Yang , S. Yu , H. Dong , G. Slabaugh , P.L. Dragotti , X. Ye , F. Liu , S. Arridge , J. Kee-

gan , Y. Guo , et al. , Dagan: deep de-aliasing generative adversarial networks forfast compressed sensing MRI reconstruction, IEEE Trans. Med. Imaging 37 (6)

(2018) 1310–1321 .

[23] T.M. Quan , T. Nguyen-Duc , W.-K. Jeong , Compressed sensing MRI reconstruc-tion using a generative adversarial network with a cyclic loss, IEEE Trans. Med.

Imaging 37 (6) (2018) 1488–1497 . [24] J. Schlemper , J. Caballero , J.V. Hajnal , A. Price , D. Rueckert , A deep cascade

of convolutional neural networks for MR image reconstruction, in: Interna-tional Conference on Information Processing in Medical Imaging, Springer,

2017, pp. 647–658 .

[25] K. Hammernik , T. Klatzer , E. Kobler , M.P. Recht , D.K. Sodickson , T. Pock , F. Knoll ,Learning a variational network for reconstruction of accelerated MRI data,

Magn. Reson. Med. 79 (6) (2018) 3055–3071 . [26] Y. Yang , J. Sun , H. Li , Z. Xu , ADMM-CSNet: a deep learning approach for image

compressive sensing, IEEE Trans. Pattern Anal. Mach. Intell. (2018) . [27] K. Gregor , Y. LeCun , Learning fast approximations of sparse coding, in: Inter-

national Conference on Machine Learning, Omnipress, 2010, pp. 399–406 . [28] B. Zhu , J.Z. Liu , S.F. Cauley , B.R. Rosen , M.S. Rosen , Image reconstruction by

domain-transform manifold learning, Nature 555 (7697) (2018) 487 .

[29] F. Knoll , K. Hammernik , E. Kobler , T. Pock , M.P. Recht , D.K. Sodickson , Assess-ment of the generalization of learned image reconstruction and the potential

for transfer learning, Magn. Reson. Med. 81 (1) (2019) 116–128 . [30] M.A. Griswold , P.M. Jakob , R.M. Heidemann , M. Nittka , V. Jellus , J. Wang ,

B. Kiefer , A. Haase , Generalized autocalibrating partially parallel acquisitions(grappa), Magn. Reson. Med.: Off. J. Int. Soc. Magn. Reson. Med. 47 (6) (2002)

1202–1210 .

[31] R. Wilson , G.H. Granlund , The uncertainty principle in image processing, IEEETrans. Pattern Anal. Mach. Intell. (6) (1984) 758–767 .

[32] C. Knaus , M. Zwicker , Dual-domain filtering, SIAM J. Imaging Sci. 8 (3) (2015)1396–1420 .

[33] C. Knaus , M. Zwicker , Dual-domain image denoising, in: International Confer-ence on Image Processing, IEEE, 2013, pp. 4 40–4 4 4 .

[34] N. Pierazzo , M. Lebrun , M. Rais , J.-M. Morel , G. Facciolo , Non-local dual im-

age denoising, in: International Conference on Image Processing, IEEE, 2014,pp. 813–817 .

[35] N. Pierazzo , M.E. Rais , J.-M. Morel , G. Facciolo , DA3D: fast and data adaptivedual domain denoising, in: International Conference on Image Processing, IEEE,

2015, pp. 432–436 . [36] A.N. Hirani , T. Totsuka , Dual domain interactive image restoration: basic algo-

rithm, in: Proceedings of the International Conference on Image Processing,

1996, 1, IEEE, 1996, pp. 797–800 . [37] A.N. Hirani , T. Totsuka , Combining frequency and spatial domain informa-

tion for fast interactive image noise removal, in: Proceedings of the 23rd An-nual Conference on Computer Graphics and Interactive Techniques, ACM, 1996,

pp. 269–276 . [38] Z. Wang , D. Liu , S. Chang , Q. Ling , Y. Yang , T.S. Huang , D3: deep dual-domain

based fast restoration of JPEG-compressed images, in: Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition, 2016, pp. 2764–2772 .

[39] J.A. Fessler , B.P. Sutton , Nonuniform fast fourier transforms usingmin-max interpolation, IEEE Trans. Signal Process. 51 (2) (2003)

560–574 . [40] F. Marvasti , Nonuniform Sampling: Theory and Practice, Springer Science &

Business Media, 2012 . [41] K.H. Jin , D. Lee , J.C. Ye , A general framework for compressed sensing and par-

allel MRI using annihilating filter based low-rank hankel matrix, IEEE Trans.Comput. Imaging 2 (4) (2016) 4 80–4 95 .

[42] P.J. Shin , P.E. Larson , M.A. Ohliger , M. Elad , J.M. Pauly , D.B. Vigneron ,

M. Lustig , Calibrationless parallel imaging reconstruction based on struc-tured low-rank matrix completion, Magn. Reson. Med. 72 (4) (2014)

959–970 . [43] J.P. Haldar , Low-rank modeling of local k-space neighborhoods (LORAKS) for

constrained MRI, IEEE Trans. Med. Imaging 33 (3) (2014) 66 8–6 81 . [44] Y. Han , J.C. Ye , K-space deep learning for accelerated MRI, IEEE T. Med. Imaging

(2019) .

[45] T. Eo , Y. Jun , T. Kim , J. Jang , H.-J. Lee , D. Hwang , Kiki-net: cross-domain convo-lutional neural networks for reconstructing undersampled magnetic resonance

images, Magn. Reson. Med. 80 (5) (2018) 2188–2201 . [46] K. He , X. Zhang , S. Ren , J. Sun , Deep residual learning for image recognition, in:

Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni-tion, 2016, pp. 770–778 .

[47] C. Trabelsi , O. Bilaniuk , D. Serdyuk , S. Subramanian , J.F. Santos , S. Mehri ,

N. Rostamzadeh , Y. Bengio , C.J. Pal , Deep complex networks, Internet Confer-ence on Learning Representation (2018) .

[48] K. He , X. Zhang , S. Ren , J. Sun , Identity mappings in deep residual net-works, in: European Conference on Computer Vision, Springer, 2016, pp.

630–645 . [49] E. Sejdi ́c , I. Djurovi ́c , J. Jiang , Time–frequency feature representation using en-

ergy concentration: an overview of recent advances, Digit. Signal. Process. 19

(1) (2009) 153–183 . [50] S.G. Mallat , A theory for multiresolution signal decomposition: the wavelet

representation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (7) (1989) 674–693 .

[51] M. Honig , D. Messerschmitt , Convergence properties of an adaptive digital lat-tice filter, IEEE Trans. Circuits Syst. 28 (6) (1981) 4 82–4 93 .

[52] Z. Wang , A.C. Bovik , H.R. Sheikh , E.P. Simoncelli , Image quality assessment:

from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4)(20 04) 60 0–612 .

Liyan Sun received the B.S. degree from ZhengzhouUniversity, Zhengzhou, Henan, China in 2014. He is

currently pursuing the Ph.D. degree with the Schoolof Information Science and Engineering from Xiamen

University, Xiamen, Fujian, China. From 2018 to 2019, He

was a Visiting Scholar with the Department of ElectricalEngineering, Columbia University, New York, NY, USA.

His research interests mainly focus on machine learning,medical image reconstruction and analysis.

Yawen Wu received the B.S. degree from Fuzhou Univer-

sity, China in 2017. She is currently pursuing the masterdegree in the Department of Communication Engineering,

Xiamen University, China. Her research interests mainlyfocus on deep learning and medical image analysis.

Binglin Shu received the B.S. degree from Xiamen Univer-

sity, China in 2016. He is currently pursuing the masterdegree in the Department of Communication Engineering,

Xiamen University, China. His research interests mainly

focus on machine learning medical image analysis.

L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 107

Xinghao Ding was born in Hefei, China, in 1977. He

received the B.S. and Ph.D. degrees from the Depart-ment of Precision Instruments, Hefei University of Tech-

nology, Hefei, in1998 and 2003, respectively. He was a

Post-Doctoral Researcher with the Department of Electri-cal and Computer Engineering, Duke University, Durham,

NC, USA, from 2009 to 2011. Since 2011, he has been aProfessor with the School of Information Science and En-

gineering, Xiamen University, Xiamen, China. His main re-search interests include machine learning, representation

learning, medical image analysis and computer vision.

Congbo Cai received Ph.D. degrees from Xiamen Uni-versity, China in 2015. He is currently a Professor with

the School of Electronic Science and Engineering, Xia-

men University. His main research interests include med-ical image processing, magnetic resonance imaging and

spectroscopy.

Yue Huang received the B.S. degree from Xiamen Uni-

versity, Xiamen, China, in 2005, and the Ph.D. degreefrom Tsinghua University, Beijing, China, in 2010. She was

a Visiting Scholar with Carnegie Mellon University from

2015 to 2016. She is currently an Associate Professor withthe Department of Communication Engineering, School of

Information Science and Engineering, Xiamen University. Her main research interests include machine learning and

image processing.

John Paisley received the B.S., M.S., and Ph.D. degrees in

electrical engineering from Duke University, Durham, NC,USA. He was a Postdoctoral researcher with the Computer

Science Departments at University of California, Berke-ley and Princeton University. He is currently an Associate

Professor with the Department of Electrical Engineering,Columbia University, New York, NY, USA, and also a mem-

ber of the Data Science Institute, Columbia University. His

current research is machine learning, focusing on modelsand inference techniques for text and image processing

applications.