RIDE: real-time massive image processing platform on

RESEARCH Open Access

RIDE: real-time massive image processingplatform on distributed environmentYoon-Ki Kim, Yongsung Kim and Chang-Sung Jeong*

Abstract

As the demand for real-time data processing increases, a high-speed processing platform for large-scale stream databecomes necessary. For fast processing large-scale stream data, it is essential to use multiple distributed nodes. So far,there have been few studies on real-time massive image processing through efficient management and allocation ofheterogeneous resources for various user-specified nodes on distributed environments. In this paper, we shall present anew platform called RIDE (Real-time massive Image processing platform on Distributed Environment) which efficientlyallocates resources and executes load balancing according to the amount of stream data on distributed environments. Itminimizes communication overhead by using a parallel processing strategy which handles the stream data consideringboth coarse-grained and fine-grained parallelism simultaneously. Coarse-grained parallelism is achieved by the automaticallocation of input streams onto partitions of broker buffer each processed by its corresponding worker node, andmaximized by adaptive resource management which adjusts the number of worker nodes in a group according to theframe rate in real time. Fine-grained parallelism is achieved by parallel processing of task on each worker node andmaximized by allocating heterogeneous resources such as GPU and embedded machines appropriately. Moreover, itprovides a scheme of application topology which has a great advantage for higher performance by configuring theworker nodes of each stage using adaptive heterogeneous resource management. Finally, it supports dynamic faulttolerance for real-time image processing through the coordination between components in our system.

Keywords: Real-time, Image processing, Distributed and parallel processing, Heterogeneous computing

1 IntroductionToday, data generated in real time, such as CCTVimages, web logs, satellite images, and stock data, isincreasing in volume, and there is a need to processlarge-scale data rapidly. Distributed processing technolo-gies such as Hadoop [1] using multiple nodes have beendeveloped to process large-scale data. It has becomepopular, due to its Mapreduce model using the HadoopDistributed File System (HDFS) [2] and automatic datamanagement. However, Hadoop is designed for batchprocessing. That means, Hadoop takes a large dataset ininput all at once, process it, and write a large output.The concept of Mapreduce is geared toward batch butnot real-time.Some frameworks adopt the micro batch processing

on Mapreduce model for real-time processing [3, 4]which performs the map and reduce operation manytimes whenever input data occurs in real time. It is a

special case of batch processing when the batch size issmall. It can simply process real-time streams usingexisting Mapreduce models. However, it is even lesstime-sensitive than near real-time. Generally, batch pro-cessing involves three separate processes such as datacollection, map, and reduce. For this reason, it couldincur latency costs when processing large-scale images.There is another approach for stream data processing,

Storm [5] which is a representative framework ofreal-time distributed processing. It is a task parallel con-tinuous computational engine. It defines its workflows inDAG (directed acyclic graphs) called topologies. Thesetopologies run until shutdown by the user or encounte-ring an unrecoverable failure. There has been an attempton Storm for distributed processing of stream images inreal time [6]. However, it has a problem of processingspeed degradation due to assignment of excessive over-lapping areas for each image among distributed nodes.Moreover, Storm does not support the management andallocation of heterogeneous resources considering the* Correspondence: [email protected]

Department of Electrical Engineering, Korea University, Seoul, Korea

EURASIP Journal on Imageand Video Processing

© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made.

Kim et al. EURASIP Journal on Image and Video Processing (2018) 2018:39 https://doi.org/10.1186/s13640-018-0279-5

http://crossmark.crossref.org/dialog/?doi=10.1186/s13640-018-0279-5&domain=pdf

http://orcid.org/0000-0001-9654-8406

mailto:[email protected]

http://creativecommons.org/licenses/by/4.0/

performance of each resource for real-time imageprocessing.The aforementioned distributed processing systems

exploit high-speed processing technology for massiverepetitive operations of simple task. They use acoarse-grained parallelism, allowing distributed nodes toconcurrently process the largely divided task assigned tothem. However, they can only achieve the maximumperformance when there is no data dependency betweenthe tasks. Therefore, they are not proper for image pro-cessing applications with large data dependency withineach image.Also, there have been attempts to process the massive

stream data such as satellite image using a GPU acceler-ator on single node [7–11]. GPU-based image processingshowed better performance than CPU based one. It usesfine-grained parallelism on single node, allowing singlenode to process the finely divided tasks. However, thehigher speed stream images are generated, the more un-processed images are accumulated, which leads to sharpincrease in latency.In this paper, we shall present a new platform called

Real-time massive Image processing platform on distrib-uted Environment (RIDE) which can process real-timemassive image stream on distributed parallel environmentefficiently by providing a multilayered system architecturewhich supports both coarse-grained and fine-grainedparallelisms simultaneously in order to minimize the com-munication overhead between the tasks on distributednodes. Coarse-grained parallelism is achieved by theautomatic allocation of input streams onto partitions eachprocessed by its corresponding worker node, and maxi-mized by adaptive resource management which adjuststhe number of worker nodes in a group according to theframe rate in real time. Fine-grained parallelism isachieved by parallel processing of task on each workernode and maximized by allocating heterogeneousresources such as GPU and embedded machine appropri-ately. RIDE provides a user friendly programmingenvironment by supporting coarse-grained parallelismautomatically by the system while the users only need toconsider fine-grained parallelism by careful parallel pro-gramming on multicore or GPU. For real-time massivestream image processing, we design a distributed buffersystem based on Kafka [12] which enables each of distri-buted nodes to access and process the buffered image inparallel [13], improving its overall performance sharply.Besides, it supports dynamic allocation of partitions toworker nodes which maximizes the throughput bypreventing worker nodes from being idle. Moreover, itprovides a scheme of application topology which has agreat advantage for higher performance by configuring theworker nodes of each stage using the adaptive heteroge-neous resource management. Finally, it supports dynamic

fault tolerance for real-time image processing through thecoordination between components in our system.The rest of the paper is organized as follows: In section

2, we give an overview about our system, In section 3, weexplain about the main methods of RIDE, and then, insection 4, we present a system architecture for RIDE. Insection 5, we describe about the experimental result anddiscussion about RIDE. Finally, section 6 summarizes theconclusion of our research.

2 System overviewRIDE is designed as a system suitable for processinglarge-scale stream image such as satellite, CCTV, anddrone images in real time. It can receive and processdata from multiple channels of real-time sensors orcameras. Its architecture consists of four layers: userinterface, master, buffer, and worker as shown in Fig. 1.In the application layer, the user creates an applicationfor massive image processing and submits it to the mas-ter layer. In the master layer, the master node allocatesbroker and worker nodes in buffer and worker layersrespectively and then distributes stream images ontopartitions in broker nodes. Finally, it divides the applica-tion into tasks, and assigns them to worker nodes eachprocessing one partition in buffer layer.The buffer layer consists of several topics, each of

which stores real-time images which are distributed ontomultiple partitions residing in the single or distributedbroker nodes based on the file system using Kafka. Eachtopic consists of multiple partitions coming from thesame source, and each of multiple partitions in the topicis processed by a single worker so that multiple parti-tions can be processed simultaneously by severalworkers as shown in Fig. 2.The worker layer is made up of worker nodes each of

which executes the task assigned from the master layer byaccessing and processing one partition of the topic in thebuffer layer. We can achieve coarse-grained parallelism bymaking each worker node work on the different partitionin the same topic simultaneously, while improvingfine-grained parallelism by further dividing the image intosub-images each being processed on heterogeneous re-sources such as GPU and multicores in each worker node.

3 MethodsIn this section, we shall describe several methods of oursystem RIDE: hybrid parallelism, adaptive heterogeneousdynamic resource management, application topology,and fault tolerance.

3.1 Hybrid parallelismThe previous distributed processing systems based onHadoop divide input data into a block of HDFS andpush them into the worker nodes [1–4]. In this case, the

Kim et al. EURASIP Journal on Image and Video Processing (2018) 2018:39 Page 2 of 13

Fig. 1 System overview of RIDE

Fig. 2 Partitioning of input stream for multiple nodes processing


input data is divided into the same size regardless of theuser application. Therefore, when processing an image,it is divided into blocks of predefined size, and they maybe transmitted to different nodes. Due to the nature ofimage processing, one frame is represented by a matrixor vector, and the most of operations required for imageprocessing have dependencies within the same matrix.Thus, in case dependent data may be distributed to dif-ferent nodes, there arises large overhead due to frequentcommunication between distributed nodes. Therefore,the previous distributed processing systems based onHadoop cannot efficiently support coarse-grained paral-lelism for image processing applications due to the datadependency between the partitioned data. Figure 3shows the characteristics of the previous systems basedon HDFS.RIDE supports coarse-grained parallelism by distribut-

ing each whole frame into one of multiple partitionswithin topic without no division based on Kafka buffersystem. Since each worker node accesses one partitioneach consisting of non-partitioned frames, it can effi-ciently perform coarse-grained data parallel processingwithout communication overhead.

After importing the data from the partition, eachworker node performs the fined grained parallelism onmulticores or GPU. Therefore, RIDE efficiently supportscoarse- and fine-grained parallelisms simultaneously bydistributing frames to each worker node and then pro-cessing them on multicore or GPU. The former isachieved by the automatic allocation of partitions toeach worker node, while the latter by parallel processingof each partition on each worker node using multicoresor GPU. Since coarse-grained parallelism is automatic-ally supported by the system, the users only need toconsider fine-grained parallelism by careful parallel pro-gramming on multicore or GPU. These characteristicsare shown in Fig. 4 below.

3.2 Adaptive heterogeneous dynamic resourcemanagementRIDE maximizes hybrid parallelism by supporting thefollowing three types of resource management schemes:

Adaptive resource management: The number ofprocessing nodes can be flexibly adjusted consideringthe amount of image frames generated in real time.

Fig. 3 Data partitioning on HDFS


User can set up a group of multiple worker nodes eachof which processes the same task. As shown in Fig. 5,several workers can be grouped together by user, andeach group is assigned the same task. Each worker in agroup is mapped to one distinct partition and grantedits ownership. Then, each worker can only access thepartition with its ownership. Therefore, user canachieve coarse-grained parallelism by adaptive resourcemanagement which adjusts the number of workernodes in a group according to the frame rate in realtime.Heterogeneous resource management: User canmaximize the fine-grained parallelism of image pro-cessing task given to a group by allocating heteroge-neous resources in each worker node of the group.According to the type of task, user can choose theoptimal memory capacity, the number of CPU coresand/or GPU. For example, in case the task containsmany conditional and branching statements, multi-core may be advantageous over GPU, and hence, theworkers in the group may be configured into multi-cores for higher performance, while in case the taskrequires high processing power, they may be config-ured into GPUs.

Dynamic resource management: If a worker node isstatically assigned data, each worker node can have adifferent processing speed, so the total throughput isdetermined by the worker node that most recentlycompleted the task, which may cause the overallperformance degradation. To overcome this problem,after resource allocation, the partitions in a topic aredynamically assigned to each node in a group.Whenever each worker node completes the processingof a partition assigned to it, it is allocated anotherpartition dynamically. The dynamic allocation ofpartitions to worker nodes maximizes the throughputby preventing worker nodes from being idle.

3.3 Application topologyIn RIDE, user can define a workflow called applicationtopology which defines the flow of stages for image pro-cessing job. Each stage of workflow consists of threecomponents: a topic of partitions, a task, and a group ofworker nodes as shown in Fig. 6. Each worker node exe-cutes the same task to process the unique partitionassigned to it. Each stage in an application topology is aunit of processing the task, and the whole applicationtopology has a great advantage for higher performance,

Fig. 4 Data partitioning on RIDE


since it may configure the worker nodes of each stageusing the adaptive heterogeneous resource management.

3.4 Fault toleranceRIDE supports dynamic fault tolerance for real-time imageprocessing. Each frame from stream source is stored into

one of multiple partitions in a round-robin manner andprocessed by the task in the node with the ownership overthe partition as shown in steps 1 and 2 of Fig. 7. Each frameis stamped with ACK (Acknowledgement) after it has beencompletely processed by the task in the node with the ow-nership over the partition containing the frame.

Fig. 6 Application topology on RIDE

Fig. 5 Resource allocation on RIDE


For fault tolerance, user may request several spareworker nodes among the remaining available resources.Resource monitor in master node periodically checksthe status of each worker node by sending heartbeat. Ifit detects the failure of a worker node, the master nodemoves one of spare nodes into the working group towhich the failed node belongs to, and then deploys thetask into the new spare node with the ownership of thepartition which the failed node have processed so far atstep 3 of Fig. 7. Then, it executes the task which beginsto process the frames starting from the oldest one withno ACK at step 4 of Fig. 7.

4 System architectureIn this section, we shall describe a system architecture ofRIDE in detail. Our resource management scheme is in-fluenced by multilayered based architectures [14, 15]. Asshown Fig. 8, RIDE consists of four layers: application,master, buffer, and worker layers. In application layer,users develop applications for image processing jobs andexecute them through job management service afterassigning the proper resources by resource managementservice based upon the information collected throughresource monitoring service. In the master layer, the userdefines the configuration of broker nodes and partitionsin each topic through topic manager, allocates the re-sources for broker and worker nodes through resourceagent manager, and submits the job onto the workernodes through task manager. In the buffer layer, brokernodes connect the input image stream and worker nodes

by providing the intermediate storage on Kafka buffersystem for storing the input stream into partitions eachof which in turn is processed by the correspondingworker node. In the worker layer, multiple workers aredefined as a group in charge of processing the partitionsin a topic by executing the same task. Each worker in agroup has the ownership over one distinct partition andexecutes the task over it.

4.1 User interface layerUser interface layer provides an interface for applicationdevelopment, resource management service, resourcemonitoring service, and job management service. Userscollect the available resource information including thenumber of resources, memory capacity, and performancevia the resource monitoring service. Users request thespecification of resources necessary to execute their ap-plication to the resource management service, which inturn assigns the most appropriate resources based uponthe current resource information by sending the re-source specification to the resource agent manager inthe master layer for creating the proper environment foreach allocated resource. There are two types for allo-cated resources: broker nodes and worker nodes in thebroker layer and worker layer, respectively. For theformer, Kafka buffer system is installed automatically forbuffering the data between input stream and workernodes, while for the latter the necessary modules forexecuting the task such as JCUDA [16]. Users submittheir application through the job management service,

Fig. 7 Fault tolerance of RIDE


which in turn asks the job submitter in the master layerto execute it on the allocated resources. The resourcemanagement service exploits adaptive resource manage-ment which adjusts the number of worker nodes in agroup according to the frame rate in real time.

4.2 Master layerMaster Layer is responsible for resource management,application deployment, execution and fault tolerance.Resource requester in the master layer asks the resourceagent manager to allocate the proper resources for bro-ker nodes and worker nodes based on the informationabout resource specification received from the userinterface layer and to create resource agent, resourcestatus monitor, task agent, and task executor in each ofbroker and worker nodes in the buffer and workerlayers, respectively. The resource agent manger createsresource controllers in the master layer each of which isconnected to one of broker and worker nodes and incharge of its life cycle and status monitoring through theresource agent and resource status monitor, respectively.

The task manager creates and manages task controllerin the master layer each of which is connected to one ofbroker and worker nodes and in charge of deploying andexecuting the task through the task agent and task ex-ecutor, respectively. The topic manager creates the topiccontroller in the master layer each of which is connectedto one of broker nodes and controls the lifecycle oftopics and the configuration of partitions in the bufferlayer. The job submitter requests the task controller inthe task manager to execute the Kafka buffer systemthrough the task executor in broker nodes, and the topiccontroller in the topic manager requests the Kafka buffersystem to create each topic and its partitions in thebuffer layer according to the application configurationreceived from the user interface layer.Meanwhile, the job submitter asks the task controller

in the task manager to deploy the task onto eachallocated worker node through the task agent and thenexecute it through the task executor in the worker layerautomatically. The resource monitor collects informa-tion about the status of nodes through the resource

Fig. 8 RIDE architecture


controller interacting with the resource status monitorand transfers the current status to users via the resourcemonitoring service in the user interface layer.

4.3 Buffer layerThe buffer layer serves as a temporary repository be-tween input image stream and worker nodes. It consistsof several broker nodes each storing input stream datawhich are processed by worker nodes. Broker nodes inthe buffer layer are allocated, and Kafka buffer system isautomatically installed in broker nodes by the resourceagent manager in the master layer. Whenever a brokernode is allocated, the resource agent manager creates aresource controller within itself to manage the life cycleof a broker node through the resource agent in brokernode. After the Kafka buffer system is being activated bythe task manager, it creates topics and partitions in eachtopic onto broker nodes by the command issued fromthe topic manager according to the application

configuration and transfers input images to the parti-tions in the buffer of the broker node in round-robinmanner for load balancing.

4.4 Worker layerWorker Layer consists of several worker nodes eachwith the resource agent, resource status monitor, taskagent, and task executor. The resource agent is in chargeof the lifecycle of a worker node, and the resource statusmonitor periodically sends the available resource statesand heartbeat to the resource monitor in the masterlayer for fault tolerance and resource information. Thetask agent deploys the task, and the task executor exe-cutes it on heterogeneous resource such as multicoresor GPUs after receiving the corresponding commandfrom the task controller in the master layer. Each taskallocated in the worker node is mapped to one uniquepartition in a topic and has the ownership over it ac-cording to the preconfigured information received fromthe topic manager. It fetches data by accessing the

Table 1 System specifications for experiments

Type CPU RAM Accelerator The number of nodes Remarks

Master DuoE84003.00 GHz

4GB N/A 1

Worker Quadi7-7700

16GB GTX10708GB

6

Broker DuoE67502.66 GHz

4GB N/A 3 HDD 3T

Streaming source node Quadi5-35703.40 GHz

12GB N/A 1 30 fpsstreaming

N/A not applicable

Fig. 9 Comparison of parallelizing performance on two modes of RIDE


partition with ownership and process it on single ormultiple heterogeneous resources.

5 Experimental results and discussionOur experiment environment for RIDE is constructed asa cluster of nodes: a stream node, a master node, threebroker nodes, and six worker nodes. Table 1 shows thespecifications of our system environment used to evalu-ate RIDE. A worker node has two types of resourcemode: CPU-only mode and heterogeneous mode usingCPU and GPU. Input images are transferred to the parti-tions in the buffer of the broker node at 30 fps in around-robin manner for load balancing. Since all thestream data used in the experiments are generated onlyfor the processing time measurement, randomlygenerated dummy data are used. For the validity ofexperimentation, stream data of various resolutions aregenerated.First, we evaluate the performance of four image

processing applications on CPU only mode and

heterogeneous mode using GPU respectively on RIDE:standard deviation filter [17], surf [18], matrix multipli-cation [19], and FFT [20]. Figure 9 shows the executiontime of four applications for 24,160 × 25,720 resolutioninput images after storing them in the topic of threebroker nodes. It shows that heterogeneous mode hasmuch better performance over CPU-only mode. Also,our system can prevent the problem of processing speeddegradation as in Storm due to the communicationoverhead arising [6] from the assignment of partitionedsub-images among distributed nodes by each workernode working on the whole image frame in the partitionof a topic.Second, we evaluate the performance of standard devi-

ation filter on CPU-only mode and heterogeneous modeusing GPU with respect to the number of worker nodeson RIDE in Figs. 10 and 11 respectively. Figures 10 and11 show the execution time for processing 1000 inputimages each with variable resolutions 900 × 900, 1600 ×1600, and 2500 × 2500 while increasing the number of

Fig. 10 Performance comparison of CPU resource with respect to the number of nodes on RIDE

Fig. 11 Performance comparison of GPU resource with respect to the number of node on RIDE


worker nodes after storing input images in three brokernodes. Each worker node is mapped onto one partition,that is, the number of partitions is identical to that ofworker nodes. They show that the total execution timedecreases as the number of resources increases, but theoverall efficiency is maximized due the low communica-tion overhead when the number of worker nodes isidentical to that of the broker nodes, since a broker nodeis mainly in charge of only one worker node.Third, we evaluate the dynamic node scalability of

RIDE when increasing the number of worker nodes dur-ing standard deviation filter processing without systemshutdown for the input stream generated at 30 fps. Inputstream resolution is 900 × 900. Initially, each of threebroker node has one partition which is processed by oneof three worker nodes. In Fig. 12, the red line indicatesthe amount of unprocessed stream data so far after gen-erated in real time, and the light blue line indicates theamount of processed stream data by worker nodes.Stream image data is not processed for 45 s after start ofinput stream generation. At 45 s, one worker node startsto process the frames with processing speed of 3.4frames per second. After 80 s, two worker nodes processthe frames with processing speed of 7.8 frames persecond. After 120 s, three nodes process the stream datawith processing speed of 11.89 frames per second.

During period between 45 and 80 s, the amount ofunprocessed data increases, since the rate of incominginput stream data is much greater than that of process-ing data. During the period between 80 and 120 s, theamount of unprocessed data begins to decrease with therate of incoming input stream data being smaller thanthat of processing data. During the period between 120and 180 s, the amount of unprocessed data decreasessharply with the rate of processing data becoming muchgreater than that of incoming input stream data. That is,as shown in Fig. 12, the red line falls down sharply, andthe slope of light blue line gets higher. Stream data isgenerated until up to 167 s, and all the data is processedabout at 180 s with the red line converging to zero.Figure 12 shows that as the number of nodes increasesdynamically during real-time image processing, we canachieve coarse-grained parallelism more efficiently.Finally, we experiment the fault tolerance in order to

show whether it is possible to recover by using the avail-able spare nodes when fault occurs by forcibly removingsome worker nodes during standard deviation filter pro-cessing. We use input stream of 900 × 900 resolutions.In Fig. 13, the blue line represents the amount ofgenerated stream data while the light blue line theamount of processed stream data. At 45 s, oneworker node starts to process data, and after 80 s,

Fig. 12 Performance of dynamic node scalability

Fig. 13 Dynamic fault tolerance on RIDE


three worker nodes simultaneously process the data.After fault occurs at two worker nodes at 110 s, it isdetected and recovered by replacing them with twoother spare worker nodes. Figure 13 shows that im-mediately after the failure of the nodes, the process-ing speed is 3.1 frames per second, but it isrecovered to 11.3 frames per second within 10 s.

6 ConclusionsIn this paper, we have presented a new platform calledRIDE which can process real-time massive image streamon distributed environment efficiently by providing amultilayered system architecture which can supportcoarse- and fine-grained parallelisms simultaneously inorder to minimize the communication overhead betweenthe tasks on distributed nodes. Coarse-grained parallel-ism is achieved by the automatic allocation of inputstreams onto partitions in the broker layer each proc-essed by its corresponding worker node, and maximizedby adaptive resource management which adjusts thenumber of worker nodes in a group according to theframe rate in real time. Fine-grained parallelism isachieved by parallel processing of task on each workernode and maximized by allocating heterogeneousresources such as GPU and embedded machine appro-priately. For real-time massive stream image processing,we design a distributed buffer system based on Kafkawhich enables each of distributed nodes to access andprocess the buffered image in parallel, improving itsoverall performance sharply. Also, RIDE provides a userfriendly programming environment by supportingcoarse-grained parallelism automatically by the systemwhile the users only need to consider fine-grained paral-lelism by careful parallel programming on multicore orGPU. Besides, it supports dynamic allocation of parti-tions to worker nodes which maximizes the throughputby preventing worker nodes from being idle. Moreover,it provides a scheme of application topology which has agreat advantage for higher performance by configuringthe worker nodes of each stage using the adaptive het-erogeneous resource management. Finally, it supportsdynamic fault tolerance for real-time image processingthrough the coordination between components in themaster layer and worker layer (Fig. 13).Our system can be efficiently exploited as a distributed

parallel image processing platform for processinglarge-scale real-time image stream data based on deeplearning model, since our system architecture for imple-menting coarse-grained and fine-grained parallelismscan be directly used for real-time image processing usingdeep learning based model. Also, we believe that oursystem can be efficiently used as distributed parallel plat-form for other AI areas for processing big data such as

natural language processing for fake news detection,chat bot, robotics, and game.

AbbreviationsACK: Acknowledgement; DAG: Directed Acyclic Graphs; FFT: Fast FourierTransform; HDFS: The Hadoop Distributed File System; RIDE: Real-TimeMassive Image Processing Platform on Distributed Environment

FundingThis research was supported by the Basic Science Research Program throughthe National Research Foundation of Korea (NRF) funded by the Ministry ofEducation (2017R1D1A1B03035461), the Brain Korea 21 Plus Project in 2018,and the Institute for Information & communications TechnologyPromotion(IITP) grant funded by the Korean government (MSIP) (No. 2018-0-00739, Deep learning-based natural language contents evaluation technologyfor detecting fake news).

Availability of data and materialsAll data is included in the manuscript.

Authors’ contributionsY-KK presented the main idea and designed and developed the system. YSKprovided the experimental data and conducted the experiments. C-SJ is aproject director for the researches on the topic of the paper, designed theoverall system architecture, and edited the manuscript. All authors discussand revise the manuscripts. All authors read and approved the finalmanuscript.

Authors’ informationYoon-Ki Kim is currently working toward the Ph. D. degree in Electronic andComputer Engineering at the Korea University. His research interests includehigh -performance computing, real-time distributed, and parallel dataprocessing for IoT, Sensor data processing.Yongsung Kim received the M.S. degree in Electrical and ComputerEngineering from the Korea University, Seoul, Korea, in 2013. He is a Ph. D.student in School of Electrical Engineering from the Korea University, Seoul,Korea. His current research interests include educational technology,e-learning system, machine learning, and semantic web application ineducation.Chang-Sung Jeong is a professor at the Department of Electrical Engineeringat Korea University. Before joining Korea University in 1992, he was aprofessor at POSTECH during 1982–1992. He was on editorial board forJournal of Parallel Algorithms and Application in 1992–2002. Also, he was achair of IEEE Seoul Section and has been working as a chairman ofComputer Chapter at Seoul Section of IEEE region 10. He was a chairman ofEE department in Korea University and a leader of BK21 project. His researchinterests include distributed parallel computing, cloud computing,networked virtual environment, and distributed parallel deep learning forreal-time image processing and visualization.

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Received: 15 January 2018 Accepted: 17 May 2018

References1. D Jeffrey, S Ghemawat, MapReduce: simplified data processing on large

clusters. Commun. ACM 51(1), 107–113 (2008)2. K Shvachko, H Kuang, S Radia, R Chansler, The Hadoop Distributed File

System. IEEE 26th Symposium in Mass Storage Systems and Technologies(2010), pp. 1–10

3. Condie, Tyson, Neil Conway, Peter Alvaro, Joseph M Hellerstein, KhaledElmeleegy and Russell Sears, MapReduce online. In Proceedings of NSDI.10(4), 20(2010)


4. M Zaharia, M Chowdhury, T Das, A Dave, J Ma, M McCauley, MJ Franklin, SShenker, I Stoica, in Proceedings of the 9th USENIX Conference on NetworkedSystems Design and Implementation. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing (2012), p. 2

5. A Toshniwal, S Taneja, A Shukla, K Ramasamy, JM Patel, S Kulkarni, J Jackson,et al., in Proceedings of the 2014 ACM SIGMOD International Conference onManagement of Data. Storm@ twitter (2014), pp. 147–156

6. D-H Hwang, Y-K Kim, C-S Jeong, Real-Time Pedestrian Detection Using ApacheStorm in a Distributed Environment. Seventh International Conference onNetworks & Communications (2010), pp. 211–218

7. L Fang, M Wang, D Li, J Pan, CPU/GPU near real-time preprocessing for ZY-3 satellite images: relative radiometric correction, MTF compensation, andgeocorrection. ISPRS J. Photogramm. Remote Sens. 87, 229–240 (2014)

8. L Fang, M Wang, D Li, J Pan, MOC-based parallel preprocessing of ZY-3satellite images. IEEE Geosci. Remote Sens. Lett. 12(2), 419–423 (2015)

9. Y Ma, L Chen, P Liu, L Ke, Parallel programing templates for remote sensingimage processing on GPU architectures: design and implementation.Computing 98(1-2), no. 7–no.33 (2016)

10. F Zhang, G Li, W Li, H Wei, H Yuxin, Accelerating spaceborne SARimaging using multiple CPU/GPU deep collaborative computing. Sensor16(4), 494 (2016)

11. I-K Jeong, E-J Im, J Choi, Y-S Kim, C Kim, in Proceeding of the 33rd AsianConference on Remote Sensing. Performance comparison of GPU and CPUfor high-resolution satellite image processing (2012), pp. 1489–1493

12. J Kreps, N Narkhede, J Rao, Kafka: A Distributed Messaging System for LogProcessing. Proceedings of the NetDB (2011), pp. 1–7

13. Y-K Kim, C-S Jeong, Large Scale Image Processing in Real-Time Environmentswith Kafka. Proceedings of the 6th AIRCC International Conference on Parallel,Distributed Computing Technologies and Applications (PDCTA) (2017), pp.207–215

14. I-Y Jung, B-J Han, H Lee, C-S Jeong, DIVE-C: Distributed-parallel VirtualEnvironment on Cloud computing platform. Intl J Multimed UbiquitousEngineering 8(5), 19–30 (2013)

15. AJ Younge, G Von Laszewski, L Wang, S Lopez-Alarcon, W Carithers, in GreenComputing Conference. Efficient resource management for cloud computingenvironments (2010), pp. 357–364

16. JCUDA, Java bindings for CUDA. http://www.jcuda.org/. Accessed 22 Apr2018.

17. AS Awad, Standard deviation for obtaining the optimal direction inthe removal of impulse noise. IEEE Signal Processing Letters 18(7),407–410 (2011)

18. H Bay et al., Speeded-up robust features (SURF). Comput. Vis. ImageUnderst. 110(3), 346–359 (2008)

19. K Fatahalian, J Sugerman, P Hanrahan, Understanding the Efficiency of GPUAlgorithms for Matrix-Matrix Multiplication. Proceedings of the ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware (2004), pp. 133–137

20. HJ Nussbaumer, Fast Fourier Transform and Convolution Algorithms, 2.(Springer, Berlin Heidelberg New York, 2012)


http://www.jcuda.org/

Documents

RIDE: real-time massive image processing platform on