An Active Learning Framework Incorporating User Input For

An Active Learning Framework Incorporating User InputFor Mining Urban Data

A Case Study in Dublin, Ireland

Nikolas ZygourasDepartment of Informatics and

TelecommunicationsNational and Kapodistrian

University of Athens, [email protected]

Nikolaos PanagiotouDepartment of Informatics and



Nikos ZacheilasDepartment of Informatics

Athens University ofEconomics and Business,

[email protected]

Ioannis BoutsisDepartment of Informatics


[email protected]

Vana KalogerakiDepartment of Informatics


[email protected]

Dimitrios GunopulosDepartment of Informatics and



ABSTRACTAnalyzing and detecting events from ubiquitous sensors acrossthe city has been an important goal in recent years. Differenttechniques that are able to automatically detect events bymonitoring urban sensor’s data have been efficiently appliedin several smart cities to improve the citizens everyday life.However, the analysis of such voluminous data streams ofteninterferes with several constraints that arise in smart citiesscenarios. For example it is impossible to hire human oraclesthat will monitor each data stream continuously to provideknowledge to these models and to annotate past instances.Thus, the development of novel techniques is required in or-der to build efficient supervised learning models that will beable to cope with urban data deluge. Our approach makesthe following contributions: (i) we formulate the problem ofbuilding supervised learning models efficiently by incorpo-rating streaming input from urban data, and (ii) we presenta novel framework that is able to cope with the restrictionsthat arise in the event detection of streaming urban data,requiring labels from carefully selected instances.

KeywordsUrban Data; Active Learning;Smart Cities;Event Detection

1. INTRODUCTIONUrban data monitoring has recently been efficiently ap-

plied in different smart cities providing immediately use-ful and accurate information regarding several incidents to

Copyright is held by the author/owner(s). UrbComp’16, August 14, 2016,San Francisco, USA.

Figure 1: The traffic control room at Dublin City Coun-cil, where different monitors visualize the traffic conditionsat different locations in the city of Dublin, captured fromCCTV cameras, and allow traffic operators to monitor thetraffic conditions

the city authorities in order to handle events of emergency.Nowadays, the large development of Internet of Things, withubiquitous sensors across the city, create voluminous andheterogeneous data streams that need to be automaticallymonitored. Such data streams may be provided either frommoving sensors or from static sensors. Examples of suchsensors are buses or taxis that move along the city and re-port their position, CCTV cameras that capture video atdifferent locations of the city, SCATS sensors that measurethe traffic flow at different junctions of the city, cell towersthat measure the mobile phone users that interact with theparticular cell tower and citizens that move in the city andreport several anomalies.

The data streams described above contain useful informa-tion and could be used in order to quickly identify severalevents of emergency (e.g., traffic accidents, traffic conges-tion, fires, floods, etc.). A typical example of a traffic con-

Twitter Data, from

Tweets that are

geolocated at

Dublin

Bus Data, from

Buses that move

at Dublin area

CCTV Data, from

cameras that

capture the traffic

condition at

several locations

Cell Tower Data,

from cell towers

that report the

users that interact

with

SCATS Data, from

sensors measuring

traffic condition at

junctions

Figure 2: An example of a set of static (SCATS, Cell Tower and CCTV) and moving (Bus and Twitter) data sources generatedfrom sensors that report continuously data to the traffic operators and need to be processed

trol room is the one from Dublin City Council presented inFigure 1 and an example of the data streams that need to beprocessed and analyzed is illustrated in Figure 2. The trafficcontrol rooms often have to address the following issues: (i)lots of heterogeneous information is received and needs tobe processed, (ii) the processing capability is limited thusefficient analysis techniques should be considered, and (iii)human based fusion is necessary. Thus, there is a need foralgorithms and techniques in order to automatically extractinformation and identify patterns in these data streams, ashumans are not able to monitor this information explosion.The research community in recent years focuses on the anal-ysis, the monitoring and the event detection of urban datastreams. However, there are many restrictions that are pre-venting the creation of accurate prediction models.

The main challenge that arises when data scientists ana-lyze urban data streams is that usually these data do notcontain information regarding whether an anomaly is occur-ring or not at a particular space and time. This lack of infor-mation prevents the building of accurate supervised learn-ing models (either classification or regression) that would beable to automatically detect events in the city environment.Commonly this information is provided either from the traf-fic operators or by exploiting the knowledge of the crowd byposing tasks or queries to the citizens, using crowdsourcingtechniques.

Examples of crowdsourcing tasks may be queries to thecitizens regarding whether they observe traffic congestion ata particular road or whether they observe a fire at a particu-lar address. Identifying the labels for each data point of theincoming data streams would be tremendously costly andthus unfeasible. Thus queries that request labels for par-ticular instances of the incoming data have to be selectedcarefully and then posed to the human oracles or to thecrowd, without wasting needlessly the available resources.

Active Learning [23] has been proven very efficient inbuilding supervised learning models, with a limited num-ber of labelled training data L. Active Learning techniques

are typically characterized from a cyclic procedure in whichthe learner: (i) carefully selects from a pool of unlabelleddata the most informative instances in order to improve itsperformance and (ii) learns from the provided labels andleverages its new knowledge in order to decide which pointsto query next.

This cyclic procedure is typically not applicable in mod-els that aim to detect events from urban data streams. Insuch scenarios a set of constraints exist and should be consid-ered regarding: (i) the limited number of queries that couldbe posed to the annotators (i.e., the learner should requesta small number of carefully selected instances to be anno-tated), (ii) the predefined time period in which the modelshould be ready, as in many cases we have strict deadlinestill which the prediction model should be built and (iii) thefact that it is impossible to require labels for past events(i.e., ask the citizens whether a specific road was congestedten weeks ago), thus decisions regarding whether or not torequire a particular label should be taken instantly. Theseissues make the existing knowledge extraction techniques notapplicable in such urban data analysis scenarios.

In this work we propose a novel framework that is capableto efficiently build prediction models that are able to detectanomalous events in voluminous data streams, exploitingActive Learning techniques. Our approach takes into ac-count the restrictions that arise in streaming urban datamonitoring regarding the limited number of queries thatcould be asked, the fact that we could not request labelsfor past events and that the model should be ready withina predetermined time period. Finally, the proposed frame-work is able to automatically decide whether to request thelabel of a newly received data point setting appropriately athreshold that dynamically changes over time and takinginto consideration the set of constraints discussed above.

We evaluated our proposed techniques in a real urban datascenario, coming from the city of Dublin. More specifically,we test our techniques building a classification model, thatdetects whether there is traffic or not at a particular junction

of the city. Our input stream was transmitted from SCATSsensors that are installed at different intersection of the cityand measure the traffic. We annotated the SCATS datausing input from images captured from CCTV cameras atthe junctions of the SCATS sensors.

The main constraints that we deal with in this work aresummarized bellow:

• Decide instantly whether to request the label for anewly received data point or not, as queries are notallowed for past events.

• Take into account the limited number of queries thatcould be posed to the data annotators, aiming to buildaccurate model with small budget.

• Build a classification model in a predefined time pe-riod.

• The created model should converge quickly to a batchalgorithm, that is trained offline using a larger amountof labelled data points and is trained once.

The main contributions of this work are summarized tothe following points:

• We propose a novel streaming Active Learning frame-work that is able to decide online if a new data-pointreceived is worth labelling. The system effectively ex-ploits the available budget, resulting rapidly to a goodpredictive model with a minimum number of annota-tions required.

• We evaluate our system and we show its usefulness un-der a real urban management scenario where the goalis to automatically train a model that detects trafficevents from static SCATS sensors.

The rest of the paper is structured as follows. In Section2 we discuss the recently proposed research works in urbanand traffic data management, Active Learning and crowd-sourcing systems. Then, on Section 3, we define (i) theframework’s parameters and (ii) the problem that we solve.In Section 4 we describe our proposed framework that sat-isfies to the set of constraints that were discussed above.Finally, the evaluation of the proposed techniques as well ascomparison with alternative techniques is presented in Sec-tion 5. The experimental results obtained from a real usecase and a summary of the lessons learned from this workare discussed in Section 5.

2. RELATED WORKRecently many research works focus in the analysis of ur-

ban data aiming to automatically detect events of interest.A traffic management system was proposed in [4] where com-plex events were detected monitoring heterogeneous datastreams. This technique uses crowdsourcing in order to re-solve possible uncertainties. The authors in [30] proposedan algorithm that extracts the mode of users transportationmonitoring their raw gps trajectories. Also in [15] a noveltechnique that uses a hierarchical Markov model and differ-ent levels of abstraction is able to efficiently infer the user’sdestination or the mode of transportation. The authorsin [26] proposed an interactive voting method for match-ing a raw and sparsely sampled GPS trajectory to roads

on a digital map, using information from the road networkand information extracted from the trajectory. The authorsin [22] proposed a system that monitors SCATS data and aGauss-Markov model that predicts the evolution of sensor’smeasurements over time is created based on the historicaldata. Then using Gaussian processes they estimate the cur-rent and the future conditions to the junctions where SCATSsensors are not installed. Similarly, the authors in [18] focuson mining the traffic congestion on the road network examin-ing co-occurring congestion locations. They proposed a treebased algorithm that is able to model the traffic congestionpropagation, revealing this way the vulnerabilities of the net-work. In [9] the authors proposed a method for identifyingevents in large spatiotemporal datasets and then interac-tively discovering other similar events using appropriatelyevent group indexes. Data Polygamy, a method that aimsto discover relationships between different spatio-temporaldata sets was proposed in [8], where the users are able toquery for statistically significant relationships between thedifferent datasets. Furthermore, the authors in [3] proposeda method for event detection in multiple time series, de-veloping a suite of visual analytics techniques that enables(i) transformations of the original data and (ii) investiga-tion of the events, combining interactive visualizations ontime-aware displays and maps with statistical event detec-tion methods. In [10] the authors proposed a method thatpredicts the travel time for a given origin and destinationpair, using methods from Queueing Theory and MachineLearning, that were trained on bus journey logs. In addi-tion an approach that aims to improve the social welfare ina smart city with limited resources (i.e. the roads in a trans-portation network are characterized by a limited capacity)was presented in [16]. In this work the central agent sendsin real time appropriate signals to the agents, that aim totravel across the city, reducing this way the social cost ofroad’s congestion.

In [29] the authors proposed a transfer learning system inorder to detect the optimal retail store placement using datafrom location-based social networks. In [13] the authors pro-posed a model that identifies the traffic condition monitoringsmall and sparsely sampled GPS data, creating a Bayesiannetwork that reflects the road network. In [5], [20], [34]and [27] complex event and stream processing techniqueshave been proposed in order to monitor the traffic condi-tions in smart cities. Finally, in our work in [33] a novelframework was proposes that is able to detect faulty SCATSsensor measurements using multivariate ARIMA model andtaking into consideration how much a sensor deviates fromits usual behavior, taking into account the SCATS sensorsin its neighborhood.

Active learning is an area of Machine Learning and thekey idea is that the learner is able to choose particular in-stances that will improve its performance. The three mainscenarios of Active Learning are: (i) the membership querysynthesis, in which the learner may generate any instancebelonging in the input space and request its label([2]), (ii)selective sampling, in which the learner samples a data pointand must decide whether to query or not that data point([32,14]), (iii) pool based, in which the learner queries a particulardata point that belongs to the dataset([11, 25]). In monitor-ing urban data streams with the set of restrictions describedin the previous section the most appropriate scenario is thatof selective sampling.

In [24] the authors proposed an Active Learning frame-work for on-road vehicle recognition and tracking. Initiallya recognition model is trained and then the selective sam-pling approach is performed querying the most informativesamples. In a similar manner, the authors in [31] proposea selective sampling Active Learning framework suitable forhigh volume streams that utilizes an ensemble of learners.Instead of minimizing the ensemble accuracy they proposeto minimize the ensemble variance stating the this performsbetter in cases of concept drift. A highly relevant system to[31] focusing on road networks is SmartRoad described in[12]. The system receives a stream of data and feeds themto a Random Forest classifier. The system similarly to oursdecides about labeling an instance according to a maximumbudget and the instance informativeness. As before the in-formativeness is proportional to the disagreement of the un-derlying decision trees of the random forest. The authorson [14] state the importance of covering the whole space ofa large data stream and propose the use of clustering beforequerying. Their assumption that each cluster represents adifferent subspace of the stream that needs to be explored.They initially select the cluster that needs labeling and fromthis cluster they select the most informative instance. Fi-nally, in [21] the authors used Gaussian Processes to dealwith possible disagreements among different annotators.

Crowdsourcing could be very helpful from both the in-dividual and societal viewpoint, as it enables the citizensin a participatory way of contributing to the society [28].A heat transfer model that estimates the daily mean tem-peratures from smartphone battery temperatures, exploitingcrowdsourcing is presented in [19]. The project that was de-veloped from U.S. Department of Homeland Security (DHS)aims to detect environmental threads exploiting the powerof crowd [17]. They equipped mobile phones with chemical-agent detectors and then through security networks the re-trieved data were transmitted in order to automatically de-tect and mitigate threats to urban populations. In our previ-ous work we implemented a mobile application, called Crow-dAlert [1], that enables human users to annotate real-worldevents and we have developed task assignment approachesfor crowdsourcing environments that consider user reliabil-ity and real-time constraints [6, 7]. This work exploits thefeedback, received either from traffic operators or from thecitizens, in order to develop an event detection frameworkthat is based on active learning.

3. PROBLEM DEFINITIONWe formulate our problem in an Active Learning setting

while taking into account the specific constraints that comefrom the nature of our data. We note that in our prob-lem, a query on a specific data point has to be answered bya human, potentially using a crowdsourcing mechanism inreal-time since it is an assessment of the traffic situation at aspecific point and time. This creates additional constraintson the total number of queries that can be asked, as well asthe rate at which queries can be asked.

The set of parameters that were considered in this workare presented in Table 1. More formally our problem can beformulated in detail as follows:Given:

• a set L of NL labelled data points, where for each datapoint x ∈ RD the corresponding label y is provided

Parameter DescriptionL The set of labelled dataU The set of unlabelled dataS The infinite stream of data that arrivesT The time period, in which the model

should have been createdB The number of available questions that is

allowed to ask the annotators in order toprovide labels

tslot The time slot that is required from the an-notator to provide a label for a particulardata point

Table 1: Parameters definitions

(y ∈ R or y ∈ D in case of regression or classifica-tion respectively)

• a set U of NU unlabelled data points, much larger thanthe labelled set (NU >> NL), where again each datapoint x ∈ RD

• an infinite data stream S of data points x ∈ RD

• a time period T in which the prediction model shouldhave been created

• a limited budget of B queries that the system is avail-able to pose to the annotators

• the time tslot required from an annotator to performthe annotation

Our goal is to create a framework that uses the given budgetB and build a robust supervised learning model in the givenperiod T .

4. OUR METHODIn this section we describe in detail our proposed Active

Learning framework that aims to create an accurate super-vised learning model minimizing the human effort and con-sequently the wasted resources. Our techniques consider thesystem’s constraints described in Section 3.

Initially our framework trains the supervised learning modelM using the labelled dataset L, estimating the model’s pa-rameters θ. A wide range of supervised learning models aresupported from our framework including Naive Bayes, Sup-port Vector Machine (SVM), Neural Networks and others.In this work we focus, without loss of generality, on classifi-cation models that support the probability estimation for anew instance x∗ belonging to each class, P (class = Ci|x∗, θ).Other classification or regression models could be easily sup-ported from our framework using a different query strategy.

An informativeness function informativeness(x∗, θ) is de-fined. This function estimates how much informative theknowledge of the label of a particular data point would befor the prediction model M. This function receives as in-put a new data point x∗ and the model’s parameters θ andestimates the informativeness of x∗. The most commonlyused technique for selecting the informativeness of new in-stances is the Uncertainly Sampling technique, that favoursto select those instances that are closest to the boundaries ofthe model M. Thus, we decided to use an informativenessfunction that favors to query the data points for which the

L Classifier

TBtslot

Percentage

PDF

τ

Informativeness

θ

U

Figure 3: Description of the Active Learning frameworklearning procedure: The labeled instances L are used in or-der to learn the model parameters θ and then using the un-labeled dataset the PDF of each data point in the unlabeleddataset U is identified, finally the threshold τ of informa-tiveness is calculated using the framework’s constraints T ,B, tslot

model is least confident. These points are calculated fromEquations 1 and 2, which are presented below:

xLC = arg maxx

1− Pθ(y|x) (1)

y = arg maxy

Pθ(y|x) (2)

When a model aims to learn from a limited set of trainingdata L and a voluminous data stream S that arrive sequen-tially and labels could be assigned only to new instancesand not to past events, only the selective sampling scenarioof Active Learning can be applied. The pool based scenariocan not be applied as learner don’t know in advance the la-bels of the streaming data that are going to be received inthe future. The membership query synthesis scenario is notsuitable as it would be impossible for the learners to identifythe label of multidimensional instances that is not directlyhuman interpretable. In our scenarios the model should au-tomatically decide whether to require the label for the newlyreceived data point. A naive solution in such a case wouldbe to randomly select a set of instances in the streamingdata and provide labels for those data points ignoring theirinformativeness and then use these instances in order to im-prove the classifiers performance. Another solution wouldbe to set a static threshold on the informativeness measure-ment that remains the same across the learning period. Inthis case if the threshold is high a small number of instanceswill be requested in the time period. On the other handif the threshold is low then a large number of meaninglessqueries will be posed to the users. Thus a model that isable to dynamically adjust the informativeness threshold isrequired.

Our proposed approach dynamically adjusts the thresh-old exploiting the knowledge provided from the unlabelleddataset U and the set of constraints T , B and tslot. Morespecifically in order to set up the threshold we evaluate theinformativeness for every point x ∈ U every time that themodel M is updated, as it is illustrated in Figure 3. Then

Algorithm 1 Dynamic Active Learning

1: Input: L, U , S, T , B, tslot, classifier2: Output: θ3: θ ← classifier(XL, YL)4: informativenessU ← calcInformativeness(XU , θ)5: τ ← findThreshold(informativenessU , tslot, T ,B)6: while currentT ime < T do7: xS ← getNext(S)8: infx ← calcInformativeness(xS , θ)9: if infx > τ then

10: yS ← annotate(xS)11: [XL, YL].insert([xS , yS ])12: θ ← classifier(XL, YL)13: informativenessU ← calcInf(XU , θ)14: τ ← findThreshold(informativenessU , tslot, T ,B)15: B ← B− 116: return θ

the calculated informativeness measurements are used inorder to create a histogram that illustrates the Probabil-ity Density Function (PDF) of informativeness. In orderto appropriately set the threshold we take into account thecurrent budget B, the remaining time required in order tobuild the model T and the time tslot needed to annotateeach data point. More specifically we calculate the percent-age of points that should be asked calculating (i) the totalnumber of queries that could be posed to the user, calcu-lated in Equation 3 and (ii) the percentage as the fractionof the budget of the remaining queries and the total numberof queries, estimated in Equation 4. Finally the thresholdis calculated as the value of informativeness that splits thehistogram in a way that NU×(1−percentage) points are onthe left of the threshold while NU×percentage points are onthe right of the threshold. Therefore the threshold consid-ers the knowledge of the historical unlabelled data and theprovided restrictions. Finally, it should be mentioned thatthe parameters B and T are updated over time as queriesare posed to the users and the time passes, respectively.

total =Ttslot

(3)

percentage =B

total(4)

The steps of the proposed framework are described in Al-gorithm 1. In line 3 the model M with parameters θ iscalculated using the initial data points M. In lines 5, 6 wecalculate informativeness threshold using Equations 1 and 2on the unlabelled dataset M. Finally the Active Learningcyclic learning procedure is shown in lines 6 − 15. Whenthe informativeness of the streaming instances exceed thedynamically identified threshold τ (line 9) then queries areposed in the annotators (line 10). Then the labelled datasetis updated with the new data point and its label (line 11).Finally, in lines 12 − 14 the model’s parameters θ and theinformativeness threshold are updated.

5. EVALUATIONWe performed an extensive experimental evaluation of our

proposed framework using a real world scenario with stream-ing urban data from Dublin City. More specifically, we

evaluated our approach with a traffic anomaly event detec-tion application applied on the SCATS data that are trans-mitted sequentially from the SCATS sensors to the DCC.The SCATS sensors are located at different junctions acrossDublin and sequentially report a variety of informative mea-surements for the traffic condition of the road. Examplemeasurements include the traffic flow and the degree of sat-uration of the different lanes in the junction. Setting staticthresholds on the measurements (e.g. a static threshold onthe degree of saturation) is not an effective approach fordetecting traffic events. Examples of such miss-informativesituations occur when a car is parked on top of a SCATSsensor, or when particular lanes of the road are closed formaintenance, then the default static thresholds will reportfalsely traffic events when the road is empty or no traffic willbe reported falsely.

The main issue that we had to deal with, in order to per-form our evaluation, was that the SCATS data data don’tcontain labels regarding the existence of traffic events. Thisinformation is necessary to build models that are able tomonitor these data streams and automatically detect events.In order to resolve this issue and evaluate our technique weprovided labels for the SCATS data using images capturedfrom a particular CCTV cameras that captured the SCATSsensors. Dublin City Council (DCC) uploads periodicallythe images of several CCTV cameras that are distributedin different locations of Dublin city1. We annotated someof these images provided from a particular CCTV cameraand use the annotations for labelling the sub-stream of theSCATS data that corresponds to the junction covered by thecamera. In the rest of this section we describe the approachthat we followed in order to generate the different datasets L,S and U , as well as as the performance of our approach com-pared against a technique which selects random instances forannotation with a various set of constraints.

5.1 Generating the DatasetsIn this section we describe how we integrated the SCATS

data and the corresponding labels from the CCTV cameras,in order to generate the datasets L, S and U that our frame-work requires. Initially we selected appropriately a camerathat is able to capture suitably a SCATS sensor, decidingto monitor the SCATS sensor 81 and the CCTV camera 31,that their locations are presented in Figure 4. Several im-ages captured from the camera are illustrated in Figure 5.Figures 5a and 5b were annotated as Traffic events, whileFigures 5c and 5d were labeled as No Traffic events.

Initially we created a crawler that downloaded periodi-cally the uploaded images from the different CCTV cam-eras. Then in order to easily and efficiently provide labelsfor the images of the CCTV camera 31 we created a webinterface, illustrated in Figure 6. The web interface receivesas input the id of the camera and a specified time period andreturns as output all the images that were downloaded forthat camera for this time period. Then the annotator pro-vides labels for each image using his computer’s mouse (leftclick → No Traffic, right click → Traffic). When the userprovides an input for a particular image a tuple is stored in aMongoDB database that contains the id of the CCTV cam-era, the image timestamp and the provided label (Traffic orNo Traffic). Finally the last step was to join the providedlabels with the SCATS data. In order to do this for every

1https://www.dublincity.ie/dublintraffic/

Figure 4: The locations of the camera and the SCATS sensorthat we decided to monitor

provided annotation for the CCTV camera 31 we identifiedthe SCATS measurements from sensor 81 with the closesttimestamp. Then the set of features x were the values ofdegree of saturation and traffic flow of the different lanes ofthe junction while the label y was provided from the anno-tation. This procedure generated a dataset D with 2, 386labeled SCATS data. The unlabelled dataset U consisted of50, 000 data points with the same features as D but withouta label y. The two datasets D and U are chronologicallydisjoint.

We performed our experiments measuring the accuracyfor a particular budget of available queries B and differentnumber of initial data points initDP , used in order to createour initial model. In order to create the dataset L and thestream of data points S we selected to use the first initDPdata points of the set D in order to create the labeled datasetL. The data stream S is created using the rest |D| − initDPpoints. Then in order to evaluate the performance of ourframework we performed 5-fold cross validation in the se-quentially transmitted data stream S. This set contains thelabels for each data point and when the learner requesteda label y for a particular instance of S that can be easilyretrieved.

Finally, we selected to perform our binary event detectionclassification using a SVM classifier, with a polynomial ker-nel. The SVM implementation that we used provided theclass membership probabilities that are required in Equa-tions 1 and 2, in order to calculate the informativeness.

5.2 Dynamic Vs Random SelectionIn Figure 7 we illustrate the histogram of informativeness

for the unlabeled data. As it can be observed, informative-ness takes values between 0 and 0.5. This is due to the factthat we perform binary classification. The probability of be-longing to each class is given from Equation 2. The mostinformative points are those for which the classifier is leastconfident.

We compared our technique with another approach, namedRandom selection, which selects the instances that will beannotated randomly, without taking into account their in-formativeness. We measured the accuracy of our technique,noted as Dynamic compared to the Random with the fol-lowing values of initial labelled data points [10, 25, 50, 100,200, 500] and for the following values of available queries’budget [1, 3, 5, 10, 25, 50, 100, 150, 200, 500]. Our resultsare presented in Figure 8.

From the different results illustrated in Figure 8 we can

(a) Label: Traffic (b) Label: Traffic (c) Label: No Traffic (d) Label: No Traffic

Figure 5: Screenshots from different hours of the day and different days from the camera with ID: 31, Figures (a) and (b)represent traffic events at the intersection, while Figures (c) and (d) represent the normal (no traffic) behavior. Our modelaims to detect these labels monitoring the SCATS data.

Figure 6: The web interface that helps the human annota-tors to easily select the appropriate time period for a partic-ular camera ID in order to annotate the respective images

0 0.2 0.4 0.6 0.80

2000

4000

6000

8000

10000

12000

14000

Informativeness

Nu

mb

er

of

Insta

ncce

s

Figure 7: The histogram of informativeness measurementfor the set of unlabeled data points

InititialData Points

Accuracy

10 0.5125 0.7150 0.72100 0.72200 0.76500 0.75

Table 2: The accuracy of the SVM classifier for differentnumber of initial data points and without querying the an-notators (B = 0)

observe that our Dynamic selection of querying points out-performs the Random selection in terms of accuracy. Thisis explained by the fact that our method selects for labellingthe instances with the highest informativeness score, settingappropriately the dynamic informativeness threshold. Thisthreshold is dynamically updated over time in order to en-sure that queries are posed to the users during the wholelearning period T , without frivolously wasting the budgetB.

Also we observe in all the plots that the greater the bud-get B the more accurate the classifier becomes, for bothapproaches. Thus applying Active Learning on urban dataanalysis scenarios is extremely helpful and creates robustmodels even with small amounts of labeled data points. Fur-thermore, we observe in Table 2 that the classifier with 10initial data points in the labeled dataset L has much loweraccuracy compared to the rest of the experiments, when nobudget is available (B = 0). However, having more than 25initial datapoints barely increases the accuracy.

Another observation is that the classifiers that are ini-tialized with smaller labeled initial datasets L have greaterlearning rate compared to the those that start with largerlabelled datasets. This could be explained by the fact thatthe initial labelled data points are randomly selected anda large proportion of these dataset may not be informativefor the model M. However, this observation suggests thateven if we start with small labeled dataset we can quicklyimprove it and end to a robust and accurate classificationmodel.

In addition when the classifier starts with 10 labeled datapoints and the given budget is set to 25 (35 annotated datapoints in total), Figure 8a, our method’s performance isequal to using 200 initial data points and zero budget (ac-

curacy around 0.75 as you can see in Table 2). Also theperformance achieved with those 35 annotated data pointsoutperforms that of models trained with larger number oflabeled instances (i.e., 50, 100 and 500 initial points andzero budget). This suggests that we can build accurate clas-sifiers with limited number of questions if we can carefullyselect which instances to query. Also it should be clear thatwe are able to build robust classification models that theiraccuracy converges quickly to the performance of batch ap-proaches that use much more training examples and withoutthe option to require labels for new data points.

6. CONCLUSIONIn this work we proposed a novel Active Learning frame-

work that is able to efficiently select the most informativeinstances from a set of sequentially received streaming datain order to be annotated. As presented in the evaluationof our technique our dynamical threshold outperformed therandom selection approach leading to more accurate predic-tion models. This framework could be extremely beneficialfor the analysis of urban data where the set of posed con-straints are similar with those considered in this work. Fi-nally, in our experimental evaluation we focused on a realworld scenario, creating models that detect traffic eventsfrom SCATS data streams at the city of Dublin. In our fu-ture work we plan to examine the efficiency of our approachwhen human users annotate traffic events in real-time usingour CrowdAlert application.Acknowledgments. This research has been financed bythe European Union through the FP7 ERC IDEAS 308019NGHCS project, the Horizon2020 688380 VaVeL project anda Yahoo Faculty award.

7. REFERENCES[1] Crowdalert. http://crowdalert.aueb.gr/.

[2] I. M. Alabdulmohsin, X. Gao, and X. Zhang. Efficientactive learning of halfspaces via query synthesis. InAAAI, pages 2483–2489, 2015.

[3] G. Andrienko, N. Andrienko, M. Mladenov, M. Mock,and C. Poelitz. Extracting events from spatial timeseries. In 2010 14th International ConferenceInformation Visualisation, pages 48–53. IEEE, 2010.

[4] A. Artikis, M. Weidlich, F. Schnitzler, I. Boutsis,T. Liebig, N. Piatkowski, C. Bockermann, K. Morik,V. Kalogeraki, J. Marecek, et al. Heterogeneousstream processing and crowdsourcing for urban trafficmanagement. In EDBT, pages 712–723, 2014.

[5] A. Biem, E. Bouillet, H. Feng, A. Ranganathan,A. Riabov, O. Verscheure, H. Koutsopoulos, andC. Moran. Ibm infosphere streams for scalable,real-time, intelligent transportation services. InProceedings of the 2010 ACM SIGMOD InternationalConference on Management of data, pages 1093–1104.ACM, 2010.

[6] I. Boutsis and V. Kalogeraki. Crowdsourcing underreal-time constraints. In IPDPS, pages 753–764,Boston, MA, May 2013.

[7] I. Boutsis and V. Kalogeraki. On task assignment forreal-time reliable crowdsourcing. In ICDCS, pages1–10, Madrid, Spain, June 2014.

[8] F. Chirigati, H. Doraiswamy, T. Damoulas, andJ. Freire. Data polygamy : the many-many

relationships among urban spatio-temporal data sets.In Proceedings of the 2016 ACM SIGMODInternational Conference on Management of Data(SIGMOD ’16), 2016.

[9] H. Doraiswamy, N. Ferreira, T. Damoulas, J. Freire,and C. T. Silva. Using topological analysis to supportevent-guided exploration in urban data. IEEEtransactions on visualization and computer graphics,20(12):2634–2643, 2014.

[10] A. Gal, A. Mandelbaum, F. Schnitzler,A. Senderovich, and M. Weidlich. Traveling timeprediction in scheduled transportation with journeysegments. Information Systems, 2015.

[11] S. C. Hoi, R. Jin, and M. R. Lyu. Large-scale textcategorization by batch mode active learning. InProceedings of the 15th international conference onWorld Wide Web, pages 633–642. ACM, 2006.

[12] S. Hu, L. Su, H. Liu, H. Wang, and T. F. Abdelzaher.Smartroad: a crowd-sourced traffic regulator detectionand identification system. In Information Processingin Sensor Networks (IPSN), 2013 ACM/IEEEInternational Conference on, pages 331–332. IEEE,2013.

[13] T. Hunter, R. Herring, P. Abbeel, and A. Bayen. Pathand travel time inference from gps probe vehicle data.NIPS Analyzing Networks and Learning with Graphs,12(1), 2009.

[14] D. Ienco, A. Bifet, I. Zliobaite, and B. Pfahringer.Clustering based active learning for evolving datastreams. In Discovery Science, pages 79–93. Springer,2013.

[15] L. Liao, D. J. Patterson, D. Fox, and H. Kautz.Learning and inferring transportation routines.Artificial Intelligence, 171(5):311–331, 2007.

[16] J. Marecek, R. Shorten, and J. Y. Yu. Signaling andobfuscation for congestion control. Int. J. Control,88(10):2086–2096, 2015.

[17] T. Monahan and J. T. Mokos. Crowdsourcing urbansurveillance: The development of homeland securitymarkets for environmental sensor networks. Geoforum,49:279–288, 2013.

[18] H. Nguyen, W. Liu, and F. Chen. Discoveringcongestion propagation patterns in spatio-temporaltraffic data.

[19] A. Overeem, J. R Robinson, H. Leijnse, G.-J.Steeneveld, B. P Horn, and R. Uijlenhoet.Crowdsourcing urban air temperatures fromsmartphone battery temperatures. GeophysicalResearch Letters, 40(15):4081–4085, 2013.

[20] K. Patroumpas and T. Sellis. Event processing andreal-time monitoring over streaming traffic data. InWeb and Wireless Geographical Information Systems,pages 116–133. Springer, 2012.

[21] F. Rodrigues, F. Pereira, and B. Ribeiro. Gaussianprocess classification and active learning with multipleannotators. In Proceedings of the 31st InternationalConference on Machine Learning (ICML-14), pages433–441, 2014.

[22] F. Schnitzler, T. Liebig, S. Mannor, and K. Morik.Combining a gauss-markov model and gaussianprocess for traffic prediction in dublin city center. InEDBT/ICDT Workshops, pages 373–374, 2014.

0 100 200 300 400 5000.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Budget

Accura

cy

Random

Dynamic

(a) Accuracy of 10 Initial Data Points

0 100 200 300 400 5000.65

0.7

0.75

0.8

0.85

0.9

Budget

Accura

cy

Random

Dynamic

(b) Accuracy of 25 Initial Data Points

0 100 200 300 400 5000.65

0.7

0.75

0.8

0.85

0.9

Budget

Accura

cy

Random

Dynamic

(c) Accuracy of 50 Initial Data Points

0 100 200 300 400 5000.7

0.72

0.74

0.76

0.78

0.8

0.82

0.84

Budget

Accura

cy

Random

Dynamic

(d) Accuracy of 100 Initial Data Points

0 100 200 300 400 5000.72

0.74

0.76

0.78

0.8

0.82

0.84

Budget

Accura

cy

Random

Dynamic

(e) Accuracy of 200 Initial Data Points

0 100 200 300 400 5000.72

0.74

0.76

0.78

0.8

0.82

0.84

Budget

Accu

racy

Random

Dynamic

(f) Accuracy of 500 Initial Data Points

Figure 8: Accuracy for different number of initial data points and for different values of available budget B

[23] B. Settles. Active learning literature survey. Universityof Wisconsin, Madison, 52(55-66):11, 2010.

[24] S. Sivaraman and M. M. Trivedi. A generalactive-learning framework for on-road vehiclerecognition and tracking. Intelligent TransportationSystems, IEEE Transactions on, 11(2):267–276, 2010.

[25] M. Sugiyama and S. Nakajima. Pool-based activelearning in approximate linear regression. MachineLearning, 75(3):249–274, 2009.

[26] J. Yuan, Y. Zheng, C. Zhang, X. Xie, and G.-Z. Sun.An interactive-voting based map matching algorithm.In Proceedings of the 2010 Eleventh InternationalConference on Mobile Data Management, pages 43–52.IEEE Computer Society, 2010.

[27] N. Zacheilas, V. Kalogeraki, N. Zygouras,N. Panagiotou, and D. Gunopulos. Elastic complexevent processing exploiting prediction. In Big Data(Big Data), 2015 IEEE International Conference,pages 213–222. IEEE, 2015.

[28] F. Zambonelli. Pervasive urban crowdsourcing:Visions and challenges. In Pervasive Computing andCommunications Workshops (PERCOM Workshops),2011 IEEE International Conference on, pages578–583. IEEE, 2011.

[29] N. Zhang, H. Chen, and X. Chen. Transfer learningfor urban computing: A case study for optimal retailstore placement.

[30] Y. Zheng, L. Liu, L. Wang, and X. Xie. Learningtransportation mode from raw gps data for geographicapplications on the web. In Proceedings of the 17thinternational conference on World Wide Web, pages247–256. ACM, 2008.

[31] X. Zhu, P. Zhang, X. Lin, and Y. Shi. Active learningfrom stream data using optimal weight classifierensemble. Systems, Man, and Cybernetics, Part B:Cybernetics, IEEE Transactions on, 40(6):1607–1621,2010.

[32] I. Zliobaite, A. Bifet, B. Pfahringer, and G. Holmes.Active learning with drifting streaming data. NeuralNetworks and Learning Systems, IEEE Transactionson, 25(1):27–39, 2014.

[33] N. Zygouras, N. Panagiotou, N. Zacheilas, I. Boutsis,V. Kalogeraki, I. Katakis, and D. Gunopulos. Towardsdetection of faulty traffic sensors in real-time. InProceedings of the 2nd International Workshop onMining Urban Data co-located with 32nd InternationalConference on Machine Learning (ICML 2015), Lille,France, July 11th, 2015., pages 53–62, 2015.

[34] N. Zygouras, N. Zacheilas, V. Kalogeraki, D. Kinane,and D. Gunopulos. Insights on a scalable and dynamictraffic management system. In EDBT, Brussels,Belgium, 2015.

Documents

An Active Learning Framework Incorporating User Input For